As Polls Eagle-Eye News Media, Their Own Work Merits a Close Look
A lot of attention has been given to the public’s reputedly low estimate of the U.S. news media’s accuracy and trustworthiness. But how accurate and trustworthy are the polls rendering this harsh verdict?
The stakes are not only the future of the news media but what will happen to American democracy. The two are entwined, and both are embattled.
The media have to mobilize the right talent and show the will to find, produce, and communicate reliable news that the public needs in order to make messy democracy work. If the news media do their constitutionally empowered job, the public will be sufficiently informed, and democracy will be able to function against intensifying domestic and foreign threats and continue its long journey to a “more perfect Union.”
Given all these implications, it’s time to examine major media polling. Does it reflect a reasonable approximation of what Americans think about the news they receive, or does it, unwittingly, promote an exaggerated view of public opinion that just makes democracy messier and very nearly dysfunctional?
Gallup/Knight Foundation Survey
I’ll start with the Gallup/Knight Foundation Survey, which gets the widest public attention. The survey’s latest polling, published on June 20, states upfront and quite damningly that “Americans believe the news media have a critical role to play in U.S. democracy but are not performing that role well.”
Gallup/Knight justifies a big part of that conclusion with one number it produced from respondents’ answers—that 44% of the news Americans get from major sources is “inaccurate.” The 44% is emblazoned in large pullout type in the text of the survey and was prominently featured in the survey’s news releases.
To arrive at that 44%, the survey went through several steps of questionable polling rigor. First, it asked 1,440 respondents from the Gallup Panel this question: “If you had to estimate, what percentage of the news that you see on television, in newspapers, or hear on the radio do you think is inaccurate?”
Next, it grouped the answers about the frequency of inaccuracy in four ranges (0%-25%, 26%-50%, 51%-75% and 76% or more) displayed in the same link above.
Then, it created a mean from the percentage of answers in each range—that’s the 44% (same link).
The methodology here raises questions about the polling science. The term “inaccurate” with no qualifications can be interpreted in as many ways as there are respondents. Grouping the answers in the four ranges adds a very thin layer of quantifiable frosting to the answers.
My second example concerns what I see as distortions in this chart, where the survey tallies how its respondents rate the relative accuracy of 17 news organizations covering various channels (TV networks and cable, newspapers, websites-only, and one magazine, Mother Jones).
As the cited chart shows, only a few of the news organizations are ranked high with “net” positives—the sums of the original accuracy ratings minus the inaccuracy ratings. The highest, PBS at 38%, is not that high on a scale that goes up to 80. A big reason is that Gallup/Knight decided not to include “somewhat accurate”—one of the five accuracy ranges respondents could consider—in arriving at net scores for the 17 news entities. If that range had been included in the math, all net scores would have been higher. This chart shows the ranges of accuracy that respondents could consider—from “Very Accurate” to “Not Accurate at All.”
The news media perform even worse in the Gallup/Knight poll when scored on bias. Only seven of the 17 news sources get net positive ratings, and their scores are relatively low, as this chart shows. Among news sources receiving net negative ratings on their degree of bias are the New York Times, NBC News, and the Washington Post.
The scatter-plot chart to the left on the net positives and negatives for accuracy and bias among Republicans and party leaners has the most astounding numbers. How much polling science is invested in a chart that shows that the only news organizations to have net positives on accuracy are Fox News, the Wall Street Journal, and—I’m not making this up—Breitbart News?
In net bias scores for Republicans and party leaners, Breitbart “falls” to fourth place, but is still ranked above PBS, AP, and NPR—and way above the Washington Post and New York Times: see chart. Nationalist Steve Bannon, the former executive chairman of Breitbart and a harsh critic of mainstream news, should be very happy with the Gallup/Knight results for Republicans and party leaners.
Media Insight Project
The second polling service I looked at is the Media Insight Project, a collaboration of the American Press Institute and the Associated Press-NORC Center for Public Affairs Research at the University of Chicago. Media Insight polls get wide distribution within the news industry but not as much public attention as Gallup/Knight. The AP-NORC Center, where Media Insight’s work is done, says it is “committed to rigorous, objective research.”
Media Insight’s package for its most recent results, published in June, includes the “explainer” article “Americans and the News Media: What They Do — and Don’t — Understand About Each Other.” Stressing the importance of the news media learning from what the public has to say about them, the article states:
“The findings reveal problems of miscommunication, as well as opportunities. They highlight shared ideals: for example, the public and journalists want the same things from the press—verified facts, supplemented by some background and analysis. But they also reveal dissatisfaction: many Americans think what they see in the news media looks largely like opinion and commentary—not the carefully reported contextualizing they hoped for.”
The Media Insight report mirrors some of the disquieting conclusions of the Gallup/Knight survey, finding, for example, that 44% of its respondents “say their trust in the news has decreased in the last year.” But, digging deeper, Media Insight, citing its 2017 polling on the public and the news media, says that number can be “misleading.”
That polling concluded that “Americans can find news sources they think are accurate, fair, moral, transparent about mistakes, and trustworthy.”
Media Insight’s newest 2018 numbers from March–April 2019 back up that conclusion for the most part. The 2,019 respondents found news organization much more accurate than did the 1,440 respondents in Gallup/Knight’s polling from the same period.
Looking at coverage of various groups (women and men, Republicans and Democrats, etc.), The Media Insight respondents had inaccuracy percentages mostly in the mid- and high 20s and low to high 30s—see this chart—in contrast to Gallup/Knight’s overall inaccuracy percentage of 44%
The newest Media Insight poll also showed very high accuracy numbers for respondents who had a personal experience with news coverage, where they were interviewed by a journalist, were knowledgeable about the subject that was covered, or saw what was covered. Here is the chart with results for respondents who had one such experience, and here’s the chart for those who had more than one experience. In both cases, the levels of inaccuracies were less than half of the Gallup/Knight inaccuracy total of 44%.
Course Corrections to Consider
I don’t accept Gallup/Knight’s judgment that the news media, in fostering and protecting democracy, “are not performing that role well.” I prefer Media Insight’s less-headline-grabbing finding where 34% of Americans said the news media protect democracy, 30% said they hurt democracy, and 35% said neither statement applies.
The real threat to democracy comes from fake news providers—many of them foreign and some of those receiving support from the Russian and other governments—and, secondarily, from Facebook, Twitter, and other social platforms that, both wittingly and unwittingly, have been used as megaphones to create disorder in American public opinion and thereby disrupt the democratic process, especially as happened during the 2016 presidential election.
The social platforms, in particular Facebook, are pledging to be more vigilant against those who would manipulate them. We’ll see how this plays out as the midterm elections heat up.
The news media do need to take corrective action against specific problems that Gallup/Knight and Media Insight polling have identified. Cable TV, especially CNN, MSNBC, and Fox News, should provide clearer signposts for opinion. All the media should flag readers, viewers, and listeners when news analysis crosses the blurry line into opinion, and itemize sources for stories, especially those from the White House, when they are a combination of hard news, unattributed insider quotes, unattributed background, and aggregation.
On accuracy, news organizations across the board have to address more forthrightly the public’s concerns about the truthfulness of what is presented. Those concerns do not appear to be as great as expressed in the Gallup/Knight numbers compared to Media Insight data. But the media should avoid being defensive about the issue and candidly detail serious mistakes when they occur, refute bogus accusations, and show what corrective action they are taking against actual inaccuracies. This kind of vigorous ownership of responsibility would encourage everyone, beginning with the public, to concentrate their minds on inaccuracies that really matter.
Finally, there are the polls.
The Gallup/Knight Survey should compose its main question about inaccuracy in the news media with more nuanced precision. Instead of just asking respondents to quantify by percentage the amount of news they see as “inaccurate”—with no qualifications about magnitude—it should ask them to focus more on serious errors in news reporting.
Gallup/Knight should also rethink how it produces “net” scores from other responses where presently it does permit media accuracy to be rated by magnitude — “extremely accurate,” “very accurate,” “somewhat accurate,” “not very accurate” and “not accurate at all.” But the survey excludes “somewhat accurate” from net scores, even though that’s the category that many respondents choose as their perception of accuracy.
Weighting of the Gallup Panel’s membership should be examined to see if it is producing the desired randomized sampling of groups, especially covering Republicans and Republican leaners, who, based on their net accuracy and bias scores for the news media, appear to be overrepresented by the party faction led by such figures as Steve Bannon.
The Media Insight Project should carry on with its research showing that a “nuanced picture suggests that while people are alarmed about the state of media, they are able to find publications and sources that they not only trust but that they think are improving.” This trend counters the questionably verified story of growing distrust of the news media, the millstone that continues to be hung around the media’s collective neck.
* * *
Course corrections along these lines, carried out collaboratively in the spirit of good will, would help produce better news reporting, better polling of the media and, most importantly, a better-informed public. And just maybe democracy would get back on the path to a “more perfect Union.”