The Chart, Version 2.0: What Makes A News Source “Good?”

In my original news chart, I wrestled with the questions of what made news sources “good” and came up with some categories that generally resonated with people. I ranked sources on a vertical axis with those at the top ranked as “high quality” and those at the bottom as “low quality.” I characterized the sources, from top to bottom, in this order: Complex, Analytical, Meets High Standards, Basic, and Sensational/ Clickbait. This mostly works, because it results in sources regarded as high-brow or classy (e.g., The Atlantic, The Economist) being ranked high on the axis, and trashy sources (e.g., Addicting Info, Conservative Tribune) being ranked low, and most sophisticated news consumers agree with that. However, the vertical placements ended up causing me and others some consternation, because some of the placements relative to other outlets didn’t make sense. The most common questions I got were along these lines:

“Does FOX News really “meet high standards,” on par with something like the New York Times?” (I think no.)

“Is USA today really that bad?” (I think no.)

“Is Slate really “better” or “higher quality” than, say, AP or Reuters just because it is analytical?” (I think no.)

“Is CNN really that bad?” (I think yes.)

These questions and my instinctive responses to them made me want to reevaluate what makes news sources high or low quality.

I believe that answer to that question lies in what makes an individual article (or show/story/broadcast) high or low quality. Article quality can vary greatly even within the same news source. One should be able to rank an individual article on the chart in the same way one ranks a whole news source. So, what makes an article/story high or low quality? It’s hard to completely eliminate one’s own bias on that issue, but one way to try to do it consistently is to categorize and rate the actual sentences and words that make it up its headline and the article itself. In order to try to rank any article on the chart in a consistent, objective-as-possible manner, I started doing sentence-by-sentence analyses of different types of articles.

In analyzing what kind of sentences make up articles, it became apparent that most sentences fall into (or in-between) the categories of 1) fact, 2) analysis, or 3) opinion. Based on the percentages of these kinds of sentences in an article, articles themselves can be classified in categories of fact, analysis, and opinion as well. Helpfully, some print newspapers actually label articles as “analysis” or “opinion.” However, most news sources, especially on TV or the internet, do not. I set about analyzing stories that were not pre-labeled as “analysis” or “opinion” on a sentence-by-sentence basis. I discovered that my overall impression of the quality of an article was largely a function of the proportion of fact sentences to analysis sentences to opinion sentences. As a result, I classified stories into “fact-reporting,” “analysis,” and “opinion” stories. Ones with high proportions of “fact” sentences (e.g., 90% + fact statements) were what I refer to here as traditional “fact-reporting” news pieces. These are the kinds of stories that have historically been the basis of late 20th century-to-early-21st century journalism, and what people used to refer to exclusively as “news.” They are the “who,” “what,” “when,” and “where” pieces (not necessarily “why”). I classified ones with high proportions of “analysis” sentences (e.g., 30%-50% analytical statements) as “analysis” stories, which are the types of stories commonly found in publications like The Economist or websites such as Vox. I classified stories with high proportions of opinion sentences (e.g., 30%-50% opinion statements) as “opinion,” which are typically the types of stories found on websites such as Breitbart or Occupy Democrats.

(If you’ve made it this far, bless your heart for caring so much about the news you read.)

In the past, national evening news programs, local evening news programs, and the front pages of print newspapers were dominated by fact-reporting stories. Now, however, many sources people consider to be “news sources” are actually dominated by analysis and opinion pieces. This chart ranks media outlets that people consider to be, at some level “news sources,” even though many of them are comprised entirely of analysis and opinion pieces.

In my previous version of the chart, I had regarded analysis pieces as “higher quality” than the fact-reporting pieces because they took the facts and applied them to form well-supported conclusions. I like analytical writing, which is essentially critical thinking. However, analysis has a lot in common with opinion, and writing that is intended to be analytical often strays into opinion territory. (Note—I’m defining “analysis” as conclusions well supported by facts and “opinion” as conclusions poorly supported or unsupported by facts). Fact-reporting articles—true “scoops”—typically have the intent of just reporting the facts and typically have a very high percentage (e.g., 90%+) of fact-statement sentences, whereas both analysis and opinion articles have the intent of persuading an audience and often have a comparatively high percentage of analysis and opinion statement sentences (30%-50%). So, although I initially had the quality axis of “news” laid out top to bottom as:

That ranking is more reflective of the quality of writing rather than the quality of news sources. Good analysis is often written persuasively and well, fact-reporting is often written directly but well, and opinion writing is often (but not always) written poorly or is most easily discredited.  I submit that given the confusion caused by the overwhelming number of organizations proclaiming to be (or which are commonly confused with) “news sources,” it is more important to rank the quality of news sources than the quality of writing. I further submit, for reasons outlined below, that the percentage of fact reporting articles and stories should be used as the most determinative factor by which a news source is ranked in quality on this chart.

Therefore, I believe a more relevant ranking of the quality of news sources would be:

I assert that one of the biggest problems with our current news media landscape is that there is too much analysis and opinion available in relation to factual reporting.  New technologies have given more people more platforms to contribute analysis and opinion pieces, so many “news sources” have popped up to compete for readers’ attention. Unfortunately news consumers often do not recognize the difference between actual fact-reporting news and the analysis and opinion writing about that news.  This increase in “news sources” has not corresponded with an increase in actual journalists or news reporting, though. Many local and national print news organizations have reduced their numbers of journalists, while many of the biggest ones have merely maintained similar numbers of journalists over the past 10 years or so.
Furthermore, primarily analytical news sources also have several downsides. One downside is that they can alienate news consumers by making what people consider “news sources” so complex or partisan that it is tiring to consume any “news.”  For example, CNN, MSNBC and FOX News, which are primarily analysis and opinion-driven, can make news consumers too weary to pay attention to fact-based reporting from, say, AP or Reuters. Another problem with analysis and opinion-driven news sources is that it can be difficult for casual readers to differentiate between good analysis and pure opinion.
There are several good reasons why we should value fact-reporting sentences, fact-reporting articles, and fact-reporting news sources high on the quality scale of news, at least on this chart. For one, reported facts take a lot of work to obtain. They require journalists on the ground investigating and interviewing. Once a story is reported, dozens, hundreds, or thousands of other writers can chime in with their analysis or opinions of it. This is not to say analysis and opinion writing isn’t important. The critical thinking presented in analytical writing—especially good, complex analysis—is essential to public discourse. Our society’s best ideas are advanced by analytical articles. This piece you are reading now is analytical. But analysis in the news wouldn’t even exist without the underlying factual reporting.

For example, AP and Reuters have maintained around 2,000-2,500 journalists each over that time, while the New York Times and Washington Post have fluctuated in the 500-1000 range over the same period. The value of these organizations with large staffs of journalists, editors, and other newsroom employees is hard to overstate; not only do they provide a majority of the fact-reporting stories everyone else relies on, but they have the capacity to provide high-quality editorial review that stands up to industry scrutiny. In contrast, even some of the most popular analysis and opinion sites can be run with just a few dozen writers and staff; the number of these “news” websites, news aggregator websites, blogs, and podcasts has seemingly grown exponentially.

I believe improvements to our media landscape can be made if two things happen: 1) if news consumers start valuing factual reporting much more and analysis/opinion articles much less and 2) if news consumers become accustomed to differentiating articles in those categories.. Regarding point #2, I think it would be helpful if we narrowed the definition of “news” to only refer to fact reporting, and referred to everything else as “analysis” or “opinion.” It would be helpful if people could recognize the relative contributions of fact-reporting news organizations versus analysis and opinion sources. If people recognize just how much of what they read and watch is intended to persuade them, they may become more conscious and thoughtful about how much they allow themselves to be persuaded. One can hope.

To contribute to those goals, I’ve reordered the chart to value fact-reporting articles as the highest quality and everything else lower, even though there is some really excellent analysis out there. As a baseline, news consumers should understand when something is news (fact-reporting) and when it is not. On the new chart, the sources with the best analysis, but little reporting are at the top, but right under the sources that are comprised of high percentages of reporting articles. The most opinion-driven sources are at the bottom. There’s room for other things at the bottom below pure opinion, which can include sources that are sensationalist, clickbait, frequently factually incorrect, or which otherwise don’t meet recognized journalism standards.

On this version, I’ve included a number of different sources, mostly in the analysis and opinion categories, and left the most popular mainstream sources from the original chart, but have reordered some of them. Now, the rankings are more consistent with my initial answers to the example questions at the beginning of this post. Fox News is now ranked far lower than the New York Times for two main reasons; one, Fox News is dominated by opinion and analysis, and two, it has gotten precipitously worse in other measures (sensational chyrons, loss of experienced journalists, hyperbolic analysis by contributors, etc.) within the last six months. USA today, despite its basic nature, has been elevated because of its high percentage or fact-reporting stories. Slate, though it provides thoughtful, well-written analysis, is ranked lower than AP and Reuters, which better reflects their relative contributions to the news ecosystem. CNN still sucks, but it is clearer why now; CNN has the resources to provide twenty-four hours of news—it could provide Americans with a detailed global-to-local synopsis of the world—but instead it chooses to spend 5% of its time fact reporting a handful of stories, comprising mostly American political drama and maybe one violent leading world news story, and 95% on analysis and opinion ranging from the competent to the inane.

My analysis of news sources in the manner I’ve described herein has revealed that individual stories can and should be ranked on the chart in the same manner, and that individual stories can be placed in different places than the news sources in which they are published. I’ll be putting out individual story rankings and reasoning for those rankings from time to time for those that are interested. I’ll also take requests for rankings of sources and individual stories in the comments and on twitter. Thanks for reading and thinking.

*Update: a high-resolution PDF version is available here: Second Edition News Chart.V2

And a blank version is here: Second Edition Blank

By |2018-08-29T19:13:07+00:00August 18th, 2017|All, Chart Iterations, Media Bias, Methodology|37 Comments

About the Author:

37
Leave a Reply

avatar
20 Comment threads
17 Thread replies
1 Followers
 
Most reacted comment
Hottest comment thread
17 Comment authors
SmithKitThe Truth Behind the FBI Wiping Route 91 Festival Workers’ Devices – Media Literacy Article ReviewsAnat RonenMedia Literacy Course Project – Media Literacy Article Reviews Recent comment authors

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  Subscribe  
newest oldest most voted
Notify of
Karen
Guest
Karen

I am currently reading the National Review for the conservative take on major news stories. I would be interested to know where that would fall on the chart.

Great chart, and even better documentation. Thank you for this work!

Georgia Morgan
Guest
Georgia Morgan

Will you rank TeenVogue and GQ? I’m seeing (unexpectedly!) interesting analysis/opinion articles from them online?

Judith Farrow
Guest
Judith Farrow

Disappointed that you did not include ‘The Guardian’

Julia
Guest
Julia

Im having trouble with your “Mostly Noise” label. Is it referring to just that centre, bottom bubble? to all sources in blue bubbles? why is CNN mostly noise but FOX isnt? I feel like FOX is only “worse” than CNN because it has more political bias, and I see that in the vertical and horizontal placement of those two sources, but the “Mostly Noise” label is confusing.

JohnInDenver
Guest
JohnInDenver

Trying to generalize “local TV news” as mainstream with minimum bias is ludicrous. There are over 1000 stations, most having some sort of news. Lumping them together does not do justice to their diversity. Ideological identification is often fairly clear between different stations. The growing consolidation becomes a problem. 5 companies own about 35% of all stations. And Sinclair, if their most recent buying proposal gains approval, will have about 15% of the stations. As such, the political bent of Sinclair matters — and there is a clear, demonstrated political bent. Finally, suggesting that local TV news has “complex analysis… Read more »

Justin Haynes
Guest

All generalizations are false.

trackback

[…] Lawyer Does the News A Denver lawyer is making news with her analysis of news — she updated her news quality chart that evaluates a publication’s partisan […]

Justin Haynes
Guest

Thanks for the monumental amount of work this took. This is meta-journalism and is journalism in its own right. It required a lot of fact gathering and independent research, and quite a bit of analysis. As you alluded to in your first post, where you encouraged readers to try this exercise on their own, opinion is also baked in as everyone will find sources to be more or less liberally biased. Hopefully everyone’s result would line up vertically fairly well as we are entitled to our own opinions but not to our own facts. I didn’t even get halfway through… Read more »

NotAComMajor
Guest
NotAComMajor

Great infographic, my only question relates to the assessment of Reason.com’s analytical rigor.

Politics aside, Vice and Reason are (imo) the only two news sources that apply a collegiate level of analysis in their journalism.

Dave Hawkins
Guest
Dave Hawkins

Vanessa, I appreciate your thorough analysis. I think you’re spot on in saying that the vast majority of what passes for “news” is actually just opinion, or perhaps analysis. I’d like to challenge your thinking on whether you’ve done enough to adjust for your own admittedly left-leaning bias. From my perspective, I think your chart would be much more accurate if you left the logos in place, and picked up the grid and moved it one inch to the right. That is to say, ABC, NBC, and CBS all lean left, the NY Times and CNN are far left, and… Read more »

Susan Z Witter
Guest
Susan Z Witter

Do you have a blank chart of this second edition? I couldn’t find it.

You really hit on something here, Vanessa. Thanks for all the work you put into this. I’m about to use this material in a community workshop on spotting fake news and stopping fake news. Your ideas are central. You give me hope.

Allyson
Guest
Allyson

Do you have a link to this second chart? Using the image you have posted here is very low resolution. I’d love to have a link to a pdf. Thanks!

Amy Bryant
Guest

Hi Vanessa,
I’m a librarian who uses your chart for workshops on Fake News. I really like the new version and your logic behind it. Highlighting the difference between fact-reporting and analysis is important. Like Susan Witter, can you also send me a copy of the blank version?
Thank you again.

Mark
Guest
Mark

I’m afraid that any conservative who sees Al Jazeera and NPR as main-stream and factual will immediately dismiss your entire chart as “fake news” and perhaps even a left-wing communist plot 😉

That being said, kudos on doing a yeoman’s job. In my years of reading I’d say you’re pretty much on the mark.

Jeff
Guest
Jeff

I would be interested in how they analyze 24-hour news programming versus the network news. In considering CBS, ABC, NBC; are they only considering their official Nightly News or are they also considering the Sunday morning shows as well as the daytime talk shows that have a strong political component. As for the Big 3, 24-hour news networks; does the analysis include only the news specific programs or the entire slate of programming. Also I’d be interested to know where they consider OAN in their charting. I didn’t see it. Finally, political bias is not only found in the opinion… Read more »

trackback

[…] I discussed in my post entitled “The Chart, Second Edition: What Makes a News Source Good?” the most accurate and helpful way to analyze a news source is to analyze its individual stories, […]

trackback

[…] This revision came with an additional explanation: “The Chart, Second Edition: What Makes a News Source ‘Good?'” […]

Anat Ronen
Guest

Hi Vanessa!
I commented earlier but I don’t see my comment here. So I’ll ask again – what do you think about putting AJ smack in the middle of a bipartisan, mainstream, fact checking zone, when they are owned and managed by the Sheikhs of Qatar, a non democratic country that supports terror organizations? Thank you

trackback

[…] it has become hard to distinguish and avoid fake news because it is now everywhere. People like Vanessa Otero and Melissa Zimdars have dedicated their time to debunking sites that unfortunately deliver fake […]

Kit
Guest
Kit

This chart is very helpful, thank you !

One of my friends is criticizing my trust in Al-Jazeera English as a media source. Could you briefly explain why you give Al-Jazeera a good ranking? Thanks 🙂

Smith
Guest
Smith

@realDonaldTrump @GOP @seanhannity
@NBCNightlyNews Investigate media for conspiracy & collusion with the Democrat party, and how they influence elections— worse than Facebook! Investigate conspiracy with Democrat biased sources & use of demographic data to manipulate the public. Clear inequality and manipulation by the Left. @DevinNunes @TGowdySC

This analysis places NBC and others in the middle with little bias. That is entirely false. The bias of this evaluation and report is clear.