Ad Fontes Media’s rating method has evolved from its initial form, in which its Founder, Vanessa Otero, performed the analyses, to its current method of multi-analyst content analysis ratings. Ad Fontes finished its first extensive multi-analyst content ratings research project in June 2019, which is described in the White Paper about this project here.
From June 2019 to August 2020, a group of nine analysts from that initial project continued to rate several dozen articles per month to add new sources and update previously existing ones. From August-October 2020, Ad Fontes conducted a second large multi-analyst content ratings project to rate over 2000 articles and 100 new news sources with some existing analysts and over 30 new analysts. Prior to October 2020, all analysts were volunteers receiving perks and/or small stipends. Since October 2020, Ad Fontes Media has contracted and hired a team of analysts to rate news content on an ongoing basis.
To date, we have rated over 8500 individual articles and shows. Each article or show is rated by at least three analysts–one left, one center, and one right, politically–which means we have over 25,000 individual ratings. We have fully or partially rated over 400 individual news sources.
We use a multi-person rating per article system to minimize the impact of any one person’s political bias on the published rating. We purposefully assign each analyst a breadth of coverage over as many sources as possible to to enhance each analyst’s familiarity with sources across the spectrum.
Our content ratings periods for each rated news source are performed over a minimum of three weeks in order to capture sample articles over several news cycles. Sources that have appeared on our Media Bias Chart for longer have articles over much longer periods of time. The articles rated covered a minimum of seven articles per source, but for some larger sources we have rated over 100 articles.
Often, our sample sets of articles and shows are pulled from sites on same day, meaning that they were from the same news cycle. Doing so allows analysts to incorporate evaluations of bias by omission and bias by topic selection.
The type of rating we ask each analyst to provide was an overall coordinate ranking on the chart (i.e., “40, -12”). The rating methodology is rigorous and rule-based. There are many specific factors we take into account for both reliability and bias because there are many measurable indicators of each. The main ones for Reliability are defined metrics we call “Expression,” “Veracity,” and “Headline/Graphic,” and the main ones for Bias are ones we call “Political Position,” “Language,” and “Comparison.” There are several other factors we consider for certain articles. Therefore, the ratings are not simply subjective opinion polling, but rather methodical content analysis. Overall source ratings are composite weighted ratings of the individual article and show scores.
In our earlier ratings projects (prior to October 2020), our analysts rated articles independently, and we reviewed scores for outliers both automatically (e.g., throwing out scores greater than three standard deviations from the mean) and manually. Articles with insufficient close scores would be discarded. In our current process, we rate most articles during live shifts (on Zoom) with three analysts (one left, one right, one center), and after each article, analysts see each other’s scores and resolve discrepancies when possible. If significant discrepancies remain, the articles are rerated by a second balanced panel.
We continue to refine our methodology as we discover ways to have analysts classify rating factors more consistently. Our analysts use our software platform called CART–Content Analysis Rating system. This ratings software is currently available for use by educators in classrooms, and by individual adult learners in our news literacy courses. For educators interested in teaching students how to rate articles like Ad Fontes Media does, see here, and for individuals who want to learn themselves, see here. Our courses include detailed video and written explanations of the factors we use to rate articles.
This process has evolved over time and with input from many thoughtful commentators and experts, including Ad Fontes Media Advisor and long-time journalist and journalism professor Wally Dean.
As a background, it is helpful to understand some principles and caveats:
- The main principle of Ad Fontes (which means “to the source” in Latin) is that we analyze content. We look as closely as possible at individual articles, shows, and stories, and analyze what we are looking at: pictures, headlines, and most importantly, sentences and words.
- The overall source rating is a result of a weighted average, algorithmic translation of article raw scores. Low quality and highly biased content weight the overall source down and outward. The exact weighting algorithm is not included here because it is proprietary. Aspects of what is disclosed here are patent pending.
- The current ratings are based on a small sample size from each source. We believe these sample articles and shows are representative of their respective sources, but these rankings will certainly get more accurate as we rate more articles over time.
- Keep in mind that this ratings system currently uses humans with subjective biases to rate things that are created by other humans with subjective biases and place them on an objective scale. That is inherently difficult, but can be done well and in a fair manner. There are other good models for doing something similar, such as grading standardized written exams (like AP tests and the bar exam), or judging athletic competitions (such as gymnastics and figure skating). You can get to good results as long as you have standards on how to judge many granular details, and have experts that are trained on such standards implementing them. We’ve begun to create that process here. Below are some of those granular details.
Our CART rating interface shows each of the factors analysts are asked to consider before providing a final rating for reliability and bias.
As shown, analysts have scoring sliders for each of the reliability sub-factors and bias sub-factors. Each of these sub-factors have specific definitions and criteria, which are available in our training materials and are linked here:
How to Rate Expression
How to Rate Veracity
How to Rate Headlines and Graphics
How to Rate Language
How to Rate Political Position
How to Rate Comparison
Ad Fontes Media analysts start with these training guidelines and receive at least 20 hours of training, which includes live practice rating articles, before rating articles for inclusion in our data set.
There are several layers of data that may be of interest to researchers, media organizations, and regular chart users. Our overall source rating coordinates and individual article ratings (which are averages of the ratings for each of the three analysts who rated the article) are available for certain types of commercial or non-commercial purposes for a fee. For more information, please email us at firstname.lastname@example.org.