Summary: Behavioral study of expectation in political media

Aims and Hypotheses


We designed and implemented an experiment to explore the ways in which political affiliation alters how people perceive information found in a digital magazine story. We expected participants to be more likely to trust information published in an outlet that is aligned with their political stance over one that is not. We also studied whether political affiliation impacts perceptions of the author’s political leanings, and whether other factors–like how participants access news, along with their knowledge of government and current events, media literacy, and magazine readership–may also affect trust in information.


Sign up for CJR's daily email

Study participants were recruited from Amazon Mechanical Turk and TurkPrime, which was used to ensure an even demographic and political breakdown across the study sample. Eligibility was limited to those who were at least 18 years old and lived in the United States. The experiment lasted between 40 and 90 minutes, and each participant was paid $6 for completing the task. Those who attempted to repeat the experiment or who did not obtain a sufficient article attention score (described below) were excluded from all analyses. After providing informed consent, participants reported information about their own demographics, including age, education, socioeconomic status, and similar information. Next, participants answered questions about their magazine readership, including how many, how often, and which publications they read, as well as how they accessed journalism (online, print, TV, radio, etc.). Then they were given a brief description of the magazine they were about to read. Each participant was assigned to read one of two descriptions, approximately one paragraph in length, about either a fictional liberal magazine (The American Progressive) or a conservative one (The Patriot). Each layout included details, such as advertisements or recommended stories, designed to reinforce the magazine’s political ideology. Participants were not told the magazines were fictional. Next, they read the same article in one of those two publications. Subjects were instructed to read the article and respond to questions once they had finished. The “back” button was disabled so that participants could not change their earlier answers after they had read the story.

After reading, participants responded to 10 questions on factual content in the story to determine whether they paid attention and read the piece completely. If subjects did not answer at least seven questions correctly, they were excluded from analysis. Next, they answered questions about how much they believed the information presented by the author, and what they thought of the author’s political affiliation, bias, and credibility. Participants voluntarily answered questions about the history of their party affiliation, political identity, and voting record. They then rated where they perceived various magazines to fall on the political spectrum, and took a short quiz on political literacy. At the end of the experiment, all participants received a debriefing form that explained our purpose and provided a link to the original article that they had just read.


Analyses were performed using Microsoft Excel, R, and the Matlab Statistics Toolbox. Differences in trust between the two groups of interest were first analyzed based on our specific hypotheses using two-group t-tests and anova to explore factor interaction. Some exploratory analyses were performed using Pearson correlations.

Participant Demographics and Political Breakdown. After eliminating participants for poor performance on the factual story questions, our sample included 83 people who read The American Progressive and 78 who read The Patriot. Of these subjects, 81 were male and 80 were female, and the average age was 35.06 +/- 9.7 years. The average participant had at least some college education, and earned between $30,000 and $40,000 annually. Self-reported political affiliation included 30 percent Republican; 30 percent Democrat; 8 percent Libertarian; 7 percent social democrat; 12 percent fiscal conservative; 0.5 percent social conservative, 0.5 percent fiscal liberal, 5 percent social liberal, 7 percent unaffiliated. Republicans and Democrats were spread evenly between both participant groups. For those who chose to respond, the self-reported current voting status included 34 percent Bernie Sanders; 26.5 percent Donald Trump; 16 percent Hillary Clinton; 8 percent John Kasich; 10 percent Ted Cruz; 5 percent none. Just 0.5 percent preferred not to answer. Half of all participants claimed to have voted for a Democrat in any past election, and 45 percent claimed to have voted for a Republican in a past election. Age and education did not differ significantly across the groups (age: t=0.24, p=0.81; education: t=1.44, p= 0.15).

Primary analyses. Our primary analyses examined the relationship between trust and credibility responses between the two groups, and how these responses differed based on political affiliation. We first examined participants’ responses regarding whether the sources were credible, whether the author consulted enough sources, perceived bias against the police, and whether the story was believable. Responses to these questions did not differ across the two groups (The American Progressive readers and The Patriot readers), regardless of political affiliation (multiple survey questions were tested; for these analyses, p > 0.05). 

We then examined whether political affiliation affected our analyses of the responses (concerning credibility, sources, bias, and information). We examined whether these responses were related to the publication in which participants read the story, their political affiliation, and the interaction of those factors. We measured political affiliation in three ways: voting history, who the participant would probably vote for in the coming presidential election, and self-identified political affiliation. All results related to these analyses, and all group-by-politics interactions were found to be not significant (p > 0.05).


Exploratory analyses. We wanted to see whether there were other variables that affect how participants placed trust and credibility in the article. First, we examined differences in credibility scores for those who said they access news online and those who didn’t. We found that people whose primary way to access news is online reported lower credibility scores overall than those who did not access it online (t=-2.0, p=0.047). Next, we examined whether news readership varied across political affiliation. Readership was a metric calculated as an additive score reflecting participants’ reported regular access to journalism across print, online, and elsewhere. The average was found to be 11.9 in our sample, meaning participants accessed approximately 12 news sources per week. No matter how we measured political stance (voting history, self-identified party affiliation, etc.), there were no significant differences in readership across party affiliation (for political affiliation, F=0.55, p=0.57; for past voting Republican t=1.46, p=0.14, for past voting Democrat, t=1.68,p=0.09, for current voting, F = 0.02, p=0.98).

We then measured media literacy and political literacy to determine whether they had any relationship to trust and credibility. Media literacy was measured as how well participants were able to identify a news sources accurately as liberal, conservative, or neutral, and each was given a summary score in this domain. We found no relationship between media literacy and the trust and credibility metrics described above (liberal publications: r=-.05, p=0.47; conservative: r=0.09, p=0.23). Finally, we examined government and political literacy, which produced a singular score for each participant based on how well they answered basic questions about American civics. The cohort in this study earned an average score of 7.8 out of 10 (+/-1.6), but we did not find any significant relationship between the credibility scores and each individual’s performance on the government and political literacy metric (r=0.04, p=0.60), nor did we find any differences in the political literacy metric based on political identity (F=0.76, p=0.47).

Has America ever needed a media watchdog more than now? Help us by joining CJR today.

Jenna Reinen is a postdoctoral psychologist and cognitive neuroscientist at Yale. She studies how our experiences shape the way we learn and decide. Follow her @jennareinen, or visit her website.