Illustration by Christie Chisholm

The New Yorker, BuzzFeed, and the push for digital credibility

It’s one of magazine journalism’s most pressing questions: How can publications that have long captivated print consumers earn the trust of wary online readers?

As the internet solidifies its role as a leading news source amid continued declines in print, news organization homepages are losing traction. Magazine stories are increasingly unmoored from the outlets that published them, and from the brands that once all but guaranteed their legitimacy. In the US, more than 60 percent of social media users now access news through platforms like Facebook and Twitter, and news organizations harvest nearly half their traffic from social media.

The fragmented nature of the digital landscape has created a conundrum for magazines and other news outlets. Being seen as reliable is crucial to a news organization’s survival. But if readers are finding stories in every corner of the Web, and may not even remember where they first read them, how can publishers build a loyal audience? Do brands even matter anymore?

It turns out they do. Readers are less likely to trust a longform story that appears to have run on BuzzFeed than the same article on The New Yorker’s website, according to a study by the Columbia Journalism Review and the George T. Delacorte Center for Magazine Journalism. 

We placed this Mother Jones story, which details how the FBI allegedly bullied a US citizen who refused to become an informant, in the digital skeletons of The New Yorker, BuzzFeed, and a third, fictional publication, and then probed readers’ reactions.

We chose this particular story not only for its captivating plot, but also for its use of unconventional journalistic techniques like secret recordings and pseudonyms. Those attributes led us to believe that readers might look to the brand as a compass on their path toward determining the article’s credibility. The trust gap between BuzzFeed and The New Yorker supports this hypothesis.

Sign up for CJR's daily email

However, our study also suggests that changes in how and where we read are pushing us to seek out reliability cues wherever they exist, and that readers make many judgments about the journalism itself when evaluating a story’s credibility. One study subject told us that he usually uses the publication where an article ran as a guidepost. “However, some journalists are very interesting and credible, prior to working for a famed organization,” he wrote. “Sometimes the audience must consider the context.”




We suspected that The New Yorker might be viewed as more reliable than BuzzFeed. After all, BuzzFeed owes its ascent to clickable fluff, while The New Yorker’s brand signals sophistication and top-tier journalism. But our study participants considered both publications more credible than not, a noteworthy result given the public’s overall dwindling confidence in journalism. Neither The New Yorker nor BuzzFeed chose to comment on our findings.

Trust levels seemed related at least in part to people’s reading habits. Readers whose primary source of news is the internet–more than 80 percent of our study subjects–had less faith than their peers in the article, no matter where it appeared to have run. At the same time, among our pool of subjects who read the BuzzFeed version of the story, people who prefer digital news deemed the article more credible than those who primarily seek news offline.

Once a participant believed the writer or story was credible, that conviction was reinforced by other elements that might have independently put readers off, such as the journalist’s use of secret recordings, the article’s funding source, or the scope of the reporting. By the same token, people who distrusted the magazine brand were also likely to distrust the writer.

Some study subjects failed to notice or recall the brand, a trend among online readers that was recently documented in the Reuters Institute for the Study of Journalism’s 2016 Digital News Report. In our study, people who did notice the brand tended to be older and better-read. “I realize that I don’t even know what the publication was, I just started reading the article,” wrote a 29-year-old man in the New Yorker group. “It’s not that the reputation of the publication did not affect my opinion … but more that I didn’t pay attention to it at all.”

“This is an industry that’s very much based on trust in publications and in individual journalists, and that trust is very fragile,” says Nick Baumann, who wrote the Mother Jones piece. “You have to keep earning it and re-earning it.”

A magazine’s reputation–its brand–may convince regular readers of a story’s merits. But our study suggests that building widespread trust requires greater efforts to confront online readers’ growing skepticism.

 

Altering the branding of a product–be it a beverage, ground beef, or a news article–and presenting it to study participants is a tried and accepted research technique. In media studies, this approach has been used to examine readers’ perceptions of articles of varying quality published in an esteemed German newspaper versus a tabloid; of fake news stories; and of a write-up on a home theater system, which was presented to some readers in the form of a traditional magazine article, and to others as a promotional piece penned by the product’s manufacturer. You can find our review on relevant prior studies here.


It’s not that the reputation of the publication did not affect my opinion … but more that I didn’t pay attention to it at all.”


What makes our study unique is its timing. How we consume news is arguably changing faster than ever before. Earlier studies of this kind were conducted before people got news from cellphones and Facebook, and before scrappy digital natives became major journalism sources.

We wondered how the discovery of articles through social media or other indirect avenues is redefining the way readers interact with magazine journalism in particular. At one point, the simplest definition of a magazine might’ve been the physical book, whether glossy or newsprint. TV programs like 60 Minutes and Dateline NBC are called “newsmagazines” because they also offer longform journalism in a cohesive package and with a distinct perspective, aesthetic, and editorial standard. But what makes a magazine in the digital age? And what makes us trust one magazine over another?

To undertake our research, we enlisted psychologist and cognitive neuroscientist Jenna Reinen, a postdoctoral researcher at Yale University. Because our research involved human subjects, we submitted a detailed proposal to Columbia’s Institutional Review Board, an independent group that ensured the study was ethically sound.

Next, we used Amazon’s Mechanical Turk, an online workforce service that connects businesses and institutions with people who want to complete brief tasks, such as taking a survey or transcribing an audio file, for money. We recruited 279 American adults to read our chosen story. Eighty percent of our participants were white, and 60 percent were men. Most were between 24 and 45 years old, with an average age of 34.5. Most had some college education, and earned between $20,000 and $40,000 per year. 




In contrast, The New Yorker’s audience is evenly split between women and men, with a median age of 51, a median household income of more than $100,000, and a college graduation rate of 62 percent. Data from comScore show BuzzFeed’s core website has an audience that includes roughly twice as many women as men. Fifty percent are between 18 and 34 years old, and the median household income is between $75,000 and $100,000. This comScore data is likely incomplete because people often consume BuzzFeed content through avenues other than its website. The fact that Mechanical Turk mostly attracts people searching for low-paying odd jobs helps explain why our subjects’ average income was considerably lower than that of the average BuzzFeed or New Yorker reader.

Our subjects’ main news source was the internet: 82 percent primarily found news online, followed by TV (15 percent), radio (2 percent), and print (1 percent). Strikingly, though, 40 percent said they still primarily access digital stories through a news outlet’s homepage, while 31 percent rely on social media, and 19 percent on search. The social media figure is remarkable, considering the Pew Research Center’s recent finding that just 18 percent of Americans use those platforms to find news “often.” 

Despite digital’s strength, most subjects–nearly 70 percent–said they subscribe to at least one print magazine. That’s not so shocking given the stable circulation numbers of news magazines in recent years. Just 19 percent of subjects claimed to subscribe to any newspaper, a symptom of the accelerating decline in that industry.

Our subjects agreed to participate in a 45- to 60-minute “journalism and psychology” experiment, for which each was paid $6. We randomly assigned them to read the story about the FBI informant in the style of The New Yorker, BuzzFeed, or a fictional publication called The Review, whose clean, pared-down design was inspired by The Washington Post’s website. The Review readers made up our “control group,” designed to test reactions apart from the biases attached to established brands like The New Yorker or BuzzFeed. In all three versions, we retained the original author’s byline. By signing our consent form, subjects agreed not to “recreate or redistribute” any material presented in the study.


This is an industry that’s very much based on trust in publications and in individual journalists, and that trust is very fragile. You have to keep earning it and re-earning it.”


After finishing the piece, subjects were asked several reading comprehension questions. Those who couldn’t answer basic plot-related questions, or tried to take the survey more than once, were eliminated from the study. That left 89 people in the BuzzFeed group, 93 in The New Yorker group, and 85 in the control group. Our findings are based on their responses to 17 questions designed to gauge their trust in the writer, the story, and the publication itself.

To measure their skepticism, we spotlighted individual facts and quotes in the story, and asked subjects whether they believed them. We also asked if they thought the writer’s sources were qualified, and how they viewed the use of secret recordings and the influence of a third-party funding source (the Mother Jones story was supported by a grant from the Fund for Investigative Journalism). These questions were designed to gather information without tipping off readers to our research goal.

Baumann, the story’s author, believes that his methods may actually have reinforced the story’s credibility. “It’s not often that you get a recording like the recording that I got,” he says of the audio, which captured FBI agents making veiled threats.

We also asked big-picture questions on the overall accuracy, amount of bias, and the perceived message of the article, along with how well our subjects understood the events it depicted. Finally, we asked if participants noticed the publication brand, and whether it had any impact.

The data we collected offer no concrete reason for the credibility split between The New Yorker and BuzzFeed. Subjects’ comments imply that BuzzFeed’s reputation for light fare caused the division. “BuzzFeed copies stories from other sources so I had no reason to question the reliability,” wrote one subject. Said another: “I’ve never visited BuzzFeed as a site but I see they make lists that women really like and visit a lot. … The article/story did not strike me as something they would typically post but I enjoyed it nonetheless.”




BuzzFeed isn’t just a place for listicles. In February, it won a National Magazine Award, a top journalism honor and a confirmation that the site’s goofy quizzes don’t preclude great reporting. But our results suggest BuzzFeed must continue to sell people on its journalism chops. Some who read the story on BuzzFeed even said it changed their view of the outlet for the better. “When I first saw how long the story was I was pretty discouraged but it ended up being very interesting and not something I would expect to read on BuzzFeed,” one subject wrote.

Our data indicated that online news consumers were more likely than print readers to trust BuzzFeed. This aligns with earlier research, including a 2012 study on the credibility of Wikipedia, which suggested that trust in information is influenced not merely by the brand, but by the medium itself. The more someone trusts the internet, the more willing that person may be to trust a website like BuzzFeed.

As far back as 1998, researchers observed this tendency among politically interested online news consumers. Even so, those study subjects were skeptical of every article they read, regardless of the medium on which it was published. Digital news consumers may be a tough crowd, but all these findings insinuate that the more credible the internet becomes, the more credible websites like BuzzFeed will become.

Our results also imply that even digital publications born yesterday can impress readers when they publish serious journalism. Some people acknowledged having never heard of our invented publication, The Review. Yet its aesthetic and the article’s content worked in the outlet’s favor. “I did not recognize the publication but it looked reputable,” one participant wrote. “Regardless the story was very well written and researched!”

Like all academic studies, ours had imperfections. For one, no single media brand can represent an entire segment of the industry, and BuzzFeed and The New Yorker are exceptionally unique organizations. Another concern was that our readers didn’t come across this story organically, through social media or a website, as they would have outside of the research setting. Finally, most people don’t read stories to their conclusions, which was a requirement here.

At the end of the questionnaire, we told our subjects that the story had run in Mother Jones and provided a link to the original piece.

 

Although more of our subjects trusted The New Yorker than BuzzFeed, the digital native wasn’t far behind. Perhaps the biggest takeaway from our study isn’t that readers judged The New Yorker more credible, but that a 10-year-old website that began filling out its investigative unit less than three years ago came close to matching the clout of a 91-year-old magazine of indisputable prominence. That, along with the seemingly inexorable rise of digital news consumption, suggests that any lingering credibility gap between online and print brands is closing.

“Some news brands–typically those that have been around a long time–are often seen as main sources of news, whereas new players–even if they have a large reach–are thought of as secondary sources or ‘guilty pleasures,’” the Reuters Institute’s report notes. Yet our results suggest people take note of small details in long, narrative stories. Those signals help form their opinions of the story. Groups like The Trust Project at Santa Clara University are researching how news outlets can use story-level cues, such as highlighting the author’s credentials and sources, or how intensely an article was fact-checked, to boost their credibility among digital readers.

Craig Newmark, a CJR board member and The Trust Project’s founding funder, spoke for most Americans when he said last year, “I’m a news consumer–not in the business; not gonna tell anyone how to do their job–but I want to be able to read news that I can trust.”

In the evolving world of digital journalism, it’s up to each magazine–and each individual journalist–to win readers’ trust, one word at a time.

Danny Funt, Chava Gourarie, and Jack Murtha are CJR Delacorte Fellows.