When metrics drive newsroom culture

Sign up for The Media Today, CJR’s daily newsletter.

As analytics have entered newsrooms the digital world has divided into two camps. There are those that think attention paid to metrics creates a world where Kim Kardashian beats out more important subjects, like the Syrian conflict. Advocates of data-driven decision making, however, argue that metrics shift the power balance to a more democratic system where readers matter more than editors.

But how do these decisions actually play out in newsroom culture? How do metrics, in practice, influence decision making? That’s the question that Caitlin Petre, a doctoral student in sociology at New York University, set out to answer through ethnographic research.

Over the course of a year, Petre divided her time between the offices of Chartbeat, leading analytics company, and two newsrooms that sit on opposite ends of the metrics debate: Gawker Media, where the numbers dictate editorial value, and The New York Times, pre-“Innovation Report” a place that was just beginning to define their analytics strategy. Through dozens of interviews with employees and analysis of meetings, Petre’s report “Traffic Factories: Metrics at Chartbeat, Gawker Media, and The New York Times,” just released by the Tow Center for Digital Journalism, paints a powerful picture of the substantial effect that numbers have on newsrooms. In the first study to look at the cultural impact of measurability in newsrooms Petre documents the digital revolution in lumbering progress, showing that all newsrooms, regardless of their digital finesse, are still learning the effects of policy as they craft it.

“Metrics inspire a range of strong feelings in journalists, such as excitement, anxiety, self-doubt, triumph, competition, and demoralization,” Petre writes. Depending on how they’re implemented, metrics can have vastly divergent effects on editorial culture. But regardless of how newsrooms shield their staff, Petre found, the emotional effect remains.

At the time of Petre’s residency at Gawker, metrics dominated company culture, starting with a giant monitor called the Big Board that displays Chartbeat analytics that hung on the newsroom wall. But analytics also factored deeply into the company’s evaluation of its employees- a rating system (dubbed eCPM) calculates how many dollars in salary a writer earns per 1,000 visitors they bring to the site. (Falling below $20 on this rating is grounds for dismissal.)

As a result, Petre observed a team of writers fixated on attracting huge numbers at-all-cost, even to their emotional well being. Staffers regularly used drug terminology (“addicted,” “fix”) to describe their relationship to the site; several told Petre they vetted their relationship to analytics with a therapist.

More tellingly, staffers said their willingness to experiment was dampened by pressure to feed the numbers. As one editor told Petre, sticking to stories that have worked in the past, like “cute things that kids did,” or “unhinged letters,” from sorority girls or “douchebags,” is a surer method of attracting traffic than exploring a new idea. And writers felt the need for quantity always hanging over them. Since traffic numbers varied wildly per post, writers pursued an approach similar to that for playing the lottery – to increase your odds, buy lots of tickets. None of this rewards creativity, or the thoughtful reporting and think pieces that Gawker’s sites have become known for. With traffic as the complete arbiter of merit, reporters responded rationally.

Reporters at the Times don’t have access to metrics, a stance copied by a number of digital native sites like Vox and the Verge. “If you go down that road, then you end up writing a lot about, you know, Angelina Jolie or whatever,” a Times staffer told Petre, a sentiment echoed throughout the newsroom. (Metrics, by default, equals Kardasian stories.)

Withholding the metrics from staffers also served to underscore the hierarchical status quo, with editors as the arbiters of what is news-worthy. “If editors alone had access to metrics, they alone could control the way in which the data was interpreted and mobilized,” writes Petre. And, she observed, editors spoke about metrics after-the-fact—when they justified a decision—rather than factoring them in during the decision making process. Yet none of this prevented reporters from gut-checking the analytics themselves, scouring the site’s “most-emailed” and “most-viewed” displays, unsure how seriously to take inclusion (or exclusion) from those daily round-ups. These lists didn’t give writers the blunt ranking that Gawker writers were encouraged to access with every new article, but it didn’t prevent Times writers from being interested in how their work stacked-up numerically.

This emotional reaction to analytics, Petre found, is something that had been considered—even capitalized on—by Chartbeat staffers. “It’s not the identity of the number, it’s the feeling that the number produces,” one Chartbeat employee told Petre, a concept reflected in their product. “The dashboard is designed to communicate deference to journalistic judgment, cushion the blow of low traffic, and provide opportunities for celebration in newsrooms,” writes Petre.

That’s why concurrents—the number of readers on a site at a given time—operate like a “candy metric,” causing a dial to swing toward the positive zone as the number of visitors goes up. Editors and reporters, according to Petre, consistently associated Chartbeat with “concurrents” rather than “vegetable metrics,” like visitor frequency or the “engaged time” metric that Chartbeat has been working to introduce to advertisers. Those metrics “contribute to its prestige and to clients’ sense that Chartbeat “gets it,” writes Petre, but in her opinion did not feed into editors’ own assessments of their sites.

Yet, much has happened since Petre’s investigation. The Times, after the launch of its innovation report, has hired a swath of digital native editors and an audience engagement team to answer the question of how analytics should factor into editorial decisions. On the other side, Gawker, driven by a failure to compete with Buzzfeed’s skyrocketing numbers, has flipped the other way, introducing a new system where editors, rather than numbers, dictate employee compensation. “These recent developments suggest that Gawker and The Times, once polar opposites in their orientations toward metrics, are moving closer to one another,” says Petre.

After all, metrics alone don’t create a newsroom culture, it’s the way they’re interpreted. Petre, for her part, recommends newsrooms take time “for reflective, deliberate thinking removed from daily production pressures about how best to use analytics.” Of course, this is a demand that runs contrary to the relentless, constantly innovating pace of the Internet. But if news organizations as divergent as The Times and Gawker can come to a similar meeting place, it stands that others can as well.

Has America ever needed a media defender more than now? Help us by joining CJR today.

Alexis Sobel Fitts is a senior writer at CJR. Follow her on Twitter at @fittsofalexis.