How a local paper built a tool to measure impact

How do you measure impact?

Many publishers are looking to outside sources for funding through foundations, investors, and subscriptions. In addition to analyzing interest in their work through pageviews, completion rates, stickiness, and other metrics, companies and newsrooms need to show how much of a difference is being made by their work across channels.

The Journal Media News Group, a news organization located in the northern suburbs of New York City, created a way to measure many of their successes through Impact Tracker, a custom online tool developed by its staff that could soon expand from three Gannett newsrooms to 16 more in the USA Today network next year.

ICYMI: NY journalist handcuffed to railing over his head

In the online interface, reporters choose from 12 kinds of impact for their work—such as policy change, major citation, and government investigation—, add specific details, and track how their stories change over time. Editors can quickly see how their reporters are doing, as well as pull out information for columns, award applications, and performance reviews.

 

Digital Director Anjanette Delgado started the project in late 2015 with the belief that investigative journalism needed to be measured by more than just page views. Executive Editor Traci Bauer agreed. “If we become a slave to those metrics at the expense of not knowing about impact, we’re going to wake up one day and be very regretful as an industry,” Bauer tells CJR. “When I said, [impact is] impossible to measure, Anjanette had this look on her face like, maybe that’s not so impossible.”

In December, the first version of the Impact Tracker started as a Google Form, a free online survey and information collection tool.“We started out that way because it had a low barrier to entry and you can experiment,” says Delgado, who also did extensive research on impact tracking and what was already being done at nonprofits and organizations like ProPublica and The Center for Investigative Reporting.

The newsroom’s seven-person investigative team was the test group for this first version of Impact Tracker. “We wanted reporters to do this because they can connect the dots and say ‘I know this happened,’” Delgado said.

ICYMI: “She identified herself as a reporter. He then walked behind her and punched her in the side of the head”

For Investigative Editor Frank Scandale, the Impact Tracker gave him and his Watchdog reporters a much easier way to organize the results of their work far beyond pageviews and other kinds of traffic. Scandale was part of the team at the Denver Post who won the Pulitzer Prize for breaking news for the paper’s coverage of the Columbine High massacre. He says contests like the Pulitzer Prize for Public Service have always focused on impact but other deeply reported stories that weren’t nominated for awards mattered, too.

“We may not win an award but the [Impact Tracker] tool shows you the value of investigative journalism,” he tells CJR from his cubicle filled with stacks of old newspapers and walls covered in pages of his team’s cover stories. “It validates what we do. It’s showing what we’re doing in black and white and it doesn’t get pushed to the bottom of some email.”

ICYMI: The surprise 2017 Pulitzer winner

After a year and a half with the Google Form, Delgado had enough experience, information, and feedback from the Watchdog team to receive management support from Bauer to bring the project to its next stage of development. “There wasn’t a question when it came to asking for additional resources,” Delgado says. News app developer Avram Billig was brought on to help build the second version of the tool—a custom, easily searchable, scalable online application with more specific ways of tracking impact beyond investigative work. “Ease of use was really important to us so that reporters wouldn’t feel like it was burdensome,” Delgado says. The new version was introduced to the newsroom in the spring of this year.

“When we rolled it out to the rest of the staff, Anjanette and I said, ‘Don’t junk it up’ because you feel like you have to be on the Impact Tracker,” Bauer tells CJR. “We’re pretty strict about what goes in there. Was legislation introduced? Did someone lose their job? Was an investigation started?”

Some of Bauer’s favorite examples of impact include a Purple Heart medal found on the side of the road being reconnected with family members and a popular German restaurant reopening after its initial closure. “It doesn’t always have to be someone getting fired or going to jail, although those are pretty cool, too,” she says, laughing. “It can be softer than that.”

 

Another, more recent, example came from a report on a local Christmas tree farm. Reporter Heather Clark told Delago that on the farm’s first day of business, sales had doubled and customers were coming in from a wider area. “The owner told [Clark], ‘The only thing different from last year is you wrote about us.’”

It didn’t take long for other Gannett papers to become interested after Bauer and Scandale did a presentation on impact at a editors meeting in the summer of 2016. But Bauer and Delgado wanted to wait for the updated version before introducing it to other newsrooms in the company this year. “Now it’s so much more sortable and searchable and scalable,” Bauer says.

Version 2.0 of the tracker automatically enters information from reporters into an easily searchable database. The application also allows easy export of raw text into the newsroom’s content management system for other articles or columns, as well as for speeches, audience analysis, and even talent recruitment.

The current expansion focus for the Impact Tracker is with smaller newsrooms like the Poughkeepsie Journal, where Bauer and Delgado also serve as executive editor and digital director, and TCPalm.com. “They can help us test it, but they can also help us with the rollout and the [training of other Gannett newsrooms],” Bauer says.

ICYMI: Gannett and the last great local hope

While adoption of the tracker is voluntary for Gannett newsrooms, Delgado says it’s critical to get management on board from the start. “If you have a newsroom where the editors don’t believe in it, don’t buy in, don’t make it a priority, then reporters forget to put their story in,” she says. “Then you end up with three or four months where nothing has happened even though you know something has.”

Meetings, training and observation of the initial impact tracking are all done by Delgado’s team remotely. And while there are currently 16 sites waiting to get access, Delgado says this is still a fraction of USA Today’s total network. “If we bring everyone in, we’re talking 100 plus,” she says. “We built it to scale and to be able to handle 109 sites. But we just want to bring people in waves instead of all at once.”

When it comes to writing stories, Bauer notes impact tracking is one of a range of measurement tools for her staff, including unique visitors or time spent on site. “We’re not going to chase the metric of the week in this newsroom,” Bauer says. “And it’s not all about impact because if we ignore the metrics and just go to impact, we have the danger of losing our audience. We have to measure all of them as a formula and put all of them together to make strategic decisions.”

Beyond its editorial function, the Impact Tracker is also changing how The Journal News markets subscriptions. Local consumer sales and marketing manager Kasia Abrams says her work will highlight results from notable stories she’s found in the reporting tool.

“I think there’s no better way to emphasize the importance of being a subscriber to a local newspaper than showing their impact,” she says. “Often in sales it’s just results oriented and you’re selling the product because of the discount. We’re selling a product that really makes a difference. We’re part of something bigger. This is why you should subscribe.”

“We would rather sell the impact on the community instead of saying, ‘Buy the paper you’ll get an extra five dollars in coupons,’” Delgado says with a laugh.

Correction: This story has been updated to clarify how the Impact Tracker started.

ICYMI: When metrics drive newsroom culture

Karen K. Ho is a freelance business, culture and media reporter, based in New York. She is also a former Delacorte Fellow at CJR. Follow her on Twitter @karenkho.