Industry conversation over the best ways to quantify reader engagement on news sites may be reaching fever pitch, with a possible shift afoot from clicks and pageviews to more complex measures of time and attention spent online.
News organizations are interested, presumably, in gathering metrics to see whether their stories are reaching intended audiences, as well as who is reading what, when, and how, information that could help shape editorial decision-making, story design, and ad sales.
Dr. Talia Stroud works with news organizations to explore how they can use these metrics to move beyond the numbers and measure understanding. That is, what do readers take away from an article after reading it? Do they understand the issue and the context? How can online news sites and stories be best designed to support the interests of citizens, as opposed to publishers or advertisers?
Stroud, the director of the Engaging News Project at the University of Texas, seeks to provide news organizations with academic, research-based numbers and techniques for engaging audiences. Below is an interview with Stroud that has been condensed and edited for clarity. (Note: The Engaging News Project and CJR’s United States Project both receive support from the Democracy Fund.)
Could you give me some background about what ENP is and how it began?
Engaging News Project is a foundation-funded entity at the University of Texas at Austin.
The overall idea is one that I started on around about two years ago. My reasons were two-fold: One is that I had written a book looking at the tendency of people to look at news matching their political beliefs and the democratic consequences of that just worried me so much. And I had also played a very small role in the FCC’s Information Needs of Communities report, and I found that report to be so influential, showing how problematic it is when news organizations aren’t appropriately funded and how much that can hurt communities.
So on the basis of those two experiences, I wanted to do something that was more solutions-oriented. Engaging News Project was my idea of how can we help the news media and reach citizens using the media as a way to do that, to try to improve democracy to try bring academic research in line with the real needs of communities, the news media, and democracy more generally.
So who is your target audience?
I would say we have three target audiences. The first and the main one is reaching out to news media organizations to let them know about what the research is, what we found, and to work with them to learn what their problems are and how we from the academy can help them with those issues. The second, kind of subsidiary audience of that is the actual news media audience. How can we put things in the news media that then will help citizens to learn more about the world in which they live, about what’s happening in democracy, just engage them? And then the third audience is an academic audience, to showcase how academic research can be used in ways that have some practical implications.
How does your work compare with the work of companies like Chartbeat or media organizations like Upworthy, who have been developing methods for measuring attention and engagement? All that seems to be in your orbit. How is your work similar or different?
I think one thing our project really adds to the work of companies like Chartbeat is that we’re focused on outcomes not only associated with pageviews or time-on-page, but also outcomes like, ‘Did people learn?’ I think that really sets us apart. We’re actually doing experiments where afterwards we are giving people a quiz to find out, ‘Did they learn more from option A or option B?’ I think that’s a really important thing: finding out in what ways we should present information–what should we include in the information that helps people learn–while still thinking about some of the business outcomes that a Chartbeat, for instance, focuses on.
I read one of the reports that came out of a workshop you did that mentioned how news producers are interested in measuring reader understanding as much as they are about measuring how much time people are spending on their site. What are you working on towards that?
Right now, we have one project we’re working on finishing where we changed the way that a homepage looks. So instead of a homepage that’s more traditional and resembles essentially the layout of a newspaper–the prominent story in the upper corner with the photo and then ordering stories in different ways around the page–versus one that’s more streamlined, one that has one photo per story and [is] more streamlined, ordered in rows and columns. We did a study contrasting those two, and we are looking at which one of those two forms will help people recall information and how that affects time-on-page.
We’ve also done some work on polls and quizzes. There’s good evidence from the research we’ve done that quizzes really help people learn and process information. We’ve actually developed a quiz tool that news organizations can use for free on our site.
And are news organizations using it?
Yes, a couple in Indiana have started using it, which is exciting for us to see.
What do think the incentive is for news organizations to measure understanding? You sit at a pretty interesting intersection between commercial viability and helping people understand news for the sake of democracy. What would drive a news organization to actually want to measure understanding when, at the end of the day, what matters most for them is whether or not they are financially sustainable?
News organizations have to be in the business of both these democratic and business objectives. If news organizations forego democratic obligations, they should be promoting a puppy slideshow everyday. They’re no longer the news. [Some outlets] recognize there is a commercial component to what they do, but if you get it wrong on the news side, then you’re competing against all sorts of entertainment-oriented media sites. So I think news organizations have a vested interested in measuring that they are doing the thing that makes them the money.
So how does the commercial viability side of your mission work? Are you providing these tools and research to news organizations for free?
We are providing the quiz tool for free, and we are providing our research for free. And within the studies we are doing, we are also looking at commercial outcomes. So, for instance, how much time do people spend on a page when it has a tool on it? For the homepage study, we looked at how many unique pages a person clicked on, on one version versus the other.
We’ve also looked at: How much do you like or enjoy this feature versus another feature? So we’re looking at the types of tools or practices that bring people back, that make them interested. I think by focusing on a) the digital metrics, like pageviews, and time-on-site and b) softer, qualitative metrics like, ‘do you like this,’ that combination really speaks to the commercial side of what news organizations in the business are doing online.
One of the main goals of the news literacy movement is to teach people to critically consume the news so that they can be empowered and then engage in civic affairs based on that understanding. It seems like your research could also inform news literacy educators.
I do think we have components of news literacy in the research we are doing. For instance, part of the impetus for developing the quiz tool was an analysis we did of over 150 news websites where we were looking at what features were contained on the sites. And a trend that I find to be incredibly disturbing is how many news organizations’ websites include a poll on their site, like ‘Tell us which candidate you prefer,’ and then anyone who visits the site can participate in it.
From a news literacy perspective, that makes me so nervous, because those results should not be trusted by anyone. In the worst-case scenario, it’s misinforming the public, because it’s not a representative sample; it could be four people, it could be an interest group that got their constituents to go and vote in the poll. So something that we are telling news organizations is: Why not consider using a quiz, which is in your skillset and could provide good information, rather than a poll on your site that contains bad information in the worst-case scenario?
I see. It’s almost as if you are indirectly representing the news consumer by teaching the news producers these frameworks.
I’m flattered you say that. Before I started this project, I was reading a couple of books about how political campaigns are influencing citizens, and trying to learn all the tricks of how you get your partisan base out to vote and how you frame issues to trigger their mindset to support your party. And I was thinking of how disturbing it was that no one was representing the citizen. Where is the person in this who is not in it for partisanship? While I don’t think we completely meet that goal, that’s definitely a component of this project.
And yet it seems that the new organizations who have the money to experiment with modes of storytelling that are more explanatory or try to address these questions, Vox.com, for instance, tend to have very broad, general audiences, who might not be as inclined to engage civically as the audiences of local or regional news sites.
The marketplace for news and the organizations who can do these experiments and find out what works–that’s fantastic. But some of these local news outlets, they literally do not have the time, and we aren’t thinking about how to distribute resources across these different types of news organizations. I’m tremendously worried if we have a news media environment that doesn’t include those folks who are minding the statehouse. We really need that resource, so the more that we can find out what works at both a national level and a local level, the better off we are.
Funding for this coverage is provided by the Robert R. McCormick Foundation.
Jihii Jolly is a freelance journalist and video producer in New York City