RALEIGH, NC–On a Monday afternoon in March, members of a North Carolina nonprofit called Equality NC hunkered down for a two-hour stint in the comment section of a story on the Raleigh News & Observer‘s website.
It was an op-ed that Equality NC director Chris Sgro had written about tax issues facing same-sex couples in the state, which in 2012 added an amendment to its constitution prohibiting the recognition of same-sex marriages or civil unions. More than 100 commenters weighed in over the two-hour span, but almost all of them engaged in civil discourse, the opposite of what netizens have come to expect in response to divisive issues.
Comment facilitation, in which a reporter, editor, or third party moderates the comment section and interacts with readers, is a popular method for many news outlets trying to battle trolls and libelists. But it’s a thankless job, and one that takes resources that many publications just don’t have.
Now a nonprofit in North Carolina and researchers in Michigan have teamed up to see if a new kind of comment analysis technology can make things a little easier, and more effective.
The March experiment conducted on the N&O story was the first of may well be many Q&As conducted by SpeakUp NC, a project of the Raleigh-based AJ Fletcher Foundation that aims to elevate comment section discourse. And results of SpeakUp’s first few experiments served as a testing ground for a new tool being developed by researchers at Michigan State University’s Writing in Digital Environments program that analyzes and sorts language in comment sections to get a better idea of what kind of facilitation leads to productive engagement.
“We want to understand the problem, and [news organizations] want to solve the problem,” said Bill Hart-Davidson, one of the MSU researchers. “That’s what makes us good partners.”
The research started when WIDE, as the MSU program is known, received several years of funding to work with science museums and educators around the country to improve the impact of their online presence. WIDE has developed a number of tools and training methods for museum employees to interact with the public online more effectively, and when the group’s work at Durham’s Museum of Life and Science caught the eye of SpeakUp director Shannon Ritchie, it became clear that the tools could serve a wider purpose.
WIDE’s algorithm, which shows promise for improving news site comment environments, is still being tweaked and is expected to be in beta form later this summer. It works by gathering raw data in the form of words that make up comments (Hart-Davidson says names and usernames are of less interest and aren’t stored) and pulls out phrases that have been “pre-coded” for their usefulness in directing conversation. The result is a collection of feedback showing which phrases seemed to draw productive comments and which brought in the trolls.
“We are less interested in the results that any one voice has than a kind of group or ecosystem-like approach, where we can say what conditions have to be in place for there to be a culture of learning to develop and be sustained,” Hart-Davidson said.
After the N&O experiment, and another one at the Fayetteville Observer, Hart-Davidson and his colleagues used a preliminary form of their approach to gauge Equality NC’s success.
“They analyzed the thread and determined what the best tactics were within the conversation, and showed how the facilitator within the nonprofit organization enhanced the conversation and made it valuable,” Ritchie said. “We are able to compare it to similar op-ed comments pertaining to LGBT issues and marriage to show that this was more valuable to public discussion.”
Hart-Davidson pointed out examples of “guiding,” “staging,” and “bridging,” buzzwords for different rhetorical styles of conversation direction, showing SpeakUp how Equality NC and others in the discussion moved the conversation forward, sometimes without even realizing it. Examples included, “Let me see if I’m understanding this correctly,” “Is there any reason for this other than opposition to marriage equality?” and “To clear up a few misconceptions…”
Being able to determine what is helpful and what isn’t in comment section facilitation provides a level of understanding and control that the N&O and papers like it see as a potential boon to readers and reporters alike. Facilitated discussions have a lot of promise, said Eric Frederick, managing editor of online news at the N&O, but it’s not clear quite yet just how useful they are, or how many cash-strapped news organizations have the means to devote resources to regular comment facilitation. Combining WIDE’s new technology with a trusted moderator could be the answer news organizations are looking for.
The next generation of the WIDE researchers’ work is to create a more real-time “dashboard” that would follow a facilitation session and provide more immediate feedback, giving the moderator results while a comment conversation is still going on, over a period of days or weeks. This way, facilitators could have the chance to potentially alter their approach to achieve a more productive result.
Hart-Davidson said that as his team completes the first working prototype of the dashboard, they are open to working with journalists and editors who see value in testing it on their own readers. The N&O and other local news organizations will continue to serve as platforms where SpeakUp’s experiments play out, and the hope is that something will stick, shining a little more light on the educational potential of what many journalists still see as a dark and dangerous place.
Kaitlin Ugolik is a journalist based in New York who covers the law, health, technology, and the media.