In a provocative November 4 essay on the Norwegian Broadcasting Corporation’s tech-news site, a Samsung Galaxy devotee baffled over and mocked the frenzy greeting the release of the iPhone X. The debate the editors braced for was one about the merits of the two brands. The one that erupted in the comments section, though, was a bizarrely contentious, off-topic scrum over the essayist’s decision to capitalize the “i” in iPhone.
After a few hours, the site’s overlords activated a unique speedbump for those who would hijack the discussion. Commenters thereafter were greeted with three multiple-choice questions and a notice: “We are concerned with the quality of our comments. Therefore, we want to make sure that everyone who commented has actually read the story. Answer the questions below to unlock the comment field.” The shouting over the “i” stopped abruptly, and the conversation morphed into the more predictable, more palatable sniping over the virtues and evils of Apple products.
ICYMI: The New York Times begins 2018 on a sour note
NRK, the Norwegian equivalent of the BBC, first began forcing would-be commenters to take such quizzes on select stories back in February after one of its journalists, Ståle Grut, had the idea in the shower before his commute. Grut works for NRKBeta, a subsite that both focuses its coverage on technology and offers its journalists a live space to try out media innovations.
Grut’s big brainstorm, aimed at improving the quality of the comments on the site, is to require visitors to prove they have read and understood a story with a quiz written by the story’s author. At the time, an NRKBeta report about how sexual assault suspects in a high-profile local case had scraped social media for images and information about young girls had prompted a wild and somewhat obscene discussion. The main NRK site, like many large media organizations, had already eliminated its comments section because they’d become unpleasant, counterproductive, and unwieldy.
“The comments for that story were just horrible—so much foul language and people lashing out,” says Grut, 27, who is on leave from NRKBeta pursuing a master’s degree in new media and digital culture at the University of Amsterdam. “We wanted to create a bump in the road to make people think a bit before they start ranting away. It’s a way to nudge people to discuss the core arguments of the article instead of piling up on other stuff that was off topic.”
Ten months on, the quizzes themselves are surprisingly popular, and the site hasn’t suffered as many off-topic flame wars. NRKBeta has gated the comments of 30 stories since February—chosen when someone suspects the discussion could turn ugly—and Grut said visitors have been able to post about 1,200 comments by passing the tests. There have been about 5,500 successful attempts to pass the quizzes and 11,000 unsuccessful attempts. (The iPhone story was a rare example of NRKBeta adding the quiz after the story posted to rein in a conversation that had strayed off topic.)
“Our theory is that people use this quiz not just as a way to gain access to the comments section but also to see if they remember stuff from the article,” says Grut, who is scheduled to speak about the effort at South By Southwest in March in Austin, Texas. “It’s like a small game. They don’t necessarily want to comment.”
TRENDING: Worst headlines bloopers of 2017
The failure rate—about 66 percent—is a tricky number to explain because, as with any start-up experiment, there have been glitches and wrong turns, Grut says. About 1,600 of the 11,000 were the result of an incident in which Grut forgot to tell the system which answer was correct, preventing everyone from commenting on a piece about Facebook adding a rainbow emoji to its “like” arsenal for LGBT Pride month.
When NRKBeta posted a story in English in March announcing that the code for the quiz module plug-in is available free on Github, the answer to one of the questions in the comments quiz could only be found by zooming in on an image accompanying the story. That, Grut said, was too obscure; nobody was able to comment.
It also reflects the editors’ efforts to find the sweet spot between testing for comprehension and asking excessively difficult or picayune questions. A more successful effort involving a complex topic came on piece posted in September about how Russia may be hacking shipboard computers on the Black Sea. Would-be commenters were asked what GPS spoofing is, how many ships had been affected by the attack described in the piece, and why it is easy to carry out GPS attacks; the story has drawn 101 comments to date.
Another element that may have thrown off the quiz results from time to time has been media attention for the experiment itself, Grut says. Everyone from CNN to the BBC to France24 covered the idea when it first launched, prompting a surge of traffic from non-Norwegian-speaking curiosity seekers abroad.
The concept has drawn some sharp criticism, notably from Andrew Losowsky of the Coral Project, a nonprofit collaborative effort of Mozilla, The Washington Post, The New York Times, and the Knight Foundation that offers free comments software it says makes it easier to manage.
“Forcing readers through a quiz-style obstacle course might make them think an organization is trying to keep them at arm’s length rather than engaging with them more closely,” write Losowsky, a former Knight Fellow at Stanford, wrote in a blog post. “And even if making users pass a memory test does force the angriest ranters to calm down, it doesn’t address the biggest problem with comment sections: There is a general lack of newsroom strategy, intention, and proactive engagement within the space.”
ICYMI: Too many people are making the same typo
Gizmodo Media CEO Raju Narisetti was equally circumspect, tweeting a message that has since disappeared: “Because creating MORE friction in engaging increasingly time-poor & promiscuous digital audiences, who can go anywhere else, is very good.”
Management of the comments sections of news sites have long vexed publishers, many of which—from Reuters to NPR to, most recently, Crain’s Chicago Business—have shut them down altogether. In a November 20 message to readers, the Crain’s editors said they’d come to the same conclusion as many other sites: “Simply put, we do not have the personnel to manage this commentary, to keep it civil and fair and to halt the back and forth before it devolves into invective, name-calling and, in too many cases, outright hate speech. We’d rather not play host to these often anonymous commenters. They drive out more civil readers and potential commenters. They sully our content, our brand, and our sponsors.”
In the process, though, those sites are ceding the conversation and the ability to build relationships with readers to Facebook and Twitter. Efforts like NRKBeta’s to keep the conversation on site are about retaining control over those discussions.
Grut isn’t claiming the idea is perfect, nor does he think it’s realistic for massive media outlets that post hundreds of stories every day. At NRKBeta, the story author is responsible for writing the quiz and is expected to participate in the comments section, duties many journalists would chafe at. All of that, however, helps improve the conversation, he says, because “it shows people there’s an adult in the room. We are always trying to show people that they can contribute but there are some rules here.”
In August, before Grut took his sabbatical from NRKBeta to enroll in Amsterdam, he wrote his own piece in English reflecting on the experiment’s first half-year. He dubbed it “the world famous quiz” and marveled at the unintended outcome: “Being tested on how much they remember from the article seem to be the most popular use of our quiz.”
Appropriately, would-be commenters were forced to answer questions before they responded. Of the 15 who did, most were complimentary. “This really is a brilliant idea,” wrote Julie Jensen of Chicago, who said she learned of the experiment from a podcast. “At least it is thinking outside the box when trying to combat ranting, and people commenting after only reading the headlines.”
Yet even there it was clear it wasn’t a cure-all. Two of the comments were random excerpts from the story itself, and one, in Norwegian, was quite a bit off topic. “Det er fullt av ål i luftputebåten min!,” Thor Hushovd wrote.
English translation: “My hovercraft is full of eels!”
ICYMI: Explosive BuzzFeed scoop raises eyebrows
Steve Friess is a freelance journalist based in Ann Arbor and a journalism instructor at Michigan State University. Follow him at @SteveFriess.