Rule of thumb, journalists: If the “study” you’re writing about is based on a SurveyMonkey.com survey, it probably deserves a wee bit of skepticism.
But skepticism was almost entirely absent in the widespread coverage of a report by the International News Safety Institute and the International Women’s Media Foundation, which issued findings asserting that female journalists suffer an alarming amount of abuse on the job.
Slate reported that “Most Female Journalists Have Been Threatened, Assaulted, or Harassed at Work. Here’s Why We Don’t Talk About It.”
Huffington Post wrote, “Women Journalists Face Rampant Workplace Abuse, Sexual Harassment: Study.”
Poynter said, “For many women journalists, workplaces are dangerous, study says.”
Jezebel offered no skepticism and left out key details, while Salt Lake Tribune managing editor Lisa Carricaburu rewrote the press release.
Click through Slate’s piece, though, and what should have been a red flag comes out right away, in the lede:
This week, the International Women’s Media Foundation and the International News Safety Institute released the results of an online survey asking female journalists around the world to detail the abuse they’ve experienced on the job.
Ah, yes. The old online survey. Perhaps a bit more rigorous than CNBC’s insta “polls”, but still statistically dubious at best, and more likely useless.
The International News Safety Institute and the International Women’s Media Foundation posted this survey on SurveyMonkey back in August, and it has been floating around Twitter since. Anyone can take it as long as you say you submit your full name—or at least a full name. In fact, the survey is still up and accepting responses.
A scientific survey would seek to get a truly random sample of journalists who are women, which would probably require finding them, rather than them finding you. But that’s a much more costly and time-consuming process.
It’s unclear how the survey’s authors would be able to tell whether the respondents are who they say they are, are journalists, or, for that matter, female (in fact, nearly a hundred respondents said they were male, though the report didn’t count their answers). The report says that “The questionnaire was designed and analysed on behalf of INSI by Samantha Scott, AFBPS, Chartered Psychologist and Registered Occupational Psychologist.” Scott says on her website that she “has extensive experience in selecting, retaining and developing talent including the design, delivery and management of leadership assessment and development centres, leadership development coaching and psychometric feedback,” but there’s nothing about polling (I have questions out to Scott and to INSI and will update if I hear back).
Beyond the whole don’t-really-know-who-answered-the-survey problem, there is the high likelihood of selection bias via self-selection. Women who are journalists who have suffered abuse on the job may be more likely to take time to take the survey. Megan McArdle, the lone skeptical voice I’ve seen on this report, has a great take on that. She also notes that the questions are poorly worded.
Contrast that with Bustle, which shows what not to do when you notice serious problems with a report’s methodology. It stuffed them in the last paragraph of a breathless 16-paragraph piece, where it also stuffed mention that this was a global survey, not a US one.
I also wonder about the finding that 22 percent of female journalists have had their phones tapped—or at least would know they’ve had their phones tapped. That seems awfully high.
And it’s worth noting, as no one else has, that the survey found that just 63 percent of the abuses came from males (we’ll assume 37 percent came from fellow women), which, if nothing else, complicates the frames offered by most of the coverage. How many of these threats came because the journalists were women or because the women were journalists? Verbal threats, or at least angry words, are rather an occupational hazard for all reporters. To really get a fix on that, you’d have to survey men too and compare the results.
Bottom line: if you want to report on this, and if you want to call it a “study,” you have to note that it’s a completely unscientific one and almost certainly not accurate. In other words, best not to hang a story on it at all.
And that’s unfortunate. Because no doubt there is a story to be told. Anecdotal reports of women journalists being abused verbally or physically on the job are all too plentiful—I’ve seen it firsthand.
I’d like to see a study on just how common gender-based abuse and threats are—in the US and globally—including how common newsroom harassment is.
Until someone does the real work, though, let’s bring more skepticism to the statistically challenged kind.
Ryan Chittum is a former Wall Street Journal reporter, and deputy editor of The Audit, CJR’s business section. If you see notable business journalism, give him a heads-up at rc2538@columbia.edu. Follow him on Twitter at @ryanchittum.