In the past, I’ve seen respected, tenured professors retreat into their warrens when faced with half this onslaught but, surprisingly, Karpinski—while understandably miffed—is philosophical about the experience. Her parents, she said, raised her to be resolute, and the episode has since netted kudos from faculty and more invitations to publish—reasonably rare positive results for a grad student.
In the end, the frenzy to be first with the news helped the media misinform the public and betrayed the essence of the research in question.
Most science reporters, and researchers, know the consequences of pushing the data too far. It’s a good lesson for other journalists to learn as well.
- 1
- 2
Bravo for this well written piece. And to the reporters from the British press and the twits from Standford, you should have 30 days of pooper scooper duty. Of course they didn't violate the embargo! Because they made things up. I am a colleague of Karpinski and her attention to detail and high level of ethics should be a beacon to these dolts (Stanford University and bad press) who manipulate and twist proper, exploratory research into a chance to further their (sorry) careers using blatant fabrications. I know about seven prisons in Ohio that have a Goose poop problem, to those of you at Stanford and the British press that practiced yellow journalism; plastic poop bags all around and 30 days to get the things clean so the prisoners don't have to dodge poop. Once this duty (no pun intended) is complete, these few opportunists will be very familiar with the stench that we know exudes off their bodies.
Posted by muckraker on Sat 9 May 2009 at 08:50 AM
sorry, no, you did not do proper reporting.
Karpinski used a sample of convenience, which is never, ever adequate.
Even if you do exploratory research, convenience samples cannot tell you anything beyond the fact that people understand your questions.
With this sample, Karpinski could not have found a correlation that is meaningful beyond the about 100 people she interviewed. But that doesn't mean anything at all. And indeed, Karpinski should have been more careful to draw attention to that.
Her snarky reply to the well written article by Pasek et al. - three researchers with quite a bit of experience in the field and actually decent data - is going to come back and hound her - not because she shouldn't be snarky, but because she's snarky and dead wrong (e.g. with respect to variable coding, understanding sampling, understanding regression).
By exposing her, you did not do her a favor..
Posted by Sebastian on Mon 11 May 2009 at 12:30 AM
Sorry, Sebastian, but I think your objections are uncalled for in the case of a student doing a trial study. If she were doing peer reviewed research, she probably would have approached the sample process differently. But considering that many if not most studies performed by graduate students is done by this method she is working within acceptable parameters for her level. Graduate students do not have the funding or access that established researchers have to use a more involved sampling technique. Besides, convenience sampling is a legitimate approach when the investigator is careful in what controls are used to balance any non-random factors, if there's no reason to suspect that a different group of subjects would act differently, or if the investigator could not even ask the question without it.
Blame the sloppy journalists not the student for the brouhaha. I am sure that she did not anticipate that her trial study would get international attention, but even if she thought so what could she have done differently given her student status and her desire to study something that is of great interest to students across the country. Her questioning attitude and her willingness to take on established researchers shows pluck that I doubt will ever be detrimental to her career, especially in the social sciences where differences of opinion about statistical methods and sampling techniques are quite common no matter how good one is.
Keep your chin up Ms. Karpinski!
Posted by Robert Killoren on Mon 11 May 2009 at 11:39 AM
Mr. Holland's spirited defense of the Karpinski paper sounds reasonable until one reads the paper itself. If Ms. Karpinski were sufficiently aware of the limitations of her study, she would not have drawn the following implication for college administrators:
"...the suggested negative consequences of (Facebook) use can alert administrators to find ways to limit access (e.g., limiting wireless access in lecture halls), and perhaps encourage time-management skills resulting in better academic performance."
Negative consequences sound like a causal statement to me. I don't know what it means to the folks at OSU's publicity department, but this type of conclusion based on this very exploratory study seems to go well beyond what is justified by the data.
Posted by DR on Mon 11 May 2009 at 05:43 PM
The paper that you are referring to was a draft of a final paper from one of the author's courses. The paper was never meant for public consumption or critique. I am a reporter for one of the news stations that interviewed Karpinski, and she gave that paper to the press office so that they could have a little background information. We got a hold of the paper, but knew that it was a draft and should not be distributed to others. She told us exactly that, it was a draft final paper that she used to make the poster - POSTER PRESENTATION! Along with negative consequences, you will also note that she took care to acknowledge POSITIVE consequences stressing the benefits of communication, and how university personnel should consider adopting this communication tool. How very biased for you to only point out the negative. Regardless, you are critiquing an exploratory study - pilot work - and using a draft of a final paper of a course to do so. That's ridiculous. Treat it as it should be - a poster presentation.
Posted by Sarah on Mon 11 May 2009 at 09:23 PM
This isn't the first time in recent weeks that The Times has blown something out of proportion and created a mess. They also reported that cyberspace was "filling up," to the conclusion that the Internet would be worthless in 3 years, based on a study that wasn't nearly as sensational.
They also wrote that the research was new and due to be released this year, when in fact the study was from 2008. I found this out when checking with the researchers for my own story.
That said, press releases also tend to spin these studies out of proportion, so the universities aren't entirely blameless.
Posted by Jared Newman on Tue 12 May 2009 at 10:31 AM
Regarding Jared's last statement, it would be more accurate to say that "SOME press releases tend to spin . . . out of proportion" -- there is no evidence that we did in this case, nor that we do in any case. Some of us are supremely frustrated to be painted by such broad brushes. The good thing about reporting on research is that there is actual data, facts, to fall back on. The tendency to inaccurately extrapolate findings, or to over-generalize conclusions, is a weekness of non-science journalism in general, and the public's willingness to accept such shortcuts costs us all in the long run.
Posted by Earle Holland on Tue 12 May 2009 at 10:47 AM
I agree that most of the original press reports took things a bit far with their misinterpretation (journalists aren't exactly social scientists). But I think what Mr. Holland is ostensibly ignoring is the fact that OSU created the opportunity for a media frenzy by blasting the airwaves with essentially "poor findings." After considering the quality of the work - an un-funded grad student whose convenience sample is hardly representative - I don't see how any respectable organization would want to inject ANY conclusions from this "exploratory" study into the mainstream. Exploratory studies are reserved for conferences and colloquia, not press releases.
Let this be a lesson to everyone on the proper handling of data results. Just imagine the inundation of (mis)information we would have if every university aggressively pushed their grad students' research into the hands of untrained journalists. Especially when they open with alarming headers such as the case with OSU's release, "College students who use Facebook spend less time studying and have lower grade point averages than students who have not signed up for the social networking website, according to a pilot study at one university."
Quit pointing the finger and admit to the fact that your department rushed this out the door in hopes of good PR. I like that you said, "The good thing about reporting on research is that there is actual data, facts, to fall back on." It's just too bad the data and facts in this case were substantially misrepresentative and methodologically flawed.
I practically feel sorry for Aryn getting caught up in this fiasco.
Posted by Mountain out of a mole hill on Tue 12 May 2009 at 12:27 PM
Mountain: It's hard to know where to begin. As a public research university, we have an obligation to convey to the public the new information we glean from our research. While you may think that this student's findings were "poor," more qualified researchers have disagreed. For the record, we didn't "blast the airwaves," with this information, as you suggest. On the contrary, we produced a fully vetted, accurate research story with appropriate caveats and issued it embargoed through a secure pipeline to journalists who are, more-than-not, trained to understand research -- through Newswise and Eurekalert. The initial story that fueled the fire was from the Sunday Times and was largely inaccurate and sensational. The subsequent release of our story two days later was an attempt to curtail some of the misreporting, and it worked in some cases. Your suggestion that such studies "are reserved for conferences and colloquia, not press releases" falls flat when you look at the common practice by most major scientific organizations which almost uniformly issue releases describing the content of presentations made at their major meetings. When properly described, there is no error in releasing information about even preliminary work. The journalism world reports on an enormous number of stories that have no basis in bonafide research at all. As to the lede of the story you quote, it is completely accurate and, given the amount of coverage the inaccurate stories garnered, clearly there is wide interest in the topic, enough to justify issuing a release. We release stories explaining university research in response to our obligation to share our findings with the public. Sure, there is a PR value in that but it is far from the only reason we do it. Given the backing we enjoy from researchers and science writers alike, we're quite comfortable with our part in this whole story.
Posted by Earle Holland on Tue 12 May 2009 at 01:07 PM
Sebastian and Mountain
There isn't much more that I can add to Mr. Hollands responses other than to say that Ms. Karpinski spent many painstaking hours talking to colleagues and professors about making sure that everyone interviewing her KNEW that this was purely a correlational study and that causal inference was NOT to be inferred or extrapolated. The problem here is that Stanford could have used a collegial approach in both recognizing that Ms. Karpinski tried to rebuff the spurious inferences (including countless interviews in an attempt to redirect the extrapolations) and piggybacking her findings by suggesting that correlation is not causation and that is evident by the findings from the Stanford study. Au contraire, it was Stanford that came out swinging against a poster session. Why come out swinging? What is the purpose? Now Stanford is getting scant recognition for their study. The times have changed against ethics in the margins. It is dubious and tenuous at best to assume that blasting and suckerpunching (friendly calls pretending to care about the study only to get the manuscript to do unscrupulous attacks) an exploratory study/poster session is going to garner respect. Moreover, picking only the negative implications highlights the spirit of Stanfords rebuttal, a continuation of swinging on a poster session. I would be embarrassed for you if it weren't so pathetic, so we are left to pitty you as you whimper with your tail between your legs. If you had any idea how much time and effor Ms. Karpinski expended trying to keep integrity about the study, you would be ashamed or yourselves. I suppose the proof is in the pudding, your research is being ignored and Karpinski has international recognition. You should have been collegial (as some at Michigan have been) and Ms. Karpinski would have extended a kind hand assisting your research and findings. i assure you all, you miscalculated
Posted by muckraker on Tue 12 May 2009 at 09:45 PM
once again - read the paper by Pasek et al.
The issue is _not_ correlation vs. causation. Everyone agrees on that.
The issue is completely invalid descriptive inference based on invalid sampling methods. Karpinski did not find a correlation that has any meaning beyond the actual people she interviewed.
And look at this from the point of view of Pasek et al.
These are all researchers who have been working on the topic for a considerable time - and have done due dilligence in their research.
Now the person who gets all the media attention is someone who has done an incredibly sloppy job, _and_ produced findings that are - if compared to the more diligently done research - wrong.
Why shouldn't they come out swinging?
+ if you deal well with it, being involved in a controversy in the social sciences isn't a bad thing (Karpinski got an actual publication out of this) - the only problem is that Karpinski's original article is relatively weak and her 'rejoinder' doesn't really defend her own finding and makes a whole bunch of unfounded accusations against Pasek et al (read the section on recoding from 0-1 in Pasik et al and in her rejoinder - a very common procedure in econometrics, that she not just fails to understand, but also wrongly attacks, making wrong methodological statements (just because a variable is recoded to range from 0-1 it's not dichotomous and of course you would not use logit to analyze it...)
Posted by Sebastian on Tue 12 May 2009 at 10:05 PM
Again, exploratory research...this is a POSTER. The draft paper was a final paper from a course that was not meant for public consumption. You are critiquing a poster of exploratory work. The research is not sloppy, it is EXPLORATORY. Also, in Karpinski's critique, the author points out that the methods and statistics are not clear at first pass. And I don't think it's not clear just to the author. If another researcher was caught in this fray, someone more established, and they were critiquing the Pasek et al study, this may not be an issue with you. But this is a graduate student. POSTER! It is a poster from a poster session that "was made in order to network with others and get ideas from more experienced researchers in the field." It should not be treated as anything different. This was never meant to be published, as it is exploratory work, and the draft paper that was critiqued was a final paper for a course. Let's keep that in mind.
Posted by Sarah on Wed 13 May 2009 at 01:52 AM
tried to post this yesterday w no luck - will split in two:
@Robert:
you are simply wrong in your statement about convenience sampling. All statistical methods that we have at our disposal assume random sampling (or at least quasi random- in the case of cluster sampling and the like).
Using regression analysis on non-random data is just an exercise in nonsense. Any undergraduate statistics yearbook will tell you that - (have a look at the discussion of the famous Literary Digest poll in the Freedman, Pisani, Purves book.)
or look at the version here http://historymatters.gmu.edu/d/5168/
And you are distorting the facts when you say there are disagreements on statistical methods in the social science - this is true in the sense that people debate what type of estimator you may prefer in a specific regression situation - but it is _not_ true in the sense that there is a large amount of statistical knowledge that allows us to make "right" and "wrong" statements - and Karpinski's statements were in the wrong category.
Posted by Sebastian on Thu 14 May 2009 at 12:15 AM
2nd part:
About Graduate Students and surveys: BS!
First - there are a large number of publically available data sets out there that are reasonably random samples - e.g. the one that Pasek - also a graduate student, works with. If you don't have the funds to collect you own data and you want to do quantitative work - go use those (as many graduate students do).
There is also a _huge_ amount of papers and articles by graduate students that _were_ able to implement their own surveys or gather their own meaningful data in some way. That's really no excuse.
Here is my problem: I think a paper based on a convenience sample does not produce any knowledge whatsoever. If the researcher wants feedback on the survey design that could be done - but then it would make sense to actually test (as in randomly vary) survey questions, which I don't think happens.
If you want to produce descriptive findings (and an alleged correlation is such a descriptive finding) you do need randomly sampled data - of the type that Pasek et al use. I don't understand why anyone would want to promote such a study. I really don't.
Btw. One of the exercises we give our students in our undergrad polisci research methods class is to pick a population and then to find a way to get a small random sample from that population and try to find ways to get a high response rate - students do quite well at that. I really don't think this is asking too much.
And standing up for yourself is only a virtue when you're actually right (or at least arguably right), not when you're wrong - re-read Karpinski's rejoinder - there is a whole bunch of _factual_ errors in there.
And Earle - as you insist you guys did nothing wrong - as a science journalist. Could you please tell me about which group - according to your understanding of statistics - the statement:
""College students who use Facebook spend less time studying and have lower grade point averages than students who have not signed up for the social networking website" is correct.
Posted by Sebastian on Thu 14 May 2009 at 12:16 AM
part3:
I'm sorry if I sound angry about this - but a lot of us (and yes, I'm a grad student, too) put an insane amount of work into learning statistical methods and gathering good, meaningful data. I don't mind other people getting the glory - my research is unlikely to garner that type of attention - but I would much prefer that glory go to people (such as e.g. Mr. Pasek, whom I don't know at all) who do careful, well crafted social science.
If you would like some examples of exceptional quantitative work by graduate students, look e.g. at some papers Chris Blattman collected last year that were both well done and addressed meaningful problems.
http://chrisblattman.blogspot.com/2008/09/cleverest-data-collection-ive-seen-all.html
or here
http://chrisblattman.blogspot.com/2008/08/explaining-immigrant-violence-in-africa.html
Posted by Sebastian on Thu 14 May 2009 at 12:17 AM
@Sarah
two points: you can't have it both ways - if the paper was really just a class paper - why on earth turn it into a press release (or let anyone turn this into a press release)? I've thrown out some crazy ideas in term papers, but I wouldn't dream of doing anything to make them extra-public.
secondly, the other reason I'm arguing here is that I was just aghast by Karpinski's response paper. I've listed some of my complaints, I could list more. I think one of the most infuriating part is her conclusion, which boils down to: Both works are imperfect, so we don't know anything.
But in reality, we have two studies, one of which relies on significantly better data, better methods, and produces a more conservative finding. By any means, as long as we don't have better data and results that is the finding we should accept. As anyone will be happy to accept it need and shouldn't be the last word, but until then, not all studies are of equal value. (If you think about this a little bit those "controversy" arguments might sound familiar to you...)
Posted by Sebastian on Thu 14 May 2009 at 12:28 AM
Sebastian: re: your point to Sarah . . . "if the paper was really just a class paper - why on earth turn it into a press release." First off, Karpinsky was presenting at the annual meeting of the American Education Research Association. That's what the news release was based on, not a class paper. Several days the initial and inaccurate reporting of the story, and after posting our release, Aryn shared with us a "paper" further explaining her research. Thinking that this was the basis of her presentation, I mistakenly linked it to our news release on the web. Barely an hour later, Karpinsky contacted us and explained our error, at which time we removed that "paper" as a link and replaced it with a copy of her poster presentation. You can see the poster at http://researchnews.osu.edu/archive/facebook2009.jpg. There was no news release based on a class paper.
re: your other question: "Could you please tell me about which group - according to your understanding of statistics - the statement: 'College students who use Facebook spend less time studying and have lower grade point averages than students who have not signed up for the social networking website' is correct."
That is related to the sixth paragraph of our news release at http://researchnews.osu.edu/archive/facebookusers.htm that says: "Typically, Facebook users in the study had GPAs between 3.0 and 3.5, while non-users had GPAs between 3.5 and 4.0." It also relates to the seventh paragraph of the release which says: "In addition, users said they averaged one to five hours a week studying, while non-users studied 11 to 15 hours per week."
Both statements refer to the population of students taking part in this study. I hope this clarifies things for you.
Posted by Earle Holland on Thu 14 May 2009 at 10:16 AM
Thanks Earle,
My comment to Sarah was about having it both ways - I think it's perfectly alright to publish a press release on a conference paper - but then, by making a scientific finding public, you also agree to subject to the critical assessment of the scientific community and you don't get to say "you can't criticize me because I'm not a real paper".
So my point is that _either_ this is just a meaningless paper, an attempt by a graduate student to think through her research - in which case it should never have been publicized - _or_ it is research that can be presented to the scientific community and beyond - in which case you can't complain about the mean bullies who dare to criticize your work. (which I don't think Karpinski does, but muckraker and Sarah are doing here.)
As for your statement about the population and your press release - you are indeed commendably careful in cautioning against any implied causality - which, I agree, is a lot better than the actual press coverage.
But looking at your press release, I do not think that you are sufficiently careful in phrasing about sampling and generalizability of the findings. Let me explain.
Consider this paragraph from the press release:
"The researchers surveyed 219 students at Ohio State, including 102 undergraduate students and 117 graduate students. Of the participants, 148 said they had a Facebook account.
The study found that 85 percent of undergraduates were Facebook users, while only 52 percent of graduate students had accounts."
Now, this doesn't say something wrong. But it is misleading in two ways:
1. Since most surveys (in and outside of academia) are administered to randomly sampled participants, a reader can reasonably assume that random sampling was used. It wasn't
2. If you say that the study found that x% of students use FB, any reader will assume you are making a statement about the entire University population, not about the ~200 people surveyed. Once again, you're statement _can_ be read to refer only to those surveyed, but any regular reader will assume that it applies to a broader population.
So instead of warning against the "relatively small" study (which really isn't much of an issue - sample size is a lot less important than people make it out to be), it would have been much more important to warn about convenience sampling.
I'll write another post in which I explain, based on the presentation poster (thanks for the link) why this is such a big issue.
Posted by Sebastian on Thu 14 May 2009 at 09:39 PM
the last comment, I promise:
Consider this statement from your press release:
"There were no differences in Facebook use between different members of racial and ethnic groups that were part of the study, or between men and women."
This looks like a very careful statement, yet it is almost certainly wrong.
It is very unlikely that there were actually _no_ differences in FB use between those groups among those surveyed - they were probably just very small - e.g. 65% and 67% or sth. like that.
So were does the "no difference" come from? It is based on Karpinski's finding no _statistically significant_ association between FB use and (e.g.) gender in the Chi^2 tests she performs.
I.e. the so called p-value of the test was higher than .05 or .1.
She does find such a difference, with a p-score
So what does a p score or It means (and this is a hard concept to understand and we spend a lot of time on that in our intro to stats classes):
"If there was no association between FB use and age and we took a large number of different random samples with N=219 from our population, less than one in a thousand of them would show an association this strong or stronger".
In statistical term this allows us to "reject the null hypothesis of no association" with a high degree of significance.
But note two things in the statement above
a) it implies a statement about a larger population than the sample (and providing p-values and talking of stat. significance makes no sense if you don't want to to that
and
b) crucially, it contains the notion of random sampling.
The point being: If your sample is not a random sample, the p-value has no meaning. It doesn't have to be regarded more cautiously. It's not more preliminary. It has zero meaning in statistical terms. You cannot calculate a p-score of a non-random sample.
And this is what this whole controversy boils down to: Karpinski did not get a random sample, but treated her results as if she did. And you reported them as if she did. And that, in statistical terms, is not controversial. It's wrong.
I would strongly urge you, as a science writer, to try to familiarize yourself with these crucial - and relatively basic, though often misunderstood - statistical terms.
There are a lot of great statisticians and social science methodologists at OSU who'll be happy to explain, I'm sure.
Otherwise, I can warmly recommend the undergraduate textbook by Freedman, Pisani and Purves, the standard text in the discipline, easily accessible even for people who aren't super mathematically inclined, yet rigorous in it's treatment of important statistical concepts.
Posted by Sebastian on Fri 15 May 2009 at 12:35 PM
Sebastian: Thanks for the feedback -- dialogue on such subjects is always good, to my mind, and when offered throughtfully and without malice, contributes to the common good. Having said that however, I want to make a couple of points.
First off, and I say this only for informational purposes and not from any defensive position, I think I'm a pretty good judge of what the best practices are in communicating about science and research from a university and directing that information to the media and to the public. Feel free to check out my c.v. on the web (it's easy enough to find) and check my background as evidence of that competency,
You're focusing your arguments on the potential that readers might assume something other than what the factual statements say. That's antithetical to the purposes of doing either news releases or news stories on ongoing research. The imperative is, therefore, to provide accurate statements -- which we did. The expectation is that the media coverage will also provide accurate statements -- which it all-too-often did not.
Add to that the fact that the average readership level for American newspaper subscribers is around fifth-grade. Also, the majority of Americans get their news from television, not newspapers, and the readership level for the copy provided by television news is even lower. Therefore the subtle distinctions that you're making, and the idea that readers will assume/interpret differently from the provided accurate inforamtion, may be potentially possible but that is not our responsibility to correct for.
You say, "And this is what this whole controversy boils down to: Karpinski did not get a random sample, but treated her results as if she did. And you reported them as if she did. And that, in statistical terms, is not controversial. It's wrong."
Karpinski's presentation simply reported on the findings of her small pilot study and the population she addressed. Our release accurately reported the findings, as well as the caveats, considering the audiences to which it was addressed. Premature media coverage inaccurately extrapolated her findings to a broad general population, something both her presentation and our release warned against.
From our standpoint, the issue is whether the release was accurate regarding the study and appropriately written for the intended recipients. It was. Study design issues don't affect the news release and the coverage in this case.
Lastly, in case you missed it, you might be interested in The Observatory piece I did immediately following the one on the Facebook study. Since it deals with research-related news releases, it might broaden your insight into this world.
Posted by Earle Holland on Fri 15 May 2009 at 01:19 PM
OK - I think the first is maybe a moot point - I pointed out how - if I had read just your press release - I would have assumed random sampling in the study. That's what I meant by 'misleading' - but you are right that I am not an expert in newspaper communication, so I'm willing to yield.
On the second point, however, I think you are wrong and you continue to minimize the problem.
a) Karpinski did _not_ simply report her finding. She reported p values and statistical significance, where that was neither appropriate nor statistically feasible. And where it (and thus by implication your press release linking to it) created the impression of generalizability. p-values only make sense if you want to generalize.
b) you argue that: "But the public and some researchers, reacting to the inaccurate reporting, blamed Karpinski for releasing her preliminary results, faulting her methodology." but Pasek et al's critique focused on the sample, _not_ on the causation implied by Karpinski.
So while it is right to blame some of the media for overblowing the causation part, I think the fact that people believed the finding was generalizable goes back to both Karpinski's presentation and your data. I still don't get the sense that you
Posted by Sebastian on Fri 15 May 2009 at 03:18 PM
Sebastian: I don't know another way to say this other than the following . . .
Our responsibility is to accurately report on the research at hand, based on the data that is provided, and the assurances of the investigators that our representation is correct. That is what we did.
The possibility that you or others inaccurately interpreted the findings, or any other aspect of this episode is not within the purview of the science communicators. We cannot control how readers, knowledgeable or otherwise, interpret the information in our releases, nor can we control whether readers interpret some "impression of generalizability," as you maintain. Suggesting that levies an unattainable demand on everyone who writes for others. You think we're not accepting some culpability but you yourself have outlined how you have interpreted the information and that clearly was beyond our control.
Posted by Earle Holland on Sat 16 May 2009 at 09:17 PM
I think this is a pretty well written report by Earle Holland. But I would say this can be a good lesson for people out there in terms of proper handling of data results.
foam mats
swan river cruises
Posted by Jack on Sat 3 Oct 2009 at 11:18 AM