Last September, CJR approached me to write an article for a special package on health and science journalism in the January/February issue. The subject: “one of the more significant trends in science journalism in the last 20 years — a shift in focus toward personal health news, especially diet and fitness.” And the premise: “the public craves simple answers and immediate solutions and the press indulges those unreasonable expectations.”
I responded that the assignment was interesting, particularly so “since I’m part of that trend and there are certainly journalists (and researchers) out there, including some old friends, who would say I’m pandering to the public’s desire to have easy answers.” Ultimately I turned it down, though, not for lack of interest but lack of time. Instead, I suggested they offer the story to my good friend David Freedman, who accepted.
The result was Dave’s cover article, “Survival of the Wrongest,” which oddly enough singled out my work, along with that of The New York Times‘s Tara Parker Pope, as examples of health journalism gone awry. Both articles were on obesity and the implication was that Parker Pope and I might be, at least in these cases, sacrificing the tenets of good journalism for the benefits of promoting a favored theory that bucks conventional wisdom.
Two journalistic questions were raised by my friend Dave’s article that are worth addressing, the second admittedly more critical than the first:
1) Should we add to the multitude of reasons why a freelancer should never turn down an assignment, the possibility that the friend whom you recommend instead (or the writer the editors choose independently) will publicly disparage your own work in the subsequent article?
2) Considering that most of what is published in the medical journals is not nearly as meaningful or even correct as we might like to believe–a problem that Dave does such a good job of addressing–how should those of us who cover these fields report on these issues? This question embraces both reporting on deadline for tomorrow morning’s paper and the kind of long interpretive/investigative articles that Dave was criticizing.
I wrote about this problem 14 years ago (“Telling time by the second hand,” in Technology Review) and it was not particularly new then. At the time, I quoted the philosopher of science John Ziman describing the front-line of research– where those of us covering science and medicine make our living–as a place where reliable knowledge is hard to come by and “controversy, conjecture, contradiction, and confusion are rife.” Using physics as an example, Ziman estimated that 90 percent of the science in the journals was probably wrong, compared to only ten percent in the undergraduate textbooks. (The job of science, he suggested, was to separate the wheat of the textbooks from the chaff in the journals.) Dave uses the recent research of John Ioannidis of Stanford to estimate that two-thirds of the papers in the medical journals are unreliable, although that’s only the “top medical journals” and we can assume the lesser journals, of which there are multitudes, have a batting average considerably worse.
With these numbers in mind, as Dave says, “a reporter who accurately reports findings is probably transmitting wrong findings.” This issue is the subject of the bulk of Dave’s article and it should engender the most consensus, at least in theory if not necessarily in practice: If we have to cover these articles at all, the liberal use of caveats and conditions, and context (the proportion of articles on the subject that came to different conclusions in the past) would be an excellent idea. As I noted in my 1998 essay, the very first reports on any new finding–the ones most likely to get media attention–were also the ones most likely to be wrong. This is the nature of science. It is fundamentally at odds with what we do in journalism, whether personal health or not, and we have to always keep it in mind.
The more interesting issue in Dave’s discussion is why experienced science and health journalists covering the same subject, (obesity, in this case) and all doing precisely what Dave counsels–approaching the literature skeptically and thoughtfully and applying copious common sense–can still come to conclusions that appear to be contradictory conclusions. Dave had his own cover article in Scientific American on obesity, just as Parker Pope and I did in The News York Times Magazine. All are attempts to make sense of the controversy, conjecture, contradiction and confusion inherent in obesity research, and yet all are at odds with each other.
Dave’s conclusions are orthogonal to mine and to Parker Pope’s, which are, as Dave notes, orthogonal to each other. My conclusions, Dave writes, left “the majority of frontline obesity researchers gritting their teeth.” Parker Pope’s apparently left a majority pronouncing her “main thesis — that sustaining weight loss is nearly impossible — dead wrong, and misleading in a way that could seriously, if indirectly, damage the health of millions of people.”
So how are we to make sense of a situation when thoughtful investigations of controversial subjects not only disagree with the conventional wisdom–what the majority of front-line researchers are said to believe–but with each other? Why does this happen? How do we cover it as journalists and interpret it as critical readers? I’ll continue to use obesity and weight loss as the case study here because it is now the subject that I (arguably) know best and it’s at the heart of Dave’s article.
One obvious explanation for the rise of personal health journalism, particularly on the subject of weight loss, is because obesity seems to be such an intractable problem on a personal level. If sustaining significant weight loss was not at least exceedingly difficult–a subject we’ll discuss shortly as it’s central to Dave’s argument–the plethora of books and articles suggesting easy fixes or never-before-tried fixes would be unnecessary. New diet books come along every day because the old books have, at the very least, outlived their usefulness. And because obesity is now such a critical societal problem, effecting the long term viability of our healthcare system and perhaps our economy itself, it’s not unusual these days to even have economists and circuit court judges speculating publicly about its cause and the best means of prevention.
In the late 1950s, less than one in eight Americans were obese, and fewer than 200 articles were published yearly on obesity in the English language medical literature. Today, more than one in three Americans are obese, more than 15,000 relevant articles were published in the academic literature last year alone, and the cost of obesity to the health care system in the US is estimated at $150 billion a year. These numbers alone suggest a research pursuit in turmoil, and a subject that should be ripe for journalistic investigation (and venal exploitation as well). If the conventional wisdom on the subject is correct–if all we have to do to lose weight is eat less and exercise more–then why so much research, and why are so many of us fat, and why so many more now than fifty years ago? And, if it’s not correct, then why not?
The articles by Dave, Parker Pope and myself are all attempts to answer these questions and to make sense of what Dave calls the main thesis of Parker Pope’s article, which he declares dead wrong: that weight loss is nearly impossible to sustain and so obesity is indeed an intractable condition. We’re sifting the evidence in different ways, approaching it from different perspectives, and coming to different conclusions. If the main thesis wasn’t at least mostly right, though, despite Dave’s claim to the contrary, none of us would need to write about this at all, and a over a third of the US population would not be considered clinically obese.
Because Dave, Parker Pope, and I take different approaches to the problem, the evidence we consider meaningful–and, of course, not all of it is, and perhaps not even most of it–is naturally different as well. Parker Pope argues that obesity is intractable based on common sense and some small, very well controlled studies (a fundamental requirement of good science). She concludes that that once we get obese, we’re pretty much stuck and public health advice and discussion should reflect that. She is implicitly challenging the kind of anecdotal accounts evoked by Dave himself (“most of us know people–friends, family members, colleagues–who have lost weight and kept it off for years by changing the way they eat and boosting their physical activity”) to make the argument that all it takes to be significantly less obese and stay that way is the right kind of behavioral modification and, well, maybe sufficient will power as well.
I agree with Parker Pope’s assessment about the intractability of obesity, but I argue that this is a product not so much of how the human body responds to weight loss but how it responds to the methods used to achieve that weight loss. I argued in my New York Times Magazine article that the old advice not to eat carbohydrates (refined grains and sugars, in particular) may have been the right advice and that these carbohydrates might be the fundamental causes of obesity.
If this is true, then the reason why obesity appears to be so intractable is that we are using the wrong means to cure and prevent it. In other words, if it’s not caused merely by eating too much and exercising too little, then it’s no surprise that it is resistant to behavioral modification aimed at eating less and, as Dave says, boosting physical activity. I recently co-founded a non-profit, the Nutrition Science Initiative, with support from the Laura and John Arnold Foundation, to facilitate and fund well-controlled experiments that should be able to establish reliably which of these causal hypotheses of obesity is correct.
As for Dave, he challenges the intractability argument itself. By doing so, he can argue that behavioral modification for obesity works, provided it’s done correctly, which is the new twist that he is bringing to the subject.
In a follow-up post on CJR’s website, “Playing the study game,” Dave defends his position that obesity is not an intractable condition by referencing five studies that he says “come up with positive long-term weight-loss results.” In doing so, he provides the case study for lesson number two in reporting on health and medicine. This one should also engender a consensus, again at least in theory if not in practice: always read the articles before writing about them and committing anything to press. (And acknowledge, as Dave does, that other articles could easily have been cited making orthogonal points.)
It’s also a lesson in the ambiguity of evidence that may be unique to medical research. Doing rigorously well-controlled experiments is exceedingly expensive when humans are the subjects, and the ethical challenges are enormous as well. But they are what is necessary to establish reliable knowledge. Unfortunately, many medical researchers and journalists have come to rely on lesser evidence of the kind Dave references, and it’s simply not good enough.
Dave’s first reference is about the National Weight Control Registry (NWCR). This is a database of 10,000 individuals who have lost at least 30 pounds and then maintained that loss for more than a year. If I did that, at 6’2″ and 240, my maximum weight, it would drop my BMI from 30.8, mildly obese, to 27 in the mid range overweight.) The NWCR reports that its members have lost an average of 33 kilograms and maintained that weight loss for more than five years.
This seems impressive and strong confirmation of Dave’s interpretation, until you take into account the point that Parker Pope makes in her article about this very database, which is that the NWCR is nothing more than a compilation of anecdotal accounts. Parker Pope quotes an obesity researcher from Yale (who happens to be obese himself, apparently intractably so) noting correctly that “while the 10,000 people tracked in the registry are a useful resource, they also represent a tiny percentage of the tens of millions of people who have tried unsuccessfully to lose weight. All it means is that there are rare individuals who do manage to keep it off.” And that is indeed all it means. It says nothing about how large or small a percentage of obese individuals can do this. It could be one in ten thousand or less. It could be one in two. We have no idea.
Of Dave’s three other references, one is a review article of behavioral weight loss programs in children that makes only the tepid claim that “limited evidence suggests” adolescents can maintain a weight loss of 4 to 8 pounds for a year. One is a study in adults claiming that two thirds of those who got intensive behavioral modification did indeed maintain at least a 20-pound weight loss for one year, but it then says “three- to 5-year follow-up studies showed a gradual return to baseline weight.”
The last reference reports that subjects who averaged a 65-pound weight loss over five months gained back all but 15 pounds after five years. This is a glass half full, half empty case. Dave apparently looks at the 15 pounds and says, this is reason to believe Parker Pope was wrong. Parker Pope, I suspect, and I would look at the 50 pounds regained as evidence that she was right.
Finally, Dave evokes the Look Ahead study, a massive trial predicated on the notion that maintenance of a five percent weight over ten years is clinically meaningful. Dave says he wouldn’t t be so quick “to dismiss a 5 percent loss as nothing to celebrate.” He calls it a “proof of concept, putting the lie to claims that we are genetically fixed to have a certain weight.”
Now consider as an example a woman like Parker Pope, who says in her article that she is 60 pounds overweight. Let’s assume that our subject is of average height, 5’4″, and so an ideal weight might be 140 pounds (a BMI of 24). If she’s 60 pounds overweight, she would weigh 200 pounds and have a BMI of 34, well into the obese range. Dave’s implication would then be that if our hypothetical case dropped from 200 pounds to 190 (BMI 32.6, still obese) and maintained that indefinitely, she should consider this a reason to celebrate and a refutation of the central point of Parker Pope’s article. I would argue otherwise and I suspect Parker Pope would as well.
In interpreting science and medicine for the public and investigating issues so critical to the public health, we’re all biased, just as scientists are, by our perspectives, our experiences and our preconceptions. If we don’t start off biased, we will soon find ourselves consciously or unconsciously taking sides, and that will bias our perceptions from then on. In a world in which virtually any argument can be made from the evidence at hand, our ultimate task, just as it with science itself, is to communicate reliable knowledge. Here, the legendary physicist Richard Feynman made two requirements for good science that I would argue are the requisites for good journalistic investigations as well: One is “honesty in reporting results — the results must be reported without somebody saying what they would like the results to have been.” The other is “you must not fool yourself and you are the easiest person to fool.”
Gary Taubes is co-founder of the Nutrition Science Initiative, and the author of Why We Get Fat and What to Do About It (Knopf 2010) and Good Calories, Bad Calories (Knopf 2007). He’s a contributing correspondent for the journal Science, and the only print journalist to win the Science in Society Journalism Award of the National Association of Science Writers three times.