Tuesday 25 January 2011

Misrepresenting the evidence

The Spirit Level’s endemic misrepresentation of the academic literature (see previous post) is made no less worrisome by its authors apparent inability to distinguish between a study which agrees with their hypothesis and one which merely mentions the word ‘inequality’. In response to criticism from Sanandaji et al. that their book focused on their own work while ignoring heavyweight academics, Wilkinson and Pickett wrote:

Other ‘heavyweight’ economists, including Nobel laureates, have also written about the significance of inequality for wellbeing and human capital formation.

As proof, they cited a study by James Heckman, winner of the Nobel Prize for Economic Sciences. Heckman is the co-author of a study titled ‘The Economics and Psychology of Inequality and Human Development’ but nothing in that paper—or in any of his work—implies support for Wilkinson and Picket’s inequality hypothesis. When Sanandaji asked Heckman about how he felt about having his study cited by the two social epidemiologists, he said bluntly: “This is a misrepresentation of my work.” As Sanandaji explains:

Note Wilkinson and Pickett’s choice of words. They write that Heckman has “written” about inequality and health, which is of course technically true. But what they don’t tell the readers is that while he has indeed written about these variables, he has not found any evidence supporting the claims of Wilkinson and Pickett. It is becoming increasingly tiresome to point this out, but Wilkinson and Pickett again and again engage in extraordinary acts of dishonesty.

Whether it be contemporary academics like James Heckman and Robert Putnam or—almost unbelievably—outspoken opponents of socialism such as Alexis de Tocqueville, Wilkinson and Pickett routinely cite the work of other scholars in a context which suggests that they agree with their hypothesis.

In some cases, the studies cited say the exact opposite of what Wilkinson and Pickett claim. As discussed in Chapter 4 of The Spirit Level Delusion, they attempt to explain the higher rate of suicide in more equal countries as a trade-off for a lower homicide rate. The problem with this is two-fold: less equal countries don’t have a higher homicide rate, and the countries studied in The Spirit Level show no evidence of an inverse relationship between homicide and suicide.

Responding to this on their website, Wilkinson and Pickett wrote: “In fact, there are several pieces of research which show that homicide rates are inversely related to suicide.” But the first study they cite as supporting evidence states quite clearly:

Our analysis indicates, overall, the correlation between homicide and suicide rates across all nations is very weak and statistically insignificant.

The shard of truth here is that homicide tends to be more common in very poor countries, while suicide tends to be more common in richer countries. But, as shown on page 82 of The Spirit Level Delusion, there is no correlation between homicide and suicide amongst the rich countries studied in The Spirit Level. And that, of course, is the relevant comparison when discussing Wilkinson and Pickett’s hypothesis.

Either Wilkinson and Pickett are relying on readers not checking their references or they genuinely believe that any study that mentions the word inequality in any context is supportive of their case. This was highlighted again when Kate Pickett was interviewed on BBC Radio 4’s More or Less programme.

It would be hard to find a less politically motivated radio show that More or Less—a programme dedicated to discussing the use and abuse of statistics in the modern media. Wisely deciding against passing judgement on such a voluminous topic in a half-hour magazine show, presenter Tim Harford opted for an interview with Pickett which, in its quiet way, was as devastating as anything written about The Spirit Level in 2010.

In this excerpt, Pickett uses the usual ‘consensus’ defence (see previous post), before being asked about a study she and Wilkinson reference in The Spirit Level to support their claim that “researchers at Harvard University showed that women's status was linked to state-level income inequality.”

KP: We wrote a book that's intended to be a synthesis of a very vast body of research. Not only our own, but those of other people... There is a consistent and robust and large body of evidence showing the same relationship.

TH: That's an interesting point that you make. Often, in response to critics, you have referred not to your own book, not to your own data but to other published research. I'd really like to focus on the research that's presented in your book. It's very easy to say there are 50 papers, there are 200 papers, that support our research but we don't really know how you've selected those papers.

KP: We actually have completed a systematic review of all of the studies of income inequality and health, and we reference that in our book. We do examine things systematically and certainly—when we are doing our own research, publishing in peer-reviewed journals—we have to be aware of all the literature in the field. But that doesn't mean that every paper in the field has good methods, comes to the right conclusion, studies the right thing.

TH: I absolutely agree. One of the papers that you refer to in support of your argument on women's empowerment and women's status which was published in 1999 by Kawachi and some other authors, you claim supports your findings on women's status and income inequality. I've looked at their abstract. It doesn't seem to attack that question at all. It's simply on another subject—a somewhat related subject but not on the subject of income inequality.

KP: They've definitely published and we may have inadvertently put the wrong reference into that document [laughing nervously]. But Kawachi and Kennedy have certainly published finding a relationship between income inequality and women's status. The paper is ‘Women's Status and the Health of Women and Men: a view from the States’ and it was published in Social Science and Medicine in 1999.

TH: That's the one I'm looking at.

The only claim in The Spirit Level that has generated anything approaching “a very vast body of research” is that related to health and inequality. Since their book was published, Wilkinson and Pickett have admitted that the correlation between life expectancy and inequality disappears when different measures of inequality are used. They have also said that “we accept that the inequality/health relationship is one of the weaker associations demonstrated in The Spirit Level.”

The best that can be said of the health-inequality hypothesis is that it remains unresolved and the scatter-plot presented on page 82 of The Spirit Level is unlikely to change that. Richard Wilkinson published a similar scatterplot in the British Medical Journal in 1992 and the peer-reviewed literature shows that he was accused of cherry-picking and data-mining at the time. It is no great surprise that he has received similar criticism now that he has filled an entire book with the same type of evidence.

But while there is an ongoing controversy amongst academics regarding the question of inequality and health, the bulk of The Spirit Level involves theories which have little or no support in the scientific literature. Wilkinson admitted as much in an interview with the magazine International Socialism:

"There are about 200 papers on health and inequality in lots of different settings, probably 40 or 50 looking at violence in relation to inequality, and very few looking at any of the other things in relation to inequality. In a way, the new work in the book is all these other variables—teenage births, mental illness, prison populations and so on—and the major contribution is bringing all of that into a picture that had previously been just health and violence."

What, then, is left of the idea that The Spirit Level is a “synthesis of a very vast body of research”? Wilkinson himself concedes that “very few” studies have looked at anything other than health in relation to inequality. Although Wilkinson and Pickett now portray themselves as standing on the shoulders of giants, in almost every important regard they stand alone.


Carl V Phillips said...

Too bad Heckman did not go with the now Copeland-ized line Marshall McLuhan cameoed in Annie Hall: "You know nothing of my work!"

Keep up the good work. More interesting than the topic, frankly, is how you have presented an argument that does not require just deciding which assertion to believe. Anyone interested in figuring out who is right (rather than just looking for an excuse to believe a particular claim) will find the answer in what you have written. I am making a study of your study of their study.

StillSearching said...

You cite in your January post from an interview with Pickett that she refers to article by Kiwachi and Kennedy saying that women's status is related to income inequality, and then a reply from TH that he had looked at an abstract by Kiwachi and did not find anything about income inequality.
I just now looked up that 2009 article by I. Kiwachi, Kennedy, and others, and it clearly states (see the abstract): "We conclude that women experience higher mortality and morbidity in states where
they have lower levels of political participation and economic autonomy. Living in such states has detrimental
consequences for the health of men as well. Gender inequality and truncated opportunities for women may be one
of the pathways by which the maldistribution of income adversely aff€ects the health of women."
This suggests that Pickett was accurate in citing this research. Perhaps you should have done a little more homework yourself before adding this to your claim that W&P cite imaginary studies.

Christopher Snowdon said...

Gender inequality is an entirely different thing to income inequality. Harford was right to say that study was not about income inequality.

The study shows that gender inequality is linked to poor health outcomes. The abstract reflects that. The fact that you quote the abstract as proof that the study is about income inequality suggests that you don't understand the difference.

StillSearching said...

"you don't understand the difference." Come on, stick to the ideas, and avoid the insults (a common habit of trying to persuade others by denigrating the person). I know full well the difference between income inequality and gender inequality, having studied and researched both of them all of my life. The abstract clearly states that it is the maldistribution of income that affects women's health through lower levels of economic autonomy and political participation. Moving on, do you really believe that heightened income inequality has no effect on the economic autonomy and political participation of the less fortunate? And that this is turn does not affect levels of anxiety, stress, and depression? And that these states have no effect on health? I would be interested to know where in that hypothetical causal chain you think the connection breaks down.

Christopher Snowdon said...

Firstly, it's Kawachi, not Kiwachi. Secondly, the study is from 1999, not 2009. Thirdly, the "maldistribution of income that affects women's health" is a clear reference to male/female income disparities, not the rich/poor inequalities that Wilkinson and Pickett discuss. That shouldn't be surprising since the study looks at whether women's status affects women's health.

Kawachi et al. compared the female mortality rate against various markers of women's status such as the number of women in elected office and the percent of women without health insurance. There were 14 different variables studied. Income inequality was not one of them. And yet, in The Spirit Level, W & P write that the study "showed that women's status was linked to state-level income inequality." This is flatly and demonstrably untrue. No one who read the study - or the abstract, for that matter - could possibly draw such a conclusion. Harford was right to highlight this, which is one of many similar examples.