Misrepresenting the evidence

The Spirit Level’s endemic misrepresentation of the academic literature (see previous post) is made no less worrisome by its authors apparent inability to distinguish between a study which agrees with their hypothesis and one which merely mentions the word ‘inequality’. In response to criticism from Sanandaji et al. that their book focused on their own work while ignoring heavyweight academics, Wilkinson and Pickett wrote:

Other ‘heavyweight’ economists, including Nobel laureates, have also written about the significance of inequality for wellbeing and human capital formation.

As proof, they cited a study by James Heckman, winner of the Nobel Prize for Economic Sciences. Heckman is the co-author of a study titled ‘The Economics and Psychology of Inequality and Human Development’ but nothing in that paper—or in any of his work—implies support for Wilkinson and Picket’s inequality hypothesis. When Sanandaji asked Heckman about how he felt about having his study cited by the two social epidemiologists, he said bluntly: “This is a misrepresentation of my work.” As Sanandaji explains:

Note Wilkinson and Pickett’s choice of words. They write that Heckman has “written” about inequality and health, which is of course technically true. But what they don’t tell the readers is that while he has indeed written about these variables, he has not found any evidence supporting the claims of Wilkinson and Pickett. It is becoming increasingly tiresome to point this out, but Wilkinson and Pickett again and again engage in extraordinary acts of dishonesty.

Whether it be contemporary academics like James Heckman and Robert Putnam or—almost unbelievably—outspoken opponents of socialism such as Alexis de Tocqueville, Wilkinson and Pickett routinely cite the work of other scholars in a context which suggests that they agree with their hypothesis.

In some cases, the studies cited say the exact opposite of what Wilkinson and Pickett claim. As discussed in Chapter 4 of The Spirit Level Delusion, they attempt to explain the higher rate of suicide in more equal countries as a trade-off for a lower homicide rate. The problem with this is two-fold: less equal countries don’t have a higher homicide rate, and the countries studied in The Spirit Level show no evidence of an inverse relationship between homicide and suicide.

Responding to this on their website, Wilkinson and Pickett wrote: “In fact, there are several pieces of research which show that homicide rates are inversely related to suicide.” But the first study they cite as supporting evidence states quite clearly:

Our analysis indicates, overall, the correlation between homicide and suicide rates across all nations is very weak and statistically insignificant.

The shard of truth here is that homicide tends to be more common in very poor countries, while suicide tends to be more common in richer countries. But, as shown on page 82 of The Spirit Level Delusion, there is no correlation between homicide and suicide amongst the rich countries studied in The Spirit Level. And that, of course, is the relevant comparison when discussing Wilkinson and Pickett’s hypothesis.

Either Wilkinson and Pickett are relying on readers not checking their references or they genuinely believe that any study that mentions the word inequality in any context is supportive of their case. This was highlighted again when Kate Pickett was interviewed on BBC Radio 4’s More or Less programme.

It would be hard to find a less politically motivated radio show that More or Less—a programme dedicated to discussing the use and abuse of statistics in the modern media. Wisely deciding against passing judgement on such a voluminous topic in a half-hour magazine show, presenter Tim Harford opted for an interview with Pickett which, in its quiet way, was as devastating as anything written about The Spirit Level in 2010.

In this excerpt, Pickett uses the usual ‘consensus’ defence (see previous post), before being asked about a study she and Wilkinson reference in The Spirit Level to support their claim that “researchers at Harvard University showed that women's status was linked to state-level income inequality.”

KP: We wrote a book that's intended to be a synthesis of a very vast body of research. Not only our own, but those of other people... There is a consistent and robust and large body of evidence showing the same relationship.

TH: That's an interesting point that you make. Often, in response to critics, you have referred not to your own book, not to your own data but to other published research. I'd really like to focus on the research that's presented in your book. It's very easy to say there are 50 papers, there are 200 papers, that support our research but we don't really know how you've selected those papers.

KP: We actually have completed a systematic review of all of the studies of income inequality and health, and we reference that in our book. We do examine things systematically and certainly—when we are doing our own research, publishing in peer-reviewed journals—we have to be aware of all the literature in the field. But that doesn't mean that every paper in the field has good methods, comes to the right conclusion, studies the right thing.

TH: I absolutely agree. One of the papers that you refer to in support of your argument on women's empowerment and women's status which was published in 1999 by Kawachi and some other authors, you claim supports your findings on women's status and income inequality. I've looked at their abstract. It doesn't seem to attack that question at all. It's simply on another subject—a somewhat related subject but not on the subject of income inequality.

KP: They've definitely published and we may have inadvertently put the wrong reference into that document [laughing nervously]. But Kawachi and Kennedy have certainly published finding a relationship between income inequality and women's status. The paper is ‘Women's Status and the Health of Women and Men: a view from the States’ and it was published in Social Science and Medicine in 1999.

TH: That's the one I'm looking at.

The only claim in The Spirit Level that has generated anything approaching “a very vast body of research” is that related to health and inequality. Since their book was published, Wilkinson and Pickett have admitted that the correlation between life expectancy and inequality disappears when different measures of inequality are used. They have also said that “we accept that the inequality/health relationship is one of the weaker associations demonstrated in The Spirit Level.”

The best that can be said of the health-inequality hypothesis is that it remains unresolved and the scatter-plot presented on page 82 of The Spirit Level is unlikely to change that. Richard Wilkinson published a similar scatterplot in the British Medical Journal in 1992 and the peer-reviewed literature shows that he was accused of cherry-picking and data-mining at the time. It is no great surprise that he has received similar criticism now that he has filled an entire book with the same type of evidence.

But while there is an ongoing controversy amongst academics regarding the question of inequality and health, the bulk of The Spirit Level involves theories which have little or no support in the scientific literature. Wilkinson admitted as much in an interview with the magazine International Socialism:

"There are about 200 papers on health and inequality in lots of different settings, probably 40 or 50 looking at violence in relation to inequality, and very few looking at any of the other things in relation to inequality. In a way, the new work in the book is all these other variables—teenage births, mental illness, prison populations and so on—and the major contribution is bringing all of that into a picture that had previously been just health and violence."

What, then, is left of the idea that The Spirit Level is a “synthesis of a very vast body of research”? Wilkinson himself concedes that “very few” studies have looked at anything other than health in relation to inequality. Although Wilkinson and Pickett now portray themselves as standing on the shoulders of giants, in almost every important regard they stand alone.