The numbers do add up

As Richard Horton's post says, the latest Johns Hopkins University study of mortality in Iraq, published in the Lancet is horrible news. When the previous study was published, a horrendous chorus of hacks sprung up and suddenly discovered a new-found expertise in epidemiological statistics.

Tim Lambert, the Australian science-blogger, and I ended up spending a lot of time and energy fighting on the online front of this Campaign For Real Statistics, and so it is with heavy heart that I see that President Bush - who is probably a better statistician that many of his online supporters as he has at least been to business school - has already expressed an uninformed opinion on the matter.

There will be a concerted attempt to persuade people that the statistical issues involved in this study are difficult. They aren't. The correct way to think about this is as follows:

First, don't concentrate on the number 600,000 (or 655,000, depending on where you read). This is a point estimate of the number of excess Iraqi deaths - it's basically equal to the change in the death rate since the invasion, multiplied by the population of Iraq, multiplied by three-and-a-quarter years. Point estimates are almost never the important results of statistical studies and I wish the statistics profession would stop printing them as headlines.

The question that this study was set up to answer was: as a result of the invasion, have things got better or worse in Iraq? And if they have got worse, have they got a little bit worse or a lot worse. Point estimates are only interesting in so far as they demonstrate or dramatise the answer to this question.

The results speak for themselves. There was a sample of 12,801 individuals in 1,849 households, in 47 geographical locations. That is a big sample, not a small one. The opinion polls from Mori and such which measure political support use a sample size of about 2,000 individuals, and they have a margin of error of +/- 3%. If Margaret Beckett looks at the Labour party's rating in the polls, she presumably considers this to be reasonably reliable, so she should not contribute to public ignorance by allowing her department to disparage "small samples extrapolated to the whole country". The Iraq Body Count website and the Iraqi government statistics are not better measures than the survey results, because one of the things we know about war zones is that casualties are under-reported, usually by a factor of more than five.

And the results were shocking. In the 18 months before the invasion, the sample reported 82 deaths, two of them from violence. In the 39 months since the invasion, the sample households had seen 547 deaths, 300 of them from violence. The death rate expressed as deaths per 1,000 per year had gone up from 5.5 to 13.3.

Talk of confidence intervals becomes frankly irrelevant at this point. If you want to pick a figure for the precise number of excess deaths, then (1.33% - 0.55%) x 26,000,000 x 3.25 = 659,000 is as good as any, multiplying out the difference between the death rates by the population of Iraq and the time since the invasion. But we're interested in the qualitative conclusion here.

That qualitative conclusion is this: things have got worse, and they have got a lot worse, not a little bit worse. Whatever detailed criticisms one might make of the methodology of the study (and I have searched assiduously for the last two years, with the assistance of a lot of partisans of the Iraq war who have tried to pick holes in the study, and not found any), the numbers are too big. If you go out and ask 12,000 people whether a family member has died and get reports of 300 deaths from violence, then that is not consistent with there being only 60,000 deaths from violence in a country of 26 million. It is not even nearly consistent.

This is the question to always keep at the front of your mind when arguments are being slung around (and it is the general question one should always be thinking of when people talk statistics). How Would One Get This Sample, If The Facts Were Not This Way? There is really only one answer - that the study was fraudulent.[1] It really could not have happened by chance. If a Mori poll puts the Labour party on 40% support, then we know that there is some inaccuracy in the poll, but we also know that there is basically zero chance that the true level of support is 2% or 96%, and for the Lancet survey to have delivered the results it did if the true body count is 60,000 would be about as improbable as this. Anyone who wants to dispute the important conclusion of the study has to be prepared to accuse the authors of fraud, and presumably to accept the legal consequences of doing so.

So what? This is always the other line from the people who want to ignore this study. Even if we accept that the invasion has been a disaster (in the strictest sense, the doubling of the civilian death-rate is usually taken to constitute a humanitarian crisis) for the Iraqi people, what should we do differently? The majority of the deaths by violence are a result of action by the insurgents, so we can't just pull the troops home. Isn't this kind of study just "picking over the rubble", to quote the Euston Manifesto and a distraction from the real debate about humanitarian intervention?

Well, there is something that we can do. We can ensure that the people responsible for this outrage suffer the consequences of their actions. A particularly disgusting theme of some right-wing American critics of the study as been to impugn it by talking about it being "conveniently" released before the November congressional elections. As if a war that doubled the death rate in Iraq was not the sort of thing that ought to be a political issue. Nobody is doing anything about this disaster, and nobody will do until people start suffering some kind of consequences for their actions (for example, no British politician, soldier or spy has lost his job over the handling of the Iraq war and no senior member of the Bush administration either).

There has to be some accountability here. It is not good enough for the pro-intervention community to shrug their shoulders and say that the fatalities caused by the insurgents are not our fault and not part of the moral calculus. I would surely like to see the insurgents in the ICC on war crimes charges, but the Nuremberg convention was also correct to say that aggression was "the supreme international crime differing only from other war crimes in that it contains within itself the accumulated evil of the whole". The people who started this war of aggression need to face up to the fact, and that is a political issue.

[1] In the context of the 2004 study, I was prepared to countenance another explanation: that the Iraqis were lying and systematically exaggerating the number of deaths. But in the 2006 study, death certificates were checked and found in 92% of cases.