I’ve gotten a lot of mail from people asking my opinion about [the study published today in the Lancet][lancet] about estimating the Iraqi death toll since the US invasion.
So far, I’ve only had a chance to skim the paper. But from what I can see about it, the methodology is sound. They did as careful an analysis as possible under the circumstances, and they’re very open about the limitations of their approach. (For example, they admit that there were methodological changes compared to earlier studies to reduce the risk to members of the survey team; and there were several
data collection errors leading to invalid or incomplete data which was then excluded from the analysis.)
My guess would be that this study is a pretty solid *upper* bound on the death toll of the war. Population-analysis sampling based techniques like this do tend to produce larger numbers than other analyses, but over the long term, while the sampling techniques tend to over-estimate, those higher numbers have tended to be quite a bit *closer* to the truth than the lower numbers generated by other techniques.
When I compare this to what the US government has been trying to feed us, I find that I trust these results much more: this study is open and honest, tells us exactly how they gathered and analyzed the data, and is honest and forthcoming about its limitations and flaws. In comparison, the official US estimates are just black-box numbers – our government has refused to provide *any* information on how their casualty estimates were produced.
Faced with that contrast, and the history of causalty recording and analysis in past wars and natural disasters, I’m strongly inclined to believe that while we will probably *never* know the real number of people who’ve died as a result of our invasion of Iraq, the figure of 600,000 deaths as of today estimated by the Lancet study is *far* closer to the truth than the US government estimate of 30,000 as of last december.
Believe me, nothing would make me happier than being wrong about this. I really don’t want to believe that my country is responsible for a death toll that makes a homicidal maniac like Saddam Hussein look like a pansy… But facts are what they are, and the math argues that this mind-boggling death toll is most likely all too real.
[lancet]: http://www.thelancet.com/webfiles/images/journals/lancet/s0140673606694919.pdf
Good Math/Bad Math
The beauty of math; the humor of stupidity.
Believe me, nothing would make me happier than being wrong about this. I really don’t want to believe that my country is responsible for a death toll that makes a homicidal maniac like Saddam Hussein look like a pansy… But facts are what they are, and the math argues that this mind-boggling death toll is most likely all too real.
I don’t mean to intentionally sound like an apologist. But nonetheless, it is different in that the reason Saddam Hussein caused as much death as he did was because it was intentional repression.
What we did, however, was more along the lines of criminal negligence. We (or our government, more precisely) inprudently invaded a country of whose culture and history we were largely ignorant. Iraq itself has always been an artificial state, originally created by the Brits in the heyday of their colonial empire. The rival factions were only held together by dictators from the start. Hussein was to Iraq what Tito was to Yugoslavia. We’ve blown the lid off and now it’s decending into chaos and civil war.
Now we’re in a position where no matter what we do, the winner is going to end being Iran. Partioning the country is reall only the last option, but even that creates more trouble. Shia Iraq will be an oil rich Iranian client state, Sunni Iraq will likely either come under control of the Baathist Party, and Turkey and Iran (who have large Kurdish populations that they repress) aren’t likely to simply tolerate an independant Kurdistan (plus the Kurds already have a reputation for being friendly to Israel, which is a big no no in the muslim world).
You’re right, it’s hard to deal with. I’d like to be wrong but this is a very dark time to be an American.
Whoa, typo city.
Corrections:
I mean to say that partioning the country is the last option.
I mean to say that Sunni Iraq will either come under controll of the Baathists or the Wahhabis.
Sorry.
Have these results been replicated by a different team of researchers?
Have these results been replicated by a different team of researchers?
I’m not aware of any other studies on this that have been conducted. Do you know of any?
I understand that the same methodology was used in Bosnia and is “the norm” for estimating deaths from natural disasters.
“Have these results been replicated by a different team of researchers?”
Are you refering to the Lancet article, or to di Pietro’s predictions?
Joseph:
The specific analysis of deaths in Iraq has not been replicated. (Given that the paper was just published yesterday, there hasn’t been time for anyone to replicate the results.) But the technique of data gathering and analysis has been used numerous times by different researchers studying different events – everything from the wars in the Balkans to the huge Tsunami of a few years ago, to the Kobe earthquake in Japan… It’s a well-validated approach to performing studies, with well-known strengths and weaknesses. As I said in the original post, based on what we know from past uses of this technique, the figures that it produces are probably higher than the actual numbers, but they’re also most likely closer to the true numbers than other analytical techniques.
Getting back to the Iraq data and analysis specifically, I don’t expect to see the data gathering replicated, for the simple reason that gathering the statistics needed to do this kind of analysis today in Iraq is quite dangerous.
However, an important point is that I haven’t heard anyone actually dispute the validity of the method used to gather the data. They clearly document the process by which the information was gathered, and it’s basically the standard way of gather data for population studies. The only problem with it is that it’s got a rather high margin of error, but they’re very honest about that.
So if the data gathering is valid, *anyone* can repeat the math. And in fact, I’m sure many people will. So if there’s an error in the analysis of the data, we’ll find out very soon.
An honest number with a big error bar is much better than a sham number pulled out of a black box by an authority figure who gives it a fradulently small error bar.
When you say that these type of estimates tend to be high, does that mean that the actual number is probably closer to the lower confidence limit or that it is probably even below that.
In other words am I pretty safe saying that the number of Iragi’s killed is *at least* 400,000? (Or whatever the lower bound actually was)
I know this is by far the largest estimate of Iraqi deaths that anyone has published but why do you think this estimate is an upper bound?
One possible bias that I can see is the fact that empty households, households who refuse to participate and households who were entirely destroyed are omitted from the survey – I reckon it’s quite likely that those three categories are more likely than normal to have had a death in the household.
Maybe there’s some other bias, such as the survey is disproportionately visiting areas with violence, or some Iraqis tend to be in more than one household (insurgents in safehouses?), but I can’t see anything obvious in the report.
Sure, it’s larger than the other estimates but that’s because of wht it’s measuring a different statistic. It’s trying to measure all deaths, or excess deaths, whereas, say, IraqBodyCount measures violent deaths reported in the press, and the Iraqi government counts deaths reporteed to the authorities. Or is it that you mean that actively trying to count all Iraqi deaths will likely be larger than passively counting subsets of those deaths, which I’d agree with?
David:
The reason that I say that I think this estimate is likely an upper bound is because historically, this kind of population-based sampling technique tends to skew towards the
upper end. In fact, I don’t know of a single application of it where a well-done population cluster sampling analysis has resulted in a number *below* the actual death rate. As I said in the post, this analysis technique does seem to come closer to the real numbers than most other techniques, but the tendency of the technique is to skew high. (Most analytical techniques for computing statistics for a population based on sampling data have a tendency to skew in a particular direction. The technique used by Iraq Body Count, for instance, tends to skew low; the technique in this lancet paper tends to skew high.)
Tyler,
I don’t mean to intentionally sound like an apologist. But nonetheless, it is different in that the reason Saddam Hussein caused as much death as he did was because it was intentional repression.
Any sufficiently advanced incompetence is indistinguishable from malevolance. I don’t think that the intent of the person dropping bombs or firing bullets makes much difference to the person on the receiving end of the ordinance. I believe that Saddam justified all (or most) of his atrocities by claiming that it was necessary for the greater good of the Iraqi state and people. How is that different from the American justification?
KeithB:
They acknowledge a fairly large margin of error – their confidence and power are 95% and 80% respectively. Those are *relatively* weak numbers for statistical studies, but for population cluster sampling during a war, they’re quite good. I think it’s very likely that their lower confidence limit is probably a reasonable guess at an accurate lower bound. Not 100% certainty, but it’s a good bet.
I was also asking the general question:
When these kinds of studies over-estimate are the more-accurate numbers usually within the lower bound? Or are the more-accurate numbers lower?
Or does the upper confidence limit of the more-accurate number overlap the lower confidence limit of this kind of study. 8^)
Ed,
I don’t know if I would call it “better”, but I do consider the circumstances different. Most of the deaths our invasion caused were probably not directly caused by our bombs but by the ensuing sectarian violence afterwards. Either way, it’s still shameful.
Mark, this may be an opportunity to expand on some of your earlier discussions of how certain statistical methods may be more (or less) prone to error when the poupulation being sampled possesses certain qualities — in this case, the extreme spatial “lumpiness” when the measured effect is the result of war.
Daniel Davies did an excellent job of describing the problem qualitatively during the last Lancet brouhaha, likening the cluster sampling technique to throwing rocks into a minefield.
It is, of course, possible in principle to criticize a study like this one without being influenced by one’s politics.
In practice, however, we saw damn little impartial criticism last time around, and we are seeing damn little of it now.
Maybe you could help us partway through the minefield? We’ll find our own way out.
I think anyone who wants to use this figure for moral commentary of any kind is obliged to know how it breaks down, i.e. who is killing whom.
Until we get a reliable breakdown, I do not think it is safe or appropriate to use the study as a stick with which to beat anybody. To be fair to the United States, its last experience as an occupying power at the conclusion of a war it won (defined as the destruction or dissolution of the enemy nation’s army, which certainly occurred in Iraq) was NOT of fanatical resistance going on for years, and you have to remember who that last experience was with (Imperial Japan) and how that enemy fought prior to the end.
What we ALSO have to remember, of course, is that in that previous experience, and also the parallel one with ex-Nazi (West) Germany, both nations had had their population centres systematically reduced to rubble and their people blockaded into near-starvation as a deliberate policy of the war (because that’s the way we did things back then, right or wrong), and that may have had a ‘calming’ influence – “Play up, and you’ll get more of the same.”
Yes, it’s tragic that Iraqis are dying. But whether or not you hold the US responsible because it invaded Iraq in the first place, ultimately I want to KNOW exactly who is doing the killing and who the dying, because only then can we start asking the really harsh question of why. And because every report I hear is of suicide bombers or plainclothed attackers striking a mosque, a police station, a wedding party or children at play, and that doesn’t sound to me like the US at all, nor in fact like anyone who really cares about Iraqis either.
Doesn’t the study estimate that 31% of all deaths were due to the Coalition forces (which would put that death toll at 200,000)?
Yes:
About 600k of the 650k deaths were violent. 31% of 600k is roughly 186k.
Rounding to 1 significant figure gives 200k, a difference that seems small compared to the sizes of the confidence intervals.
Here is an seemingly viable critique who lowers the death count by 20 %, or by 100 000 at most: http://scienceblogs.com/authority/2006/10/the_iraq_study_-_how_good_is_i.php . (Varying baseline population 2004 & 2006, nonjustified method to estimate nonviolent excess deaths.) Even if true, still tragically large.
On the moral side much more can be said, but that is perhaps OT. But one comment here says the reasons (morality) behinds deaths are irrelevant, one other that it is decisive. As so often, the truth seems to be in between. The active agent behind the occupation must take full responsibility for making the result possible, but there are other agents too, and dictatorships engenders deaths and often especially when dissolved.
Now I would like to see a study adressing how much money, people and live training worldwide terrorism has been able to recruite as a consequence of the occupation.
Now I see the post I linked to is already commented on a newer thread.
Thanks for the math analyasis, the way I see it: 1 dead for every 4 randomly selected home — That’s bad no matter how you look at it. (They interviewed 1,840 random people and found over 547 dead – 92% of those showed the death certificate)
If you think about it we’ve dropped over 240,000 cluster bombs. We’d be fools to think they didn’t kill anyone. Add in gunfire and car bombs and 600,000 dead doesn’t seem that big.
A large number of the excess deaths were due to non-violent causes. As well as suffering from cluster bombs, stray RPGs, and sudden martyrdoms, ordinary Iraqis have had to deal with the effects of a destroyed infrastucture (contaminated water, streets running with raw sewage, etc.) as in the aftermath of the first Gulf War. In addition, the flight of professionals and an oft-reported, apparently deliberate, targetting of hospitals has had a significant impact on the delivery of health services.
I am not very surprised by the numbers, but I’m not an American.
Mark, you mention that this type of analysis tends to over-estimate.
I’m curious: could you explain?
This is not a snarky question & I don’t have an agenda.
My understanding was that statistical sampling general gave symmetric errors: you might be higher or lower but you didn’t know which.
If you know that you are high, isn’t there some way to correct for that? After all, there has been a huge amount of experience so wouldn’t a “scale by 0.9 because by now we know that is what is needed to fix the error”
Antd why is there a consistent error anyway?
I would (as non-expert) have thought this was an underestimate:
– surveys didn’t go to places that were to dangerous
– if your family was all killed, there is no-one left to interview.
As I say, just trying to learn.