John Lott, scholar

I haven't said much about the John Lott affair directly; just this rumination on the Mary Rosh thing. But I've been reading Tim Lambert's updates as well as associated commentary by folks such as Kieran Healy. The IRB question Kieran raises had occured to me, but I didn't follow up. Here's how it now looks to me:

Even if Lott conducted the survey exactly as he says he did, it was sloppy, shoddy, and pretty dramatically invalid. It shouldn't have been used as the basis even for one sentence in the book, much less for a claim that he repeated over and over and over again. If, as seems likely, he didn't go through the IRB (one would think that if he had, he'd by now have gotten a copy of his proposal back from them and released it; they keep such things), then he violated professional rules. If he used the methodology he described, then he has no more business conducting surveys and using them as evidence in his scholarly writings than, well, I do. But I know better. Even if I had a spare couple of grand burning a hole in my pocket *and* a point that I wanted to make using opinion data, I wouldn't just spontaneously make up a survey and hire students to conduct it. (I should note that, given the way things are done here at Chicago, I find the notion that he paid out of pocket with no records particularly implausible.) I'd work with someone who knew what he or she was doing. There are plenty of such people at Chicago.

In short: if Lott's not a liar, then he's a very, very sloppy researcher who violates rules of professional conduct. Not good, and not someone who ought to be defended, regardless of how much we may want his conclusions to be true.

UPDATE: I've received additional confirmation of Lambert's suggestion that IRB procedures aren't SOP in law schools in general. (The confirmation didn't come from colleagues here at Chicago, whom I haven't asked.) At some point, this may well blow up in someone's face. When my friends in business complain that their lawyers won't let them do stuff they want to do, I always remind them: Your lawyers are there to protect you against making mistakes, and against other people being able to say later that you screwed up. Now I'll start saying the same to law professors about IRBs.

Lott's defenders, and Lott himself, are making a great deal out of the peripheralness of the 98% figure. But that makes it all the stranger. Why keep repeating and repeating a claim for which one has no valid evidence, when it would be no great sacrifice to drop it? Why misattribute it, and fail to cite the existing studies that show different results, when it matters so little?

ANOTHER UPDATE: Mark Kleiman hits many of the same points.