Bad review

Things are getting just a wee bit contentious in this Cliopatria thread. Perhaps I'm overly skeptical--or cynical--but why is it shocking that peer reviewers don't always catch people like Michael Bellesiles? (It's about as shocking as a prestigious press publishing a bad book.) In Bellesiles' case, the problem was largely one of statistics--or, rather, the unwillingness of anyone to check same. But even when statistics aren't involved, humanities peer review often fails to work as advertised, especially when data abounds.* Certainly, there are the big mistakes that anyone in the same field should be able to catch: major factual errors ("Benjamin Disraeli became Prime Minister in 1864..."), misquotations from famous primary or secondary sources ("A little learning is a delicious thing;/Drink deep, or taste not the Pierian spring"), overegged generalizations ("All Victorian women were strictly confined to the home..."), and the like. Unfortunately, "same field" tends not to cut it when the question becomes not the date of Disraeli's first ministry, but the date of the Rev. Arthur Brown's birth; not a quotation from Alexander Pope, but a quotation from John Angell James; not cod generalizations about Victorian women, but cod generalizations about Primitive Methodists in Manchester.

"Well, send the manuscript to someone in the same line of work, you dolt," snaps the impatient reader. "Golly, we'd like to," snarls an exasperated editor-in-chief, "but there isn't anybody in the same line of work." Once the scholar steps out of the major byways and highways of literary or historical research, even basic fact-checking becomes a real problem. It becomes even worse when the source material is largely archival. All those who want to pony up the expenses to double-check someone's quotations from a manuscript located in, say, Lancashire, please raise your hands. (I don't see any hands. Odd, that.) For that matter, someone who wanted to double-check the quotations in one of my articles would have an arduous chore--even UCLA's Sadleir Collection, the most significant Victorian novel collection in the United States, doesn't have everything I cite.

"Surely you don't mean that referees examine research about which they know nothing!" cries my reader (no doubt somewhat tongue-in-cheek). Well, no, that's not what I'm saying. A responsible reader will know quite a bit about the "umbrella" field, as it were. She'll do her best to fill in the blanks in her own knowledge, try a bit of spot-checking, and so forth. But, unless she has already gone through the same archives or rare books, she has to presume that the manuscript has been put together by someone possessing basic competence (and honesty). From there, at least two things have to happen: the reader has to extrapolate the scholar's more general trustworthiness from his mastery of those things the reader does know; and she has to examine how the argument has been put together. Given the quotations provided, do the analyses make sense? Are the citations current? Are there major lapses in logic? Does the author target the proper audience? In practice, this approach stops a fair amount of incompetent work. It's the illusion of competence, accompanied by esoteric subject matter, that causes real difficulties.

There's another problem: time. The Bellesiles case is a good case in point here, because it took forever to fact-check him properly after the book was published--and it seems as though his best critics had to drop everything (or a lot of things) in order to devote their full attention to researching his errors. Now, it's one thing to go grab an anthology in order to check some obscure poems, as I did the first time I refereed a paper, and quite another to turn someone else's manuscript into your own research project. Once again, all those who want to fund sabbaticals so referees can completely retrace the steps in someone else's work, please raise your hands. (Darn. Still no hands going up.) To be blunt, there's little motivation to abandon your own work entirely in order to body-block someone else's--nothing tangible will come out of it. Moreover, it's not really the done thing for a referee to sit on a book or an article for a year. I mean, it is a done thing, but it's not likely to earn the referee in question Valentine's Day cards and Godiva chocolates. The great irony, of course, is that there is far more motivation to go into full research-and-attack mode after a bad/fraudulent book has been published than before. At that point, it does become "your work," in the same way any response to any book becomes "your work."

*--Dad the Emeritus Historian of Graeco-Roman Egypt points out to me that, by and large, everyone in his field has read every source available. But the more stuffed the archives, the less likely scholars will share an intellectual commons.