Fun and games with citation counts

Justin Wolfers at the Freakonomics blog on Hayek's scholarly impact:

Taking social science seriously surely means teaching the insights of the most prominent, most important, or most influential economists. This involves teaching important theories—even those you disagree with. There’s no doubt about the influence of Smith, Marx and Keynes; Friedman also belongs. But does Hayek belong on this list?

Let’s use data to inform this debate. I counted the number of references to each economist in the scholarly literature indexed by JSTOR, finding 30,708 articles mentioning “Adam Smith”; 25,626 articles mentioning “Karl Marx”; and 4,945 mentioning “John Maynard Keynes” (the middle name was required to avoid articles by his father, John Neville Keynes). “Milton Friedman” sits easily with this group, and was mentioned in 8,924 articles.

But searching for “Friedrich von Hayek” only yielded 398 articles; adding “(("Friedrich von Hayek") OR ("Friedrich Hayek"))” raised his total to 1242 mentions; also allowing “FH Hayek” raised his count to 1561.

He later corrected FH Hayek to FA Hayek and got up to 1745. This seemed odd to me.

So I went to JSTOR to replicate the results.

JSTOR, full text search, no restrictions as to years or discipline or kind of publication.

"Friedrich von Hayek"-- 397. Close enough to think I'm conducting the searches the same way Wolfers did.

"Friedrich von Hayek" OR "Friedrich Hayek"-- 1231. The discrepancy grows a little bit, but I'm still in Wolfer's ballpark. So these unrestricted full-text searches are what he was doing.

Next step, to get a ceiling estimate:

full-text search on Hayek . 12088 results. Browsing through these yields very few false positives-- so now I'm suspicious.

I notice a lot of references to "Professor Hayek." This seems to have been the convention in some academic journals at midcentury. "Professor Hayek" by itself yields 582 hits, and a quick browse yielded no false positives.

I also notice that the search engine cares about the difference between "F.A. Hayek" and "F. A. Hayek" (with a space between the first period and the A). This makes a big difference. Simply performing the search as "'F.A. Hayek' OR 'F. A. Hayek'" already yields 2219 results-- more than Wolfers' most complete version of his search.

Adding in F. A. von Hayek and Friedrich Hayek as options gets us to 3342.

Now there are more permutations than JSTOR can readily handle. But stretching out the search to ("Friedrich Hayek") OR ("Friedrich von Hayek") OR ("F.A. Hayek") OR ("F.A. von Hayek") OR ("F. A. Hayek") OR ("F. A. von Hayek") OR ("Friedrich A. Hayek") OR ("Friedrich A. von Hayek") OR ("Professor Hayek") yields 4267.

Proceeding from the other direction: a search just on Hayek restricted to business, economics, finance, law, linguistics, philosophy, political science, psychology, public policy, and sociology eliminated all the false positives I could find. 9385 . Searching for "milton friedman" in those same disciplines (and as far as I know there's no ambiguity in how to refer to him): 8088.

Now, I don't really think that citation counts are going to do the work Wolfers wants them to do here. But on his terms, Hayek is now out of Larry Summers' company, and into Friedman's.

Update:

See Wolfers' reply, and mine a few below his, in comments at Marginal Revolution. There's just a problem here in comparing names that are difficult to compare. "Hayek" almost always refers to the relevant person in academic searches (unlike, say, at Dan Drezner's blog.) You get very few false positives with just the last name. But the false negatives are extremely sensitive to variations in how you specify his given names. Approximately the same is true of Keynes, though as Wolfers notes Keynes' father was also well-known.

Milton Friedman, Adam Smith, and Karl Marx are opposite cases. Last names only will yield a huge number of false positives; the last names are just too common. But there's very little variation in how given names are specified-- I can't think that I've ever seen any of the three referred to by initials or with a middle name. So simply applying "the same" rule to Hayek and Friedman will get incomparable results-- vastly too many false positives or vastly too many false negatives. And so any of these count-comparisons are going to look more precise than they really are. But my strong impression from trying lots of variations: Smith, Marx, and Keynes are in a class by themselves, and Hayek and Friedman are basically comparable.

See also D-squared, who rightly emphasizes that "if something isn't worth doing, it isn't worth doing well".


Further update:
As long as I'm getting so much traffic sent here by Marginal Revolution and other blogs, might as well link to my own most recent engagement with Hayekian themes: Not So Novus an Ordo: Constitutions Without Social Contracts, Political Theory, Vol. 37, No. 2, 191-217 (2009)