Thanks to everyone who read and/or commented on the earlier PGR Minutiae post. I wanted to correct a couple of defects and incorporate some issues that came up in comments.
1. I presented the points in the order that seemed to flow the best, not in order of importance. In descending order of importance, they would go (2), (1), (4), (3).
2. I didn’t really “sum up” or anything, but it seems to me that the four criticisms have a sort of net effect that is significant, even if on their own, the problems don’t seem that significant. The evaluative scale the rankings are based on is probably not particularly suitable for the finding of averages—not that the resulting numbers are nonsense, but they don’t really mean exactly what we think they mean; there is no way to tell whether a given difference in mean score is significant; the ordinal scale that is the final product greatly exaggerates these differences.
3. I am inclined to endorse Zach Ernst’s point about Leiter’s sampling techniques. The issue is whether the sample is representative or not. In comments someone asks why you’d want the sample to be representative. You want the sample to be representative because you’re trying to understand how the philosophical community sees the departments—it’s a survey about reputations—and if the sample doesn’t represent the community, the results won’t represent the community’s views. The best way to ensure representative sampling is to collect the sample randomly. This is probably not feasible, which is why Leiter makes use of the snowball technique. But it does not seem obvious to me that the group of respondents accurately represents a cross-section of the discipline—most of the respondents come from and teach in highly ranked department, but not all “research-active” philosophers teach in ranked departments (some teach in unranked departments; some teach at SLACs), and not all of them graduated from top departments (some graduated from medium- or low-ranked departments; some graduated from unranked departments). If the advisory board is just going to invite people to participate, an effort should be made to invite philosophers from a wide variety of teaching and graduate-school backgrounds. The sample as it is currently collected appears to represent a judgment about what kind of philosopher will or will not have worthwhile opinions.