Tuesday, October 19, 2010

Does Letter Quality Correlate with Pedigree?

As a weird coincidence, these two comments showed up overnight, making the same interesting point:

As someone who has chaired a number of SCs at a highly competitive SLAC, [snip] For all the righteous indignation about the role pedigree plays in hiring decisions, I will say that the letters from faculty at more highly ranked institutions are, as a whole, an order of magnitude better than those from less selective places. The level of detail, engagement with, and subtle assessment of an applicant's work and potential is--by and large--much more impressive when the letter writer is from a top place, and I think that plays far greater a role in hiring decisions than many people acknowledge. It's not just the aura of the Leiter rankings--it's also the quality of the letter that come from the more esteemed departments. This disparity in the nature of rec letters is likely unfair to those very smart people applying from programs lower down the food chain, but I think it's worth noting in the context of complaints about how pedigree exercises an undue influence in the application process: one additional advantage to being at a higher ranked program is the strength of the endorsement letter you receive from your recommenders.


and

This might be a tangential remark, but in the context of various complaints about the role that pedigree plays in the hiring process--an admittedly fractious issue--I wanted to note that there is a high correspondence between the rank of a department and the quality of the letters from an applicant's recommenders (I speak as someone who has chaired three searches in the last five years). It's not just the reputation of the institution an applicant comes from--it's also the level of detail about, engagement with, and subtle assessment of an applicant's work that comes from a highly ranked department's letter writers. Given the importance of recommendations in the application process, it's important, I think, to keep this in mind when raising complaints about the supposedly undue influence pedigree plays in hiring: it's not just the aura of the letter writers, but the informativeness of the letters themselves, that plays a key role in why applicants from better-ranked places do better on the job market than those from less prestigious places


Smokers with search committee experience, what do you think? Is that true?

--Mr. Zero

19 comments:

Kevin Klement said...

Of course, for every generalization, there are many exceptions, but as a crass generalization, yes, I think it's true. The explanation is obvious. People who have more experience writing letters and reading letters know better what it takes to make one successful. If you're on a graduate faculty at a school that has a high ranked graduate program, you have a lot more experience reading these (e.g., on grad admissions committees, plus hiring committees for what tend to be larger faculties with more turnover), and writing them (for their own students).

Chris said...

Coincidence? Clearly written by the same person (maybe because of comment screening or something). A bunch of the phrases are exactly the same.

Elizabeth said...

"Weird coincidence"? They're clearly by the same person (check the repeated phrases describing the good qualities of letters from prestigious places)... possibly someone who didn't understand why their comment didn't appear at first.

Now back to lurking, because I'm interested to see what people say here. In particular, I'm interested to see whether it's only philosophers from elite schools who think the letters from elite schools are better, or whether philosophers now housed in non-elite schools do too (and whether that depends on where they were trained).

Xenophon said...

Given the near-exact repetition of the phrase "level of detail, engagement with, and subtle assessment of an applicant's work" and other similarities in the comments, I think this was double-posted by accident by the same person. This is not a criticism of the poster or the (interesting, and plausible) content of the post, just a comment about the fact that two people apparently said the same thing in the same night.

Anonymous said...

I'm not sure what to say here. Now that I see the comment in question, it does occur to me that most of the letters that I remember and all of the few that wound up being important in my deliberations were by especially prominent philosophers. I'd like to think those letters are the memorable ones because they were the most informative. However, I worry that I'm deceiving myself, and that what's really going on is that the letters are memorable because of my positive assessment of their authors (as talented and deservingly famous) and not because of the virtues of the letters.

That said, I generally didn't find letters very helpful, and some of the very praiseful letters from talented and famous philosophers fed my colleagues' suspicion that their subject would be a flight risk.

Anonymous said...

I have not found a strong correlation between letter quality and leiter rank. I've been on four search committees in 8 years and I'm on another this year.

In a couple of cases letters from top places have been amusingly hyperbolic and not so amusingly insensitive to the reality of teaching in mid-level universities.

Anonymous said...

I've been on a few SCs. It's true--folks at top departments put more into their letters. I want to even go that step further, and say that top departments work harder on placement in general.

I'll also second that letters from good people mean more. Quite frankly, how couldn't they? I know everyone is up in arms about pedigree, but the fact is that if the best folks out there say you are damn good, we believe them. Does that mean Joe B with a great letter from someone at a second or third tier place isn't just as good? No, it doesn't. But I know that the first one is good, and I have to guess a little with the second. I'd like to see the second if there's room, but at our R1 it's the first we want to see.

And that, when all is said and done, why pedigree matters. THose who've gotten themselves into top places, and have proven themselves there have demonstrated that they are well trained and have lots of potential. Others surely have it too, but better the devil you know...

Morlock said...

Before taking such a generalization wholesale, there is a definite possibility that the fallacy of confirmation bias takes place in such an assessment. The letterhead or name of the letter writer could influence one's judgment regarding the content of the letter. Given the expectation that such letters will be better, they may be judged to be better based upon that expectation. Only if letters could be blind reviewed would such generalized assessment be worthwhile.

KateNorlock said...

I've been on five search committees(and in some freaking awful economies, so, sadly, have read hundreds of letters) and found a weak correlation between rank and letter-quality (with some horribly memorable exceptions, oy). But given all the research out there about bias, even bias that I'm trying to resist, I'd like to see a really super study of completely anonymized letters which are then 'ranked' in quality, and only then have their origins revealed. I wonder if my weak correlation would hold up to such an analysis.

Still, what Kevin Klement says makes sense.

Popkin said...

1:09, I don't think "everyone is up in arms about pedigree" as such. Most people who complain about the influence of pedigree aren't angry that pedigree counts at all; they're angry that pedigree counts so much more than other things like publications or quality of written work.

Anonymous said...

I've taught at two highly-ranked PhD programs, and at both of them my recommendation letters have been vetted by the placement officer: he read my letters and then sent them back to me with comments. In one case, a letter I wrote did not do a good job of clearly describing a student's work and I substantially revised my letter in light of that. In another case, I was urged to add more detail. This process made my letters considerably better.

This is one of the ways that highly-ranked PhD programs end up with good letters.

Anonymous said...

I think 7:07 has much of the explanation: In my experience, the best letters come from schools that have "placement services" in a serious way.

But there is another issue that may be more sensitive to pedigree. The leaders of particular subfields tend to write letters that speak more authoritatively about the lay of the land. To take a random and fake example, if I got a letter from van Fraassen about how Candidate X's work moves the literature on empiricism forward, it would very likely be accompanied with a synoptic view of the state of play and the major players. While anyone could write this, it would be natural for him to do it and would probably be believed by the committee. Since search committee are typically not experts on the relevant subfield (that is usually the point of the search), this kind of information gets relied on whether warranted or not.

I haven't been on a committee in a few years, but perhaps some colleagues could try to remember letters from very prestigious people in less prestigious departments? This would test this hypothesis to some extent.

Kiki said...

Thanks 7:07, very interesting info. I wish my department had an effective (or even mildly caring) placement advisor.It is not surprising that excellent programs tend to be excellent on so many dimensions!
Discussions on this blog ( typically very informative) tend to seriously dampen my spirits and kill my self-esteem as I think about my dossier on the job market.

Anonymous said...

Aren't these letters supposed to be *confidential*? If so, is it ok to let a 'palcement officer' see and comment on them - and make this an institutional practice? Excuse my naivety...

10:36: I wish I had at least one piece of advice about the job market while finishing graduate school. Where I studied, finding a job after graduation was treated as a private matter - as private as whether one was going to marry or have children, for example.

Anonymous said...

I think that all of the worries about bias here are misplaced, because of the nature of the real issue at hand. What is really pertinent here is how detailed letters are, and there is no direct connection between such detail and matters of pedigree (the original posts are, I think, noting an accidental connection, i.e. that people from places we associate with pedigree write detailed letters).

First, in my limited experience (2 SCs in the past three years), there is a rough correlation along these lines. But what is really important to note is that some letters are way more detailed than others. Many letters are slightly expanded versions of the kind of thing I would write for a kid applying to law school--they are around two pages, have a paragraph or two about the candidate's academic abilities, a paragraph about what a nice person the candidate is, and maybe something about teaching. I have read other letters, though, that are 4-5 pages long and include things like detailed descriptions of the candidate's dissertation, assessments of the candidate's current or potential place in the relevant debates, and in some cases very detailed teaching reviews as well.

All of the stuff I have mentioned that potentially goes into a detailed letter (and that is in no way meant to be an exhaustive list) provides real, and very useful, information that can clearly be untainted by issues of pedigree. This stuff can also be relatively untainted by the usual hyperbolic rec letter code language.

So the main point here should not be to worry about pedigree, but to worry about whether your letter writers are writing detailed letters. Obviously sensitivities are different about this kind of thing with different people or at different places, but I would recommend in general that if possible candidates speak candidly with their letter writers about this issue (I hope it isn't too late, though I guess it might be)...

zombie said...

The placement at my school is pretty sucky. The department's placement advisor changes every year, and most of them are fairly bad and uninvolved. No one has ever vetted my letters. I've been tempted to order them myself from Interfolio so I can have a look at them. Not that I could really tell if they're good, but I suppose I might be able to tell if a letter was really bad.

So, if department prestige correlates with quality of placement services, then that correlation holds in this case.

Anonymous said...

First year job-marketer here, so no experience on SCs at the faculty level, but did some informal SC work for our school's graduate student group.

I'm wondering if the correlation, which we seem to be in rough agreement on, may also be explained by the demands on faculty at top schools vs. the rest. If one of my letter-writers is teaching 2 undergrad classes, one grad class, advising 10 students currently applying to law school, organizing the department's colloquium series and finishing a book with a colleague in Europe all while trying to have a personal life AND stay sane, I expect that my letter isn't going to be as good as someone who has a lighter teaching and service load or administrative/research support.

Having an involved placement director has helped immensely, as she has put pressure on faculty to get letters in (relatively on time, some of my apps are past due), and corrected a discrepancy between two letters on my defense date. She has been known to send letters back for rewrites, call faculty at home to get overdue letters and she vets any and all application materials we want her to. My school has a decent placement record with SLACs (for being non-Leiter ranked) and I think part of it is her commitment to the process.

Anonymous said...

I've said before, and I'll say again:

I don't know that attempting to systematize any aspect of the job market is practical or even possible. I've been on one SC for a CC TT Philosophy job, and one SC for a CC TT non-Philosophy job. Here are my observations.

The reference letters for most candidates from either Leiter ranked programs (for the phil job) or schools with a notable pedigree (for the non-phil job) were completely unsuitable to the job in question, whereas letters from the "lower pedigree" schools in both cases were better suited to the job in question. What this suggests is that (obviously) whether a letter is good or not depends on what the letter is for.

I read lots of cover letters from candidates from top-notch Leiter programs, and reference letters from people who I could only dream about working with. But none of them served to convince me that the person would be good //at this job// because none of them had anything to do with teaching. So even though they may have been excellent letters for other kinds of jobs, they were crap letters with respect to our search.

It seems clear that highly skilled researchers are may be better able to talk about research, and being on faculty at a place which either churns out or turns one into a highly skilled researcher might mean that one is likely to write a better letter on that topic. On the other hand, it seems most people from these programs -- either referees or candidates -- have no idea how to talk about teaching; those from "lower" ranked programs (which are likely to put a greater emphasis on teaching) do, and thus the best letters //for our search// came from those places.

Just a small point worth mentioning -- basically, the quality of a letter can depend on things other than how much the faculty member talks about the person's research, especially for all those non-research jobs; and in many of those contexts, discussions of research (as opposed to teaching) can actually hurt a candidate, since at some jobs research really, truly, genuinely is totally irrelevant (I know, gasp) and we all know there is absolutely no correlation between a person's research capability and their teaching capability (and in many cases, there may even be a negative correlation).

So there's that.

Polacrilex said...

Totally off the topic of this thread (except for the recommendation letter request), but you must post this: http://www.xtranormal.com/watch/7451115