Wednesday, January 14, 2009

14% of people know that you can come up with statistics to prove anything

A few days late, and maybe a few dollars short, but you guys have been having so much fun discussing pedigree that I didn't want to ruin it. But, the time has come to move on. So, let's discuss the new PGR results that the Good Professor is slowly leaking out and that has us all on the edge of our philosophical armchairs (except for y'all experimental philosophers, who are at the edge of your surveys).

What's that you say? This doesn't really get us away from all the talk about pedigree, why it matters, and what these results and the survey itself has to bear on such a discussion? Fuck it. Let's tackle some of these questions:
Any surprises? Anyone sitting prettier now that these new results are out? Anybody worse off? Does my asking these questions drastically overestimate the impact of these results on graduate students? Is the Smoker simply buying into the PGR hegemony so many of you see out there (I'm staying out of that one) by linking to these results?
--STBJD

37 comments:

Anonymous said...

An observation:

The mean score of most of the schools in the top-15 of the 2006 PGR went down in the new survey. A few stayed the same. Only NYU went up (go figure).

I'm sure there are many different things that could account for this effect, but I do wonder what the real cause was.

Anonymous said...

P.S. Everyone who participated in the PGR surveys is implicitly endorsing the notion and value of pedigree.

PGRator said...

Good work, Yale.

I'm hard-pressed to see how Harvard deserves their ranking; perhaps the reason is that they're named "Harvard."

I thought USC was supposed to be moving up, but they've either moved down or stayed the same.

Anonymous said...

my university/ has never shown up on a pgr of any kind, ever. and i have received a top-notch education in philosophy. (not that such a claim is worth anything, since all measurements we have for these things are always reduced to omfg, how leiterrific am i?!)
but ohnoes! because i did not go to yale/nyu/colum/harvard/princeton/OXFORD (?!), i will never get a job, evah!!1!11

Anonymous said...

Anon 8:48,

Some evaluators may detest the whole charade, but feel compelled to play the game so that their evaluations of the strength of departments (at least, those strengths assessed by PGR, which leaves out a lot) is heard.

Paul Kersey said...

I think it is important to read a couple lines on PGR's "What the Rankings Mean" page:

"Before choosing any program, of course, make sure that the faculty there are committed to training graduate students. This Report only measures the philosophical distinction of the faculty, not the quality of their teaching or their commitment to educating young philosophers."

Now, I am completely and utterly opposed to the claim that one can measure 'the philosophical distinction' of anyone. This seems to be a case of making a qualitative statistic sound like a quantitative one. Distinct to those who make up the committee of PGR? Perhaps, but so what?

However, PGR is really supposed to be a tool for undergraduates who are looking into graduate programs. It is one tool among many, and it is for undergraduates/potential grad students who are seeking to continue their education in philosophy.

Here is the major problem with the PGR: it's devotees. These do not seem to be undergraduates, but certain factions of graduate students or faculty seeking legitimation for their departments. The report is not ranking the teaching abilities, publishing abilities, or even the 'brilliance' of any given department's faculty or current students. It is attempting to rank how 'distinguished' a department is, which supposedly will help people decide what program to which they will want to apply.

Another important line on the same page is: "Faculty quality and reputation correlates quite well with job placement, but students are well-advised to make inquiries with individual departments for complete information on this score. (Keep in mind, of course, that recent job placement tells you more about past faculty quality, not current.)"

'Correlates quite well' might be an over-statement (job placement can include any job, in or out of academia, at any type of university or college, etc). Note, however, that student should make inquiries into those particular departments in which she is interested. Just like the job market, the choice of a program is really going to come down to fit more than anything else.

The general intent of the PGR seems to be a good one: offer some kind of alternative tool (apart from U.S. News & World Report rankings) for initially helping students find programs that *might* be well-suited for their interests. I am sure this is an especially helpful tool to those undergraduates who go to tiny colleges that have little or no philosophy program.

The important thing here is that it is merely a tool for the initial search. It's not definitive until all of these faculty members and graduate students start regarding it that way, and the ones who take it the most seriously sound the most desperate and pathetic (I am specifically talking about those individuals who make claims, which I have heard far too many times, like "I think we're going to crack the Top 40 on the Leiter Report this year!") [Sidenote: Leiter is actually less statistically valid than Casey Kasem in regards to a "Top 40". Kasem was at least dealing with quantitative data that measured popularity via sales figures.]

I suppose that the intended audience for the PGR (undergraduates) is not the actual audience, which is unfortunate because the actual audience (certain faculty and graduate students) have turned a handy little tool into something that it's not: gospel (or anti-gospel) concerning the quality of their program.

If you really want to ascertain the quality of a specific program, then go and sit in on some of their seminars, read what the faculty and graduate students have written, and sit down and talk with the faculty and graduate students. Undergraduates should do this after they have narrowed down their choices. Faculty and graduate students should also do this if they are really interested in how their program compares with other programs. That information is not in the PGR.
-----------------------------------
"By "decent people," you mean people who can afford to live somewhere else."

Anonymous said...

Now, I am completely and utterly opposed to the claim that one can measure 'the philosophical distinction' of anyone.

Zoinks! Are you serious, or is this statement motivated by some idealistic desire to be more egalitarian or less judgmental?

It seems silly, if not just plain false, to say there's no distinction between the philosophical talents/body of work of, say, Bertrand Russell and any given faculty at my institution. Certainly, some comparisons may be more difficult, say between Kant and Hume; but that's not to say an argument or evaluation can't be made for one over the other.

As far as Leiter's disclaimer that the rankings don't measure quality of teaching, just philosophical distinction, I suspect that he's being careful here but does believe that the rankings say something (but maybe not a lot) about teaching quality.

All things being equal, it seems better to be taught by someone who is "better" than another, whether measured by win-loss record for football coaches or by publishing record and acclaim for philosophers. And if we assume all students are equal in their potential, it doesn't make much sense to think that an average philosopher can produce an outstanding student (again, assuming no large amount of innate talent to be a philosopher such that this student would be outstanding anyway); that student's growth is limited by that of her/his instructor (see Plato's argument on the same). However, a distinguished philosopher (again, cet. par.) does not have the same theoretical limit thrusted upon her/his students.

Anonymous said...

Some evaluators may detest the whole charade, but feel compelled to play the game so that their evaluations of the strength of departments (at least, those strengths assessed by PGR, which leaves out a lot) is heard.

But this does not mean they do NOT implicitly endorse the business of ranking philosophy programs; they just regret having to make some programs feel less worthy than others. Programs have different merits, strengths, liabities, etc. -- why would they not? Given these differences, we can rank programs according to some scheme. Now it is an altogether different question of whether those criteria that generate the rankings are the right ones or indicative of what they are asserted to mean.

Threadjacker #4 said...

What's up with job #382? I quote:

"For full consideration, applications should be received by 11/1/2003. Any being that is both actual and qualified, is strongly urged to apply. (180W), posted 1/14/09."

Obviously the date's wrong-or they want time-travelling philosophers, or the APA website is really fucked up. "Any being that is actual"? Is this a joke? Did someone steal the password for CSU San Bernardino and pull a hoax? I can't believe that someone in a philosophy department would write this ad, even if they are neo-Buddhist (I assume it's an allusion to the "all sentient beings" phrasing; the ad is looking for a specialist in non-Western philosophy, after all). I don't see the ad on San Bernardino's website (yet).

Asstro said...

There are many things I appreciate about the Leiter report, but there are a few things I think deserve a closer look. Among these things, one that bothers me particularly is a phenomenon that runs like this: Prof A is well-known in an AOS. Profs B, C, and D are less well-known, but still known and of high or decent quality. Prof A is in one department. Profs B, C, and D, are in another department.

I am a prospective graduate student hoping to study the AOS in question. It seems clear to me that I should want to study in the latter department and not in the former, despite the apparent difference in quality between Prof A and the others. Matter of fact, A would have to be _mind-bogglingly_ better than the others for me to consider the first department over the second. Moreover, to make matters more confusing, suppose that Prof A is in a department with several other distinguished professors, all of whom are working in areas that are not the AOS that interests me.

Why would I choose to gamble on Dept A when Dept BCD is more likely to provide me with what I'm looking for?

I can think of several higher and lower ranked departments where this is certainly the case. And, unfortunately, the ranking by specialization doesn't wash this out.

Paul Kersey said...

Anon 2:47.

The operative phrase is 'one can measure,' that is what I oppose, not that there is 'no distinction,' as you state. Of course there are distinctions, and I have no problem admitting that some philosophers have had a remarkable influence when compared with others. My contention really has to do with the standard of measurement: it seems to be completely contingent upon those doing the measuring, which undermines any claims to objectivity or statistic validity.

Has anyone seen any recent work on obsessions with rankings, btw? Just something I have noticed: philosophers seem to have hopped on this bandwagon of ranking EVERYTHING. Whenever I see yet another ranking of something, I cannot help humming, "Deutschland, Deutschland uber alles."

Anonymous said...

shrug, the measurements are basically ordinal. no one thinks we are counting philosophy-distinction quanta. it's more like, Arizona is overall better than Syracuse, about the degree to which the very top schools are better than Arizona.

and the measurements are "objective"--they are an statistical measure that objectively approximates of the aggregate views of the profession. how objective you think that is depends on how objective you think the views of people in the profession are. if you think most philosophers are radically wrong about most other philosophers, then of course one shouldn't take the rankings as correlating with anything beyond their views. but if you think that there is some (positive) correlation between the views of most philosophers on this score, then there will be some non-arbitrary relationship between the scores and the quality of the departments.

Anonymous said...

There are at least two reasons why attending a top program matters:

1) top programs get their pick of the most promising undergraduates. One learns more from one's peers in grad school than from one's professors. More often than not, top programs will have the brightest and most philosophically promising grad student population.

2) In order to contribute to current philosophical debates, one must know the state of the art. To read journal articles as they come out is insufficient. Once an article comes out, it is old news. Being at a top department means you will be engaging the current work of those who are making the biggest contributions to the current debates. One shouldn't underestimate the value of this.

So even if one can receive a fine education elsewhere, which i don't deny, they field of play is not neutral.

missamerica said...

Asstro,

What do you mean when you say the rankings by specialization don't 'wash out' the problem you highlight? Is it because of a problem with the specialty rankings, themselves? Or is it, as I'm inclined to believe, that infatuation with the overall rankings leads people to ignore the specialty rankings?

Incidentally, as one who was in a situation similar to the hypothetical you described, I concur with the general thrust of your comment. In my case, Dept A didn't have a true expert in my anticipated AOS (though it was strong in a number of my prospective AOCs). Dept A was however 15-20 notches up the overall rankings from Dept BCD. Once the dust clears on this year's job market, I'll be interested to see how I fared in relation to the candidates from Dept A.

Mr. Zero said...

Paul,

I don't get it. I mean, I guess I think you've kind of got a point, but were the Nazis big on rankings? Even if they were, you would have to think that this was one of their least objectionable habits, right?

You know that the holocaust was really serious, right? And that the PGR has nothing to do with anything like that, right? What's wrong with you?

Anonymous said...

Rankings aren't Nazi, I guess, but Paul's got a good point. All this ranking seems juvenile and counter productive, like a junior high school "friends and enemies" list. Why don't we just concentrate on doing philosophy instead of spending so much time creating and carrying on over these stupid lists that don't advance a thing, other than our egos? Not only philosophers, but everyone in our society seems to get pumped up about these retarded lists and rankings. My school/football team/salon/favorite book is ranked #12 on list xyz, which is higher than my neighbor's which is ranked #18! So what? Who does that help? Most graduate students applying to schools will know who is well known at these schools and who they ought to work with (if their advisors are worth anything at all). These rankings seem to just fill this intrinsic human need to feel better than the next guy, and create (meaningless) competition that has little to do with philosophy and more to do with stroking egos. Sure, some philosophers are better than others, but who does it help to make numbered rankings of this type? And now we're doing it with everything...

So much for philosophers as critics of silly trends in popular culture. We've gone in for this rankings thing full force.

Asstro said...

MissAmerica:

When I say that it doesn't 'wash out', my sense is that the appeal of Prof A often ends up being such a draw for some of the Leiter evaluators that they can't see the forest for the trees. This isn't really their fault, so much as a failure of the methodology. Evaluators aren't looking at the overall climate for research in a given AOS. They're looking at the prestige and quality of the top figures in the field. So, a particularly unavailable mucketymuck may pull an AOS ranking in a given department up very high, despite his/her unavailability; despite his/her relative minority status against the backdrop of the other strongly represented AOSs. All of this needs to be taken into consideration by graduate students looking at a program.

I guess I just have a strong sense that when someone reads that, say, Rutgers is a top ranked school in, say, ethics, that they may not be getting the whole story. I certainly wouldn't want to go to Rutgers to study ethics. (No offense to Temkin or McMahan -- any reference to them as 'unavailable' is unintentional.)

From the standpoint of those of us already in positions, we can see this easily: Sure, candidate X went to Dept A, but that doesn't mean as much as it seems like it might mean. Candidate Y studied with B, C, and D at department BCD, so they're more likely to be well-informed about their given AOS. They got a healthier exposure to their topic and they weren't sidetracked by other pressures to study in other areas. We know our fields, and we know what the climate is like at various departments.

But for prospective graduate students hoping to crack this nut, I think things are quite a bit foggier.

rankaholic said...

Anon 4:46 PM's comment was pretty good, but I'd have to put Anon 10:47 AM's above it. But neither is as good as 2:47 PM's, which started out fairly low in the rankings but moved up at least two places with the addition (albeit possessive) of "Leiter" in paragraph three.

Paul Kersey said...

Mr. Zero. The holocaust was serious? Quick, somebody call Mel Brooks. I don't think he knows either.

All kidding aside:

My point was not that PGR has something to do with Nazism, it was that whenever I see new categories of rankings come out, I simply wonder how long it is going to take before it becomes accepted to rank things like race, gender, etc. I realize that might sound completely unfathomable, but I honestly would not be surprised if it did happen. "Are Harvard graduates better than Princeton graduates?" is only a couple steps removed from "Are East Coast philosophers better than West Coast philosophers?", which is a few steps removed from "Are black philosophers better than Latin American philosophers?" I'm not saying it should happen. I'm not saying it is happening. I'm not saying it's going to happen. I am simply saying that when people begin throwing their judgment around by establishing rankings for more and more things, I just wonder how far it will go. Everybody's a friggin' expert.

Anonymous said...

I simply wonder how long it is going to take before it becomes accepted to rank things like race, gender, etc. I realize that might sound completely unfathomable, but I honestly would not be surprised if it did happen.

You honestly would not be surprised if it becomes accepted to rank things like race and gender, and you wonder how long it is going to take.

Paul Kersey, you are really wildly out of touch with reality.

Anonymous said...

Most graduate students applying to schools will know who is well known at these schools and who they ought to work with (if their advisors are worth anything at all).

Actually, I didn't know this when I was an undergrad applying to philosophy grad schools. I had no idea who was who in philosophy. None. It hadn't occurred to me that that was something I was supposed to know about. And while I had great advisors, because I went to an undergrad where the professors were really plugged into the profession (e.g. it was highly ranked on the report), I can imagine that I would have been completely in the dark at another school. That, I think, is the benefit of the report.

That being said, I think you should totally ignore the report after you choose where to go to grad school.

Paul Kersey said...

From Hannah & Her Sisters:

"You missed a very dull TV show on Auschwitz. More gruesome film clips, and more puzzled intellectuals declaring their mystification over the systematic murder of millions. The reason they can never answer the question "How could it possibly happen?" is that it's the wrong question. Given what people are, the question is "Why doesn't it happen more often?" "

No, I would not be surprised. I *would* be absolutely disgusted, but not surprised. Out of touch with reality? Regarding this particular subject, I truly hope so... but I remain cynical and wary.

cogitated said...

Paul Kersey - I really hope that you don't teach critical reasoning/thinking courses.

I guess on your view we should really just avoid any sort of comparisons or rankings because otherwise we might have another Nazi regime. Yeah, I agree with you that's a better idea. Oh wait, I just made an evaluation of idea. Shit, that means I ranked an idea. Fuck, that means we'll eventually be ranking races!!!

Mr. Zero said...

Paul,

You mentioned the Nazi national anthem, not me. I just pointed out that you weren't making any sense. Subsequent comments haven't improved the situation.

g said...

Now, while we're on the topic of ranking, what do we make of this:
http://www.philosophylists.info/GraduateDepartments.html
a ranking of departments by productivity?

Pat said...

I haven't bothered to read all of the comments on pedigree, but I just wanted to point out that Leiter has, it seems, justified the worthiness of using pedigree as a criterion of hiring with nothing but reference to pedigree.

Check it: http://leiterreports.typepad.com/blog/2009/01/pedigree-and-hiring.html

chris said...

My overall view is that the increasing fetishization of the PGR is not a good thing for philosophy or philosophers [especially younger ones].
However, I am interested in a narrower question about the PGR: is it unduly focused on certain areas in 'analytic' philosophy?
(I had thought this whole 'analytic vs. everything else' war was over some time ago, but I gather not.)

Anonymous said...

Wow, how did Pitt manage to inch up in the PGR??

They've lost several (four?) of their best mid-career people since 06, and the only new hiring they've done is at the junior level. They should at least have fallen below MIT and Michigan.

How do you explain that if we're measuring "philosophical distinction of the faculty"?

Clayton said...

Pat,

I don't see that at all. The gist of his post, "Where pedigree has substantial evidential value about the quality and character of philosophical training, and where pedigree also informs the interpretation of letters of recommendation, it would be foolish to eliminate pedigree as an important point of information in the evaluation of job candidates."

I rail about the evils of pedigree often, but I don't disagree with this point. It seems harmless enough.

inchworm said...

Anon 4:16,

Pitt didn't really "inch up" so much as Michigan "inched down". Which makes sense -- the people Michigan lost are probably more important than the people Pitt lost, for graduate students.

sweater said...

Dudes--this thread wasn't supposed to become all "meta." It was supposed to be petty and specific. So props to Inchworm and Anon 4:16 for their contributions. I have a similar major complaint:

19th Century #1: NYU.

Also Chicago, which I can see. But WTF with NYU? This might be a fringe area that others can't comment on...But it looks like it's based on Longeunesse and Richardson, who are awesome, BUT...

She does primarily Kant, who is NOT a 19th century philosopher, and of course she also does Hegel, who I guess counts, but he's not what most of us think of as 19th century. Most of us think of Nietzsche-Freud-Marx, Schopenhauer, those German Romantics who I know nothing about, and what else...maybe Feuerbach (I doubt there's any Feuerbach scholars per se...)...maybe Brentano...Meinong...

Richardson seems good on Nietzsche, BTW (though I admit I don't really know his work). But even if he is like the best on Nietzsche, and even if Longeunesse has written about ALL of these people (again, I think she's great), how on God's-green-Earth could NYU be a place for 19th century philosophy? Am I missing somebody?

I won't say who I think should be the best 19th century place, but I think I can say: something's rotten here.

Anonymous said...

Asstro:

Regarding Rutgers and ethics, you should note that Leiter includes moral psychology (including that of the x-phi bent) in the ethics category. So you're missing the elephant in the room - Stich.

Anonymous said...

chris:
yeah, but the promulgator of pgr believes firmly that "analytic" philosophy does not exist, and that he is a specialist in continental philosophy.

Asstro said...

Still, I would neither go to Rutgers for ethics, nor would I recommend it to any of my undergraduate students. Not to say I don't sometimes recommend trying to get Rutgers faculty as outside readers on dissertation committees -- no question, some of their faculty are superb -- but only that if a student comes to me and asks where to go for ethics, even x-phi, Rutgers is very low on my list. The climate is just not conducive to the study of ethics.

I think that's a significant shortcoming of the Leiter report. There are many departments where the influence and prestige of a few faculty members skews the department ranking away from a more natural observation about the climate for a given AOS. Rutgers is a great M&E department. It's not a great ethics department, despite their stars.

Now, if the Leiter report is just ranking the influence of departments on the general philosophical community, as I think perhaps it is, it may be pretty accurate. But as a guide for prospective graduate students, which it is probably most often used as, I think it can be misleading.

genuinely curious said...

Asstro,

Can you explain why the climate at Rutgers is "not conducive to ethics"? I see four *very* good ethics people on the faculty.
Do you mean that the ethicists there would not be good to work with, or is there some other feature of Rutgers that makes it a bad place to do ethics?

I think it's true that a limitation of the PG is that it only tells a student how strong the faculty is in an area. If there are other important considerations, the student is going to have to find out about them elsewhere.

tap said...

Sweater:

Is there some compelling statistical evidence backing up your claim that "most of us" do not think of Hegel when considering 19th Century philosophy? You view is, to most of us, patently asinine...

Nonetheless, I agree with your main point. I think Longeunesse may be working more on Hegel and German Idealism than Kant these days--but having only two people in one general area, when the main thrust of their work is on pretty different sub-areas, does not amount to an overall strength.

Anonymous said...

It is by stipulation that Hegel doesn't count as part of the 19th century. Leiter lists the 19th century category as excluding Hegel.