Wednesday, 24 September 2014

Thoughts on university rankings

I've been thinking about university rankings.  In light of the latest hullabaloo in the online philosophy community about whether the Philosophy Gourmet Report (PGR) should continue, I wanted to drink some scrumpy, and express the rather conflicting feelings I have on the matter of rankings.

*N.B. I'm not going to voice an opinion on the more personal aspect of this whole affair.  I have a view, and I think it's the obvious one in light of the information I have, but I don't want to perpetuate what must be a very distressing affair for certain people.  My thoughts are about what the PGR is, not how it is done or who does it.

As a (soon to be ex-)grad-student looking for their first job, I feel like my application for any job will be judged based on the school I went to.  As someone who's already paranoid about being judged on non-scholarly grounds, this causes me no small amount of trepidation (even though I went to what I consider to be a very good school!).  Even if this is not the case, I feel like it is, and I know for a fact that I'm not alone in thinking this.  A lot of grad-students lose a lot of sleep worrying about whether their school is good enough, and whether the school they get their first job at will affect their chances of good employment further down the line. With this perception, the damage is already done.  Even when I get  job, I'll feel like the school I'm working at will have in impact on how I'm viewed as a scholar.  For many, this will extend to how they perceive their worth as a scholar (not just how they think others will perceive their worth).

I think the presence of rankings exacerbates this problem.  I obsessed over the PGR when I was applying for grad-school, and since then I've seen more than a few other people do the same.  The presence of rankings encourages students to think of themselves and their peers in terms of those rankings.  A student at a middle rank school may end up thinking of themselves as a mediocre scholar (and conversely, students at top ranked schools may hold the default assumption that they are therefore better and more worthy than those at lower ranked schools).  Indeed, even the application process reinforces this; when someone from a middle ranked school is reluctant to apply to high ranked schools simply because of their institution rather than their ability.

The contrast to this is that prospective grad-students DO WANT this kind of information.  When I applied to grad-school I was very aware that where I ended up could affect my future career, and I wanted to get into 'as good a school as possible'.  Whilst I confess to engaging in that kind of thinking then (and to some extent even now), I really don't think it is helpful.  It contributes to the kind of authoritarian thinking and smug elitism that I (and I hope many others) take philosophy to be against at its very core.

Now, idealism is one thing, but the fact is that the world we live in is the way it is, and a PhD from Oxford is given more value than one from a school lower on the rankings.  Choice of programme IS important, but it should not be because of how a school is ranked.  Having supervisors who support and challenge you, and a lively and engaged research community, are by far the most important things (apart from a well subscribed library).  There's nothing to stop any department from having these features.  Indeed, different students thrive in different environments, so there's no one set standard by which to judge how well a department will serve a student's interests.

I genuinely can't imagine that I could have gone to a better (for me) school.  My grad-school experience was fantastic.  I had amazing peers and supervisors that helped me to unlock potential I didn't even realise I had.  No ranking or external report could have told be that in advance, and there are many schools higher in the rankings where I'm convinced I would have ended up as a worse scholar than the one I am now.  Now, here is the obvious problem for all prospective grad-students:  How do I end up in a school that's so well suited to me?

Sorry folks.  I just don't know.  The reason why I am so conflicted is because I recognise that the choice of department is important, and being the intellectually rigorous folk we are, we want to make informed decisions.  I don't know how we can provide prospective grad-students with the information they need to make good decisions, but I really don't think rankings are the way to help people find the programmes that are right for them, nor do I think it is helpful to the profession as a whole for someone at one school to be able to look down on someone else for being at or from a "shit school" (oops!  Gave myself away, there.  Consider it a reward for reading till the end).


  1. PART II

    So when I went to apply to graduate school, I knew I could not stay where I was. I would have gotten a good degree from good people and been well educated. But (odds are) I wouldn't have found academic employment and I would have been in serious debt.

    I found the PGR on the internet and learned that my best bet for a career in academia was to get into a ranked school. I knew that I had to publish early, attend conferences, and get up-to-date with my field. I worked extra hard because I knew how far up I had to jump. Publications, funding, and conference presentations were the only way to distinguish myself from the other 10,000 students trying to do the same.

    I ended up in Leeds, with which I was perfectly happy. I never once worried about ending up at school #20 in my field rather than #12 or #6 (both of which accepted me). Maybe if #1 had been an option the precise rankings might have swayed me, but really I wasn't worried. I was going to get a degree from a place that people had heard of and was what really mattered. The rest was up to me. My decision came down to the friendliness of the dept, meeting my supervisors, the availability of financial aid, and the size of the PG population. (Sidenote: Why did I think about all of those things? Because an article on PGR told me that *these* were most important, not minute differences in ranking.) I think my use of PGR was extremely useful and productive.

    The way you used PGR was unhealthy. We probably should be educating our students about how to use it in healthier ways. Nobody should be worried about tiny differences in ranking, particularly given all of the things you mention, plus the speed with which academics leave one school for another (something which can affect ranking by 10 or more places).

    You are in the academic upper class. It might not feel like it, but you are. You have opportunities and privileges unknown to most philosophy students. Members of the upper class needn't worry about the rankings. They're already at the top. This is a point that gets made on the PGR, but should probably be made more loudly.

    Students in top50 programs probably represent <5% of all of the philosophy PGs out there. There are literally thousands of philosophy departments out there that you have never heard of, charging lots of money for PhDs that will be intellectually rewarding but professionally unhelpful. For people considering these sorts of places, PGR is a great resource that helps them assess their career prospects and plan accordingly. If they really want the PhD for the education and not to seek employment, then great! More power to them. At least they'll know that that's the choice they're making. If somebody looks down on that person for that choice well fuck them. They're an asshole and are going to be an asshole whether the PGR exists or not.

    For too long students have self-funded PhDs at institutions that are statistically unlikely to land them academic jobs, only to find themselves bankrupt at the age of 35 after years of under-employment (I know two such cases). Do they deserve that? No! And something like PGR is a good tool for knowing the likelihood of said outcome.

    In terms of your graduate experience: you're right. It was awesome. And there are top50 departments that are not nearly so awesome. But you'd be a lot less happy about how awesome it was if nobody from your program had ever landed a job, and you'd all gone through the program under the false assumption that you'd all be working as profs in a few years. So part of what makes it awesome is that it is home to the sorts of academics who provide the sort of teaching, networking, and pastoral care that lands students postdocs and teaching appointments.

  2. PART I

    Preface: I'm going to bracket worries about Leiter, since I think we agree on those. My interest is in the existence reputation-based ranking systems like PGR.

    You are going to be judged on the basis of your school whether PGR exists or not, whether you are in philosophy or not, and whether any formal ranking system whatsoever exists or not. Pedigree evaluation exists across academia, yet philosophy is the only field with a formalized procedural ranking system.

    The fact that this judging happens at all might worry you, but in that case we should worry about the psychological phenomena itself, not whatever tools we happen to use in exercising it.

    If anything, tools like PGR make certain that said judgings are actually based on something. For example, in many fields ivy-league degrees are given much more weight than anything else. This is a problem when those ivy-league schools in fact offer slightly inferior education than some other less-recognized schools. This is less-likely to happen in philosophy, where the pedigree of the school has very little impact (since its about the actual faculty in the dept now, not the perceived long-run reputation of the university as a whole).

    It is because of PGR that schools traditionally overlooked can be highlighted for having excellent departments. I bet there are hundreds of schools in the US that come to mind before the University of Connecticut Storrs (wtf?) or the University of Missouri at Columbia. But both get recognized for their philosophy programs because of PGR. And PGR is probably a main way in which those departments seek support from their administration. How would they convince their admin that their department is doing well? Compile lists of citations? Thats a crap metric and takes a lot of time and effort to compile. They can point to PGR fairly easily and convincingly.

    A second problem you highlight is the fact that students worry too much about the school to which they apply. Here are two possible cases. The first is you, the second is me.

    So you were at a top 50 school, worried about which of the other top 50 schools in the UK you should attend. This caused undue harm, since you would have been better suited worrying about finding a nice department, a loving supervisor, etc. In this instance, you obsessed about the rankings too much. I won't re-hash the details of your story.

    Now me. I was at a school nowhere near PGR contention. Never has anybody thought about including the school where I did my undergrad in the PGR. I had excellent profs, received an lot of attention, and was surrounded by excellent students. The people who taught me were not engaged in the latest research, were not publishing ground-breaking research, did not always go to the best conferences, etc. In short: they were not high on the inside track of their fields. They were wonderful, smart, teachers, but they were no superstars (the sort of profs we know by the dozens, the sort of people that most graduate students end up becoming).

    Many of the people who taught me did not have full-time jobs. They earned $4000/course and lived in abject poverty their entire lives. They all had PhDs from good but not great institutions -- institutions like my own. They did not have a PGR when they came up and so had no idea what was in store for them when they graduated. They had no idea that many PhDs do not get full time jobs, and they were under the false impression that their small provincial schools were among the best in the world.

  3. (Please excuse typos, the comments box is really small and makes proof reading difficult.)

    That's a very compelling argument (even for someone from a lesser school! *cringes and feels dirty at own joke*), and I can't help but agree with pretty much all of that. I guess the sentiment of my post comes from two sources. First, as you point out, a certain degree of naivety (I think you sum that up pretty accurately). Second, some more idealistic tenancies. Whilst I accept that students need this kind of information given the current system, it seems a shame to me that we should be in a system where getting a good education and being a good scholar (something which I can't imagine top-50 schools have a monopoly on) will leave you in a worse off position career-wise. Now, I realise that real life is far more complicated than that, and that top -50 schools are where they are largely because of the distribution of scholars, but... Well, I'll stop on that point before I feel obliged to start wearing a tie-dye headband.

    A point related to this discussion is that there are methodological issues. I'll ignore the claims that there is bias in the methodology of the PGR (if there is, then I think it's uncontroversial it needs to be excised). But it seems to me that there may be alternative ways of gathering and presenting the kind of information that prospective grad students need without resorting to something that is so explicitly hierarchical. I've seen a few posts around about student feedback surveys, or transparent employment statistics, and various different proposals for how we could give information that would be helpful, without contributing to a culture where someone is judged by the reputation of their school before their quality as a scholar. Now, I don't think I'm in a position to judge how well options like those would work. I don't have the experience or the insight to venture an opinion of how this kind of info should be gathered or spread, but I do still have some doubts about whether the current form of the PGR is the best way to go.

    Like you said, it's great that prospective students have that kind of information, given the harsh realities of the job market, but if by giving this information in a certain way we perpetuate those harsh realities, then something needs to change. Maybe that's the PGR, maybe it's not, but I think the discussion is definitely worth having (so thanks for having it with me!).

    P.S. That is BY FAR the longest comment anyone has made on my blog. Thanks!

  4. I'd like to focus in on the claim that PGR 'contributes' to the ranking phenomena. What might this mean? Let's think counterfactually.

    So 'contributing' shouldn't simply mean that people cite PGR rankings when making comparisons. Eliminating PGR will replace reference to it with reference to something else (like perceived reputation, historical reputation, wealth, etc.). The existence of comparisons in other fields is proof of this.

    'Contributing' might mean that elimination of it would not permit such fine grain comparisons. That seems likely. Though people might turn instead to one of the far worse and *more* fine grained ranking systems like QS or THE, etc., which give a terrible idea of the state of our field. Nevertheless this fine grain thing is interesting. I'll return to it in a second.

    'Contributing' might also mean that elimination of PGR would mean less comparison overall. Also possible. No PGR might mean people were less-aware of difference in general. In this case, we might wonder whether the benefit of eliminating the 'contribution' is outweighed by any costs. I suggest it would, for the reasons detailed in the previous post(s).

    Now back to fine grain. PGR is most useful for specialist rankings. I regard the overall rankings as a curiosity, but I don't imagine any intelligent person would recommend using overall rankings as a basis for choosing programs (even BL advises against this). Specialist rankings are better. And those are not fine grained. They are broken into groups, using rather sound statistical methods.

    There are usually four groups. These eliminates the grain problem. Nobody should care that Leeds finished 0.5 above Maryland in phil bio. The difference reflects margin of error in the ranking system, not genuine difference. The difference is so small that it might change once someone at Maryland publishes a new paper, or someone leaves Leeds, etc. The difference between Leeds/Maryland and ANU/Duke, on the other hand (groups 4 &1) does reflect genuine difference. There are more scholars, higher-quality scholars, etc. at the group 1 schools than the group 4. (mostly these things are affected by #s). That's worth knowing. For a graduate student who has no idea who most people in their field are, this is useful information. So presented with offers from a number of schools, the fine grain differences should be overlooked as a function of methodology, but the group differences should impact my decision a bit more, subject to BL's caveat about finding a 'good fit', mentioned above.

  5. Okay, so to answer your question, I was thinking of the contribution of the PGR more in terms of the way you describe in your fourth paragraph. When it comes to whether the costs outweigh the benefits again I must confess that I don't think my opinion in worth much (for reason of ignorance of important relevant information), but I think you do make a very compelling case, and I'd be interested to know what a more informed (than me) interlocutor might say in response to you.

    I don't think I have more to add without going in circles, and I think you've probably convinced me, but I reckon I'll think about it a bit more first.