Academia: Rankings and Nonsense

Thursday's Child Is Saying the Same Thing as We Say Every Year

At this point, the US News and World Report rankings are rather like a horcrux in Harry Potter in that the parent-named magazine is dead and the conceptual era that produced the rankings is dead too. But it’s the last bit of value that the remnant media company retains, so it is keeping them alive in this brave new online world.

Just as the company annually issues its ratings, so too does higher education grouse about the many varieties of inaccuracy and irrelevancy that are baked into the ranking system. But no more than grouse, and often that relatively quietly in terms of institutional leadership. If you’re at the top of your rankings category, you know that’s helping you recruit ever-larger number of applicants that you get to reject in ever-larger numbers; if you’re moving up, it’s making a difference in meeting your goals and maintaining tuition revenue. If you’re at the bottom, it may focus your strategic efforts on one manipulable category that might move you up.

Don’t get caught manipulating anything, even though every institution in the country knows how to quietly goose their data. The people maintaining the rankings are fiercely vengeful if you bring negative news to their doorstep. Columbia did not suddenly become only the 18th-best institution of its kind this year; US News is slapping them hard across the face for being so stupidly obvious about cooking the books that a faculty member was able to see it. Reed College is not #72 of small liberal-arts colleges: it’s being made to sit in a permanent punishment box for refusing to cooperate with the rankers.

I can—and have—go on and on in this vein for a while, and so can most faculty and administrative leaders. The bad thinking about proxies for data nobody has are evident, and the refusal to track some forms of data that might matter to applicants and their families is infuriating. And there’s just something basically silly about the implied precision of the rankings: a whole host of institutions that produce basically similar outcomes for most of their students end up sorted into immense hierarchies that imply vast distances between them.

Let’s ask instead why the rankings continue to be used by prospective students and their families, at a minimum to structure their initial decisions about which colleges and universities to consider applying to. Imagine a world where there were no rankings at all, just the huge range of available, public data that many colleges and universities offer (with varying degrees of accessibility and discoverability). Families with two or three generations of college education would know something of how to sort between hundreds of colleges and universities in terms of the most crucial kinds of information. Families where at least one person in the extended family works or has worked in academia would be able to sort still other kinds of information, including some of the informal and reputational narratives that US News ineptly tries to use as the secret sauce in its rankings. Families where the applicant’s prospects and interests are very strongly divergent from the previous generation, on the other hand, might well drown in the vastness of the choices they’re facing.

Rankings are a structure that we all understand how to work through. If you know there’s a feature or variable that you weight differently than others, you can sift through rankings to find what you really want, even if it’s lower down. If I’m going to use Wirecutter to help decide what lawn mower to buy, if the rankings system is built well enough, I should be able to sift for price, for electric versus gas, for longevity or ease of use, and so on, and get what I know I want. If it’s built well enough it might also educate me about what the key variables in my product category are and why they exist. (Or equally to inform me that there are no variables that really matter, that my product category is nothing but the same device made in the same way with meaningless cosmetic differences.)

What do applicants and families need to know about colleges and universities that a meaningful, well-tended and relatively transparent system might try to evaluate for them? And does that have to be about rankings?

1. Can I get in? (This should be first, because why bother looking at the rest of the information if the answer is no.)

2. How does financial aid work at this institution? What offer might I receive, how much is it actually going to cost for me to go there? Does it cover costs and fees that I might encounter during the year, if I am receiving aid? If not, what are those costs? How much are they? What’s the cost of living in the area around the institution?

3. How hard is it to graduate from this institution (e.g., what are its graduation rates?) Are some programs of study at this institution harder to complete than others? Are some parts of the curriculum gated off or restricted to a subset of matriculated students?

4. Roughly speaking what are the outcomes of being a graduate of this institution? How satisfied are graduates, what kind of reputation does this institution have in the wider world, what are the employment outcomes 1, 5, 10 and 20 years out for graduates?

5. What’s it like to be a student at this institution? Are students happy or unhappy on average? Are some kinds of students happier or unhappier in relative measure? What do students do during the week and the weekend? How much is there a match between the kind of person I think I am at 17 and the kind of people the students are? Is there a single ‘culture’ or many ‘subcultures’ at this place?

6. How well run is this institution? How well maintained are its facilities? (What facilities does it have?) How financially secure is the institution? Is it obsessed with cost-cutting or are there plenty of resources to do the things I want to do? Are the administrators I am likely to deal with skilled, friendly, and responsive? Do they share my values, goals and beliefs?

7. How big is this place physically? How does it relate to the communities around it? What’s it like to be there from September to May? How hard is it to get to the place generally and how hard is it to get there from where I live? Does it offer housing to undergraduates? What’s the housing like if so? (Typically, normally, for each year, not ‘this is the nice dorm we pretend everybody gets to live in’.)

8. Can I study the things I think I want to study? Are there lots of curricular choices available so that if I change my mind about what interests me I have lots of options? If not, is there a coherent alternative vision about why the institution and the curriculum are the size that they are? Are there are lot of faculty in the areas that interest me now? In areas that I don’t even know about that might become interesting to me later? Are the faculty available, inspiring, engaged by the students and the institution? Are they disaffected, disengaged, burnt-out or working under highly unfavorable conditions? Will the faculty member that I have made a strong connection to be there in following years or are they on a short-term contract? Will I just be stuck talking to teaching assistants for most of my time? Are the departments and majors I am interested in accessible, engaging and responsive? Will I be supported if I turn out to be less prepared than other students? Are the faculty snobbish or discriminatory? Are there lots of “weed-out” classes? Are there pre-professional or vocational courses in addition to more academic or liberal arts courses? How real are the programs and departments that appear on the website or in the catalog?

Do the US News rankings help with any of this? Barely. Not really. Only if you have a really sophisticated understanding of how to use what they do measure as crude proxies for some this information.

Do colleges and universities generally provide data and information that could be used to answer some of those questions? Here and there. Sometimes not at all, because the answers are so qualitative and subjective that there couldn’t be a definitive answer anyway. Sometimes they’re holding data that’s relevant that they absolutely don’t want to share (say, qualitative assessments of student, faculty and staff satisfaction, exit interviews, etc.; often they don’t want to share anything about the formula they use to assess financial need when they make aid awards). Sometimes institutions are actively pushing out deceptive information on these questions—for example, university websites will sometimes make some programs of study look much more regularized, supported and viable than they actually are, and a skilled reader can frequently detect that if they look hard enough.

Part of the problem also is that a really informed approach to making this decision takes the applicant and the family doing a kind of thoroughly introspective audit of their own perceived needs and desires when it comes to going to college that is simply improbable for almost everybody. Parents and children frequently can’t talk straightforwardly about how they are thinking about this decision: parents or families may not want to disclose the reality of their financial situation, they may not be fully aware of the anxieties or biases they have regarding the future lives of their children, they may have some harmful or dysfunctional views of the whole situation. 16 and 17 year olds may genuinely have no idea of what they want from universities or life in general, they may quite reasonably have no idea of what to study, they may have some serious misinformation about college in mind, or they may be actively wrong about their own preferences in a way that can’t be fixed until they’re actually in college.

To some extent it’s the kind of decision that you can’t get clarity about until you’ve already done it. A lot of this information only makes complete sense when you’re finished with college, and either informs your satisfaction with the experience or your bitterness about having made a bad choice. I know what I’m looking for in a lawn mower because I’ve bought enough regrettable, shitty, malfunctioning lawn mowers in the last thirty years that I know what to look for. You only buy an undergraduate education once or maybe two or three times (if you start over or transfer).

If I were going to start to build a compact, easy-to-use evaluation system from scratch that spoke to the things that really matter, I’d work from easiest to hardest:

Easiest:

• How big is the institution?

• What’s the weather like?

• How hard is it to get there, to get around once you’re there?

• In basic terms how wealthy is it? (Size of endowment, budgetary model, revenue sources).

• How selective is it? (As proxy for “can I get in”?)

• Graduation rates and average time to graduation

Medium difficulty (may require using public data in ways that are not enabled or encouraged by colleges and universities):

• What’s the average financial award? How many students are receiving some form of discount? How deep are the discounts, and where’s the income cut-off for discount eligibility?

• Some visible aspects of student culture: athletics and athletic facilities, how many sports are varsity and how many are club sports, are there Greek organizations and how many students live in them, how many students live in dormitories and how much off-campus housing is there (and how expensive is it). How many places are there to eat on campus and off-campus? How expensive is the meal plan if there is one? Are on-campus residences managed by the university or by an outside contractor?

• Most common majors, number of majors and minors, average class size (not faculty-student ratio, which is a bad proxy for average class size), requirements to enter majors, general education requirements and how they’re structured, average number of credits/courses taken per semester, etc. I think maybe it would be possible to build a crude quantitative tool that would measure how ‘open’ the curriculum is (how many requirements, the intervals of requirements, how many choices students have available, how variable openness is across different majors) so that this wasn’t just an empty marketing word.

• Percentage of classes taught by tenure and tenure-track faculty vs. faculty on short-term contracts. Faculty compensation by rank compared to other institutions in the same region and against a comparison group. Structures of faculty governance and whether they have changed recently.

Medium-hard (would take critical reading ‘against the grain’ of some public information, the collection of news stories and social media discussions to measure or evaluate, or thoughtful combination of existing data to create an approximating metric):

• Budgetary austerity (whether justified or not by the financial health of the institution); ease of access to needed resources; availability of up-to-date facilities.

• Maintenance of facilities. Average comfort level and maintenance of dormitories (could use proxies like ‘numbers of one-room triples’ etc.)

• How many listed departments, programs, and majors are ‘real’ (e.g., have steady governance, long-term faculty commitments, dedicated facilities, consistently available courses) and how many are underfunded, undersupported or mostly fictional veneers?) I can kind of tell this from long experience on reading departmental webpages and course catalogs, but it takes work and a lot of insider knowledge (so I’m less good at it for some kinds of institutions).

• Some kinds of rough outcomes data, but not all, based in part of visibility of alumni networks and on some of the analysis of the connections between inequality and different tiers of higher education. (e.g., I can’t tell you the really specific outcomes differences between two similar institutions in the same rough tier of institutional wealth and selectivity, but I can tell you the difference between them and an institution in another tier.)

• How accessible programs of study are to students with a range of prior preparations and qualifications, and how often there are ‘weed-out’ classes in the curriculum.

Really hard (requires ethnographic knowledge, insider knowledge, is really subjective or shifts constantly):

• Is this college or university well-run? Will the average student experience with administrators at any rank be positive and helpful? How easy is it to resolve problems? How transparent is the administration, how easy is it to get information about a problem or issue? Especially regarding financial aid and costs. Are the services I might need any good, run at a high professional standard? (career placement, health care, mental health, information technology support, academic support services, etc.)

• Are the faculty secure in their situation, confident of their place in the institution, and accessible to undergraduates? Are they inspiring? Do they reflect the diversity of the student body? Are there mentors to connect to? Is there trust between faculty and students? Between faculty and administration? Are the faculty quick to address issues with the program of instruction, including availability of required or important classes? Are they being given the resources to address those situations?

• What’s the student culture really like? Is the student body arrogant, pretentious, exclusive, snobby, intolerant?

• What are the typical outcomes for graduates and how much are they a result of things the institution is intentionally doing? (e.g., the vexed question of how much value the institution is actually adding as a result of its distinctive design, culture and services vs. how much is just the general outcome of being of a certain age, having a certain existing quality of academic preparation, and being in any program of study at the general level offered by most colleges?)

The problem is that I think much of what’s in “medium hard” and “really hard” accounts for most of what actually makes a difference in student experiences between specific institutions. A student and their family could accurately pinpoint the kind of public data that will help them sort between institutions to end up with a list of good choices, but might never know that one of the colleges or universities on that list that looks utterly identical to the others in terms of public data is at this moment a colossal shit-show in some major respect. It can take years before an institution that is badly run starts to show obvious public signs of that situation, or before a dysfunctional faculty culture starts to erupt into view, but those conditions will often have a disastrous impact on a newly arriving student in ways that they might take three or four years to fully process. By the time they know that they’ve ended up with a bad choice, it’s all over.

Existing rankings are no good for any of that information. But often nothing available could really help to provide it, despite how consequential it can be.

Image credit: Photo by Joshua Golde on Unsplash