Academia: A Choice of Fig Leaves

Thursday's Child Has Far to Go

Benjamin Schreier wants an end to letters of recommendation. They’re empty, hyperbolic, time-wasting, useless instruments that encode bias into their content or are read with bias on the other end.

It’s hard to disagree. I’ve been involved in many instances of reading through piles of applications for grants, fellowships, honorifics and jobs where the letters of recommendation made little difference one way or the other. There are really only three kinds of letters of recommendation that get discussed at all in a selective process. One is the letter that is so poorly written or so lacking in diligence that it calls attention to its author. That usually doesn’t hurt the candidate. One is the rare letter that is openly hostile to the person being considered, which usually kicks off a discussion about the ethics of the author but also typically does hurt the candidate—you may think the recommender is a complete shit for writing it, but you can’t forget what was said.

The other is the unusual sort of letter that tells you something important and positive about the candidate that you could not fully discern from the rest of the application. Maybe the letter-writer tells you a really specific story of achievement or the overcoming of hardship that the candidate is too modest to tell. More importantly in the contexts that I operate in, the letter-writer will sometimes help you understand just how original or important the candidate’s research or project really is, or will reassure you that the candidate is capable of undertaking a really difficult kind of research. I remember one letter from a graduate advisor in anthropology explaining that a candidate for a dissertation grant was capable of undertaking extraordinarily sensitive work on a community of people engaged in illegal activity, noting in particular that the candidate had already spent five months getting to know the people he would work with. The candidate told us that, but it was crucial to hear that reassurance from an advisor who had undertaken similarly delicate work.

In most highly competitive processes of selecting a few—or just one—person out of a crowd of applicants, the basic work tends to proceed as follows:

1. An initial process of screening that is quick and relatively superficial where applications that are incomplete are removed and applications by candidates who are unambiguously not qualified (they don’t meet the basic criteria of the search or the competition, they don’t hold the necessary degree or credential, etc.) are taken out of the pool.

2. A second process of screening, usually by a small group of evaluators, that sifts out candidates who are seen as clearly non-competitive. At this stage, nobody’s looking at letters of recommendation—this is done from cover letters and project statements. Candidates who aren’t really in the right field, candidates who don’t really have pertinent experience, candidates who aren’t ready to undertake the work, candidates whose materials are incoherent or messy.

3. A third process of screening, usually by the full group of evaluators, that asks each evaluator to rank the dossiers or applications independently of the other evaluators. When the numbers roll in, there’s often a small group of candidates who are very highly ranked by everyone and a small group of candidates who are ranked at the bottom by everyone. Those candidates generally don’t even get talked about unless it’s a job search and only one person can get the nod. If there’s five spots in the fellowship and three people are at the top of everybody’s rankings, those three generally get selected and the discussion is about the remaining two slots.

4. It’s only in the final selection that anybody looks at letters of recommendation at all. It’s at this point that Schreier’s criticisms are pertinent. Most letters are full of generic superlatives that don’t do anything useful to distinguish candidates from one another. That’s not a knock on letter-writers—most of them know that to write a highly distinctive, individualized assessment of a candidate is not only hard work but it might actually hurt someone. E.g., when selection committees are reading those letters quickly, they’re often scanning for the letter that stands out. If you say “This candidate is especially remarkable at X but weak at Y” that language is being compared quickly and superficially to “This candidate walks on water, resurrects the dead and does brain surgery in their spare time" in all the rest of the letters. You can read an application pool and find the same recommender saying of three or four candidates that they are the best student they’ve ever taught. Every once in a while, a specific letter is descriptive enough that it becomes useful in making these final few difficult choices. But not very often.

But here’s the problem. Schreier mulls over an alternative to letters of recommendation and confesses that he doesn’t have any ideas. Given his critique, why bother? Why not just drop them completely?

The problem, it turns out, is not really with a given genre of information included in applications and dossiers. The problem is meritocratic reasoning itself as a way to arbitrate the allocation of scarce resources that will change the future prospects of a small number of individuals.

Every genre of information in a competitive application has bias in it. Grades are biased. Past achievements by the candidate are evidence of their previous privileged access to scarce resources as much as they are evidence of their capabilities or potential. Sensitive, careful, thoughtful evaluations of a portfolio of work by a teacher or a critic reflect the biases of the evaluator and the biases that surround that type of work. You could pluck a complete stranger out and ask them to write about a candidate’s writing sample or laboratory experience with everything double blinded and what you’ll get is something that still has the stranger’s bias (and you won’t even know what that might be because the stranger’s identity is blind to the selection committee) and you’ll get something that’s likely to be an inaccurate assessment in some other way because it doesn’t have access to context, because it doesn’t know anything about the candidate.

This is part of how human resources professionals and many others are justifying using black-box algorithms to sift through job applications: get the biased, subjective human beings out of the process altogether. Let a merciless, objective form of automation do that work cleanly. But that’s not only wrong—the algorithm encodes the bias of its makers—it creates a fiction about equity that sees it as the result of consistency. That’s the same way banks redlined Black applicants for mortgages and business loans: they had the numbers, they knew where loans paid back and where the defaulted. Not their fault, not their problem, just being consistent.

Equity in the allocation of scarce resources is already something of an impossibility: scarcity is the shadow of a larger injustice. You can’t run far enough to get out of that shadow. Best to face it honestly, without believing you can build a process free of any bias. You can decide that letters of recommendation require too much labor and that the genre is so free of useful information that we should stop using them. You’re still going to need something to arbitrate decisions about who will get the job or the grant.

And maybe we need to spend less time talking about perfecting applications and more time sharpening our skills as interpreters, more time talking about what it is we think we’re selecting for. Are we looking for merit? The best person for the job or the grant? Are we looking to encourage research or inquiry that we think is original or transformative, independent of the person proposing it? Are we looking for the person we think most needs the opportunity, where the chance to do the job or have the fellowship will have the greater transformative impact on an individual life? Are we looking for someone who will be most devoted to the organization that is hiring them or providing them financial support? Are we looking for a person who is the most different from what we already have or already fund, to deepen and enrich our own gestalt? Are we looking for the safest bet?

And on any of those: how often have we been right in our guesses in trying to do one or more of those things? If we’re often wrong, is that our fault, or something more systematic? Is it hubris to think anybody (or any system) can ever be more than random or lucky in applying any criteria for selection?

These aren’t questions that get answered by a perfect application process or a well-tuned algorithm. They belong to the process of reading and choosing. In my experience, we rarely pause to discuss what we’re doing in these terms. When we do, we often find that everybody in the room is operating from a very different premise about what we’re selecting for, and that at least a few people in the room aren’t honestly disclosing what they’re really doing or thinking. (Moreover, a few people in the room aren’t even reflective enough to answer the question for themselves.) At least some people are consistent in what they think they’re selecting for but inconsistent in the ways they interpret the material they’re given. If you’re worried about bias (and you should be), I’d worry about its tricky manifestations inside the least transparent and least accountable part of any process of allocating scarce resources to a lucky few.

Some of the values that people use in a selection process might require some form of humane, contextual witnessing of the candidate’s potential or accomplishment. Maybe that’s not what letters of recommendation typically are, but they’re often the closest we get. I can’t think of a way to push people to write more effective and focused letters, so in that respect I have sympathy for the impulse to just throw them out. But that impulse feels like a bit of a distraction from the deeper histories and more profound flaws of any application process.

Image credit: "Letter of reference for Jack Alcock" is licensed under CC BY-NC-SA 4.0.