Thinking About Admission Biases

I have been recently reading Thinking Fast and Slow by Daniel Kahnmen, the only psychologist to have ever won the Nobel prize in economics for his work in decision making and behavioral economics. Based on over three decades work on cognitive psychology, Thinking Fast and Slow delves into how our brain thinks, comparing our subconscious intuition and our deliberate cognition.

Apart from being a fantastic read, probably the best book I have read in, it got me thinking about how we select medical students. Long time readers may know that I help out at my medical school’s admission committee with choosing applicants each year. Our school is not unique in its way of selecting from the applicant pool. We use a criteria of GPA, MCAT, Interview Scores, Extracurricular Activities and References to score and rank each applicant.

After reading this book, I am more aware of many flaws in the current system. Specifically, when it comes to the many methods and heuristics that we use in evaluating applicants. Heuristics being problem-solving methods our brain uses to solve complex problems by experience. a mental short-cut if you will.

Evaluating Attributes

One of the cognitive biases that medical school admissions use is attribute substitution. This heuristic occurs when an individual has to make a judgment of a target attribute that is complex. Instead of having to work through the more complicated problem, you substitute a more easily calculated attribute.

Rather than having to evaluate students holistically, medical schools use quick substitutions.  GPA are used as a measure of diligence, the MCAT of raw intelligence and interview scores a test of interpersonal communications. There is little to no evidence that these criteria will select great doctors, but hundreds of medical schools continue to use these criteria. For example, when an applicant who has good grades and a high MCAT score but did poorly on their interviews, they are automatically judged as someone who is a bookworm, even when they make in fact be outgoing and social. More important attributes such as empathy, communications and integrity are left ignored as they are difficult to assess.

Anchoring Biases

Similarly, anchoring is a commonly seen phenomenon. It is a cognitive bias that makes one piece of information weigh more heavily than it should when it comes to decision making. I previously wrote in a post that the best way to increase your chances of medical school is to Not Get Eliminated. I didn’t truly understand at the time why this was true other than by personal experience working on the adcom. After reading Thinking Fast and Slow I can see that the admission committee often gets blinded by anchoring biases.

When the admission committee reviews which applicant to accept, they are more drawn to red flags that then effectively “blinds” them to neglect the rest of the application. If an applicant said an off-hand remark during the interview, they are all of a sudden seen as unprofessional and deemed unfit for a career in medicine. Similarly, a negative sentence in a reference letter might just sink your whole application.  It becomes anchored in the minds of the admission committee and becomes weighted more negatively than it should. Even if the rest of the application is outstanding, “red flags” can create an anchor bias and lead us to make insufficient judgments.

An Imperfect Process

After reading Thinking Fast and Slow and understanding all the different heuristic methods our brains employ, it’s not hard to see the flaws in the current admissions process.

The first point is that it is extremely difficult to predict whether an applicant will be a good physician or not. We are given scant information, some academic numbers and a quick interview before we have to decide whether to accept them or not. There’s not enough concrete evidence to know whether someone will be an innovative leader or whether they will be involved in multiple lawsuits.

Secondly, we employ many cognitive biases that ends up screening out great applicants. Someone who is quieter and shyer may in fact be a great physician one day, but we might score them poorly on the interview because they don’t have as much to say. Every admission committee member has their own biases as to what attributes to look for. Even the discrepancy between different adcom members can affect the application process. There is a lot of variability on how well your application would score based just who reviews your file.

Finally, there’s not a lot of data for the correlation between the criteria to select a medical student to how they will eventually perform as a doctor. We have to realize that we need to have all sorts of doctors from including surgeons, family doctors and even pathologists. And I think it’s naive to believe we know how to best select those will succeed in each field. Especially if all we are trying to do is extrapolate marks and extracurricular activities.

Where to go from here?

The current admissions process is flawed but by no way is it broken. Medical schools still manage to graduate great doctors every year and have no problems filling their classes with amazing applicants.

But in order to improve, I think admission committees have to be more aware of the potential heuristics and biases that we have when we evaluate medical school applicants. Furthermore, there needs to be a stronger push towards a more holistic evaluation of individuals instead of one where each person is evaluated on their numerical scores. And above all we need to make the admissions process as fair and transparent as possible, so that great applicants are not excluded for the wrong reasons and that bad applicants do not end up gaming the system.

If you want a sample of what Thinking Fast and Slow is like, the Nobel prize site has an excellent lecture that summarizes all the main points of the book. I also didn’t realize the influence of Kahneman’s work. It was the basis of many of the books I have read in including How Doctors Think, Blink, Paradox of Choice…