Surviving Step 1: MCAT Scores & USMLE Performance

By Walter Wiggins

Are MCAT scores predictive of USMLE performance? This question strikes the minds of many medical students, particularly when trying to set performance goals for Step 1. I wondered the same thing when I was preparing for Step 1, and I was a little concerned because I hadn’t done as well on the MCAT as I felt I could have. In college, I majored in mathematics and took the bare minimum science requirements to apply to medical school plus a little extra chemistry. On top of that, I didn’t take MCAT prep as seriously as I should have. The result was a score that was successful by most standards (and, obviously, got me into med school), but I knew when I started preparing for Step 1, that I wanted to do better.

Let me preface the discussion of the relationship between MCAT scores and USMLE performance with the following observations. There are a wide variety of tracks one can take to get to medical school. Other than the minimum eight (or so) science and math courses required for admission, you can choose whatever major and whichever electives you like and still get accepted. As a result, everyone has a uniquely different experience leading up to the MCAT. However, with the exception of some minor variation between med school curricula, every med student sees the same content prior to studying for Step 1. Already, that significantly alters the predictive value of MCAT scores due to a significant reduction in one major source of variation amongst the population of individuals taking the test. Another source of variation that is reduced in Step 1 is the variation in content. While one can argue that there are significant differences between how you study for microbiology and how you study for renal physiology, the variation between aptitude for the physical sciences and verbal reasoning sections of the MCAT is likely greater. Finally, most of the folks who did well on the MCAT wound up in med school (and vice versa) and are taking the USMLE, yet the USMLE is still scored on a standard distribution.

Personally, I take the sum of the preceding observations to mean that your MCAT score neither dooms you to a low Step 1 score, nor guarantees you a 260. However, since that is simply my interpretation, let’s take a look at the more objective data that is out there. When I first investigated this question, I arrived (via Google) at the USMLE Step 1 Score Estimator provided by MedFriends.org. This site relies on self-reported scores of users who have used the Kaplan Q-bank, taken the free practice test provided by the USMLE, taken an NBME self-assessment, or taken the MCAT and subsequently taken Step 1. Based on intermittent linear regression analysis of the accumulated data, the site provides Step 1 predictions to new users based on the scores they enter into the calculator. Of the measures this site uses, they report the MCAT score as having the least predictive value. This is evidenced by their reported 70% prediction interval (that’s 1 SD of the mean, if you were interested) of ±17 points. For point of comparison, the errors of the other predictors are in the range of 11-13. As an example, if you scored a 30 on the MCAT, this site predicts that you will likely (at 70% confidence) score between a 216 and a 250 on Step 1 (mean = 233). That is an awfully broad range. Again, to compare, Scoring a 550 on an NBME self-assessment also nets you a mean predicted score of 233 but narrows the range to 220 to 246 (again, at 70% confidence), which is really not much better in the grand scheme of things.

In a more official study, performed by the AAMC and reported in 2005, MCAT scores were more predictive of Step 1 performance than undergraduate GPA (1). However, in this study, MCAT scores were only able to explain about 40% of the variance in the USMLE data, as determined via calculation of a validity coefficient. A more recent study revealed enhanced predictive value of the MCAT when the number of MCAT attempts was taken into account, with repeaters performing lower on the USMLE than non-repeaters with the same MCAT score (2). Importantly, these studies demonstrated incremental decreases in predictive value with respect to Step 2 and Step 3 scores.

In summary, there is a clearly a relationship between MCAT success and USMLE success. However, there are likely too many confounding variables contributing to both MCAT and USMLE performance to assign a clear predictive value to MCAT scores. Medical school grades and performance on practice questions and self-assessments have greater predictive value, but care should be taken not to put too much stock in the results of such predictions. My advice is to take a step back from all this talk of numbers and…relax (there it is again) so you can focus on being prepared for test day.

References

1. Julian ER (2005). Validity of the Medical College Admission Test for predicting medical school performance. Acad Med 80(10):910-17.
2. Zhao X et al. (2010). Validity of four approaches of using repeaters’ MCAT scores in medical school admissions to predict USMLE Step 1 scores. Acad Med 85(10 Suppl):S64-7.