It All Comes Down to Validity: Thoughts on the MCAT

By Kimberly A. Swygert, PhD, senior psychometrician, National Board of Medical Examiners

A set of upcoming articles in Academic Medicine thoughtfully summarize the performance of the current MCAT and present a summary of the processes and decisions behind the development of the MCAT2015. While the articles cover an expansive range of topics, when viewed as a whole, they address the most important question one can ask about a test: how valid are the MCAT scores for the decision process of medical school admissions?  As validity expert Michael Kane often reminds us, validity is a property not only of a test but of the interpretations and uses of the test, and the MCAT research summarized in these articles is a way to “kick the tires” and show that MCAT scores work in the ways that test developers and users intend for them to work. Given the importance of the MCAT in determining not only who becomes a physician, but also the premedical curriculum designed for their education, the critical question of validity will require constant, focused research efforts by the Association of American Medical Colleges as the old MCAT gives way to the new.

As several of the articles note, the conceptualization of what makes a good physician has changed since the MCAT was last updated in 1991. Medical schools are seeking students who understand both the natural sciences and the social sciences underpinning the human side of the practice of medicine. Assessments of potential physicians may need to expand beyond academic competencies to include personal competencies such as resilience, reliability, and ethical responsibility. The applicant pools for medical school have changed greatly in the last three decades, becoming more diverse with respect to gender and ethnicity. The responsibility of the MCAT developers is to piece together the validity puzzle by finding the answers to specific questions that address these changes. Do MCAT scores continue to predict crucial academic outcomes? Do MCAT scores add value over and above the non-academic factors that are increasingly used in the admissions context? Does the MCAT work for all types of applicants?

Much has already been published on the utility of MCAT scores for predicting performance in medical school and on the United States Medical Licensure Examination Step exams, and Dunleavy and colleagues add a twist to the predictive validity argument by introducing the concepts of “unimpeded” and “impeded” progress through medical school. Their discussion sheds new light on the utility of the MCAT for the prediction of valued outcome in medical education, as opposed to the more standard grades and Step scores. Complementing this is Monroe and colleagues’ updated overview of the medical school admission process, in which we are reminded that a trend towards “holistic” admissions does not mean standardized assessments hold less value. Indeed, as a larger, more diverse population is applying to medical schools in the 21st-century, the MCAT continues to play an important role in the decision of whom to invite to submit secondary applications and for interviews, even as the importance of non-academic factors increases with respect to the final acceptance decision. Finally, the articles demonstrating that the MCAT scores predict medical school performance equally well across ethnic subgroups and asking whether the MCAT should be used in admission to Canadian medical schools both address a vital piece of the validity puzzle. It’s essential for test developers to assess and monitor phenomena such as score gaps and test use expansion and investigate whether the potential for biased decisions will increase for current subgroups or new groups. Taken as a whole, this body of work provides support for the use of the current MCAT exam and a blueprint for examining the usefulness of the MCAT2015.

Related Posts

Comments are closed.