This episode of the Academic Medicine Podcast is a companion to the August 2023 Ensuring Fairness in Medical Education Assessment supplement, which was sponsored by the Josiah Macy Jr. Foundation. The supplement focuses on creating an optimal, equitable system of learner assessment. In this episode, Holly Humphrey, MD, president of the Macy Foundation, discusses the origins of the supplement and the recommendations shared by the authors for fostering equity in assessment. Then the authors of each of the included papers share a summary of their work, including their key findings and takeaways to guide thinking on promoting fairness in assessment. Finally, Dowin Boatright, MD, MBA, MHS, a member of the planning committee for the conference that led to this supplement, discusses where medical education should go from here.
This episode is now available through Apple Podcasts, Spotify, and anywhere else podcasts are available.
A transcript is below.
Read the complete supplement, including each of the articles discussed in this episode: Ensuring Fairness in Medical Education Assessment. Acad Med. 2023;98(8S):S1-S85.
Transcript
Toni Gallo:
Welcome to the Academic Medicine Podcast. I’m Toni Gallo. Last month, the journal published the Ensuring Fairness in Medical Education Assessment supplement, which was sponsored by the Josiah Macy Jr. Foundation. Today’s episode is a companion to that special issue. You’ll hear from Dr. Holly Humphrey, president of the Macy Foundation, about the origins of the supplement and the recommendations shared by the authors for creating an optimal, equitable, system of learner assessment. Then the authors of each of the included papers share a summary of their work, including their key findings and some takeaways to guide your thinking on promoting fairness in assessment. Finally, Dr. Dowin Boatwright, a member of the planning committee for the conference that led to this supplement, concludes with a discussion about next steps and where medical education needs to go from here.
Think of this episode as an overview of the supplement, giving you a glimpse into each of the included papers. I hope you’ll listen here and then go read the full text of the papers that grabbed your attention. The complete supplement is available to read for free on academicmedicine.org right now. Please share it and this episode with your colleagues and join the journal and the Macy Foundation on social media. Let us know how you’re thinking about equity in assessment and how you might use these papers and the authors’ recommendations to advance your own practice. With that, I’ll turn things over to Dr. Humphrey for her introduction.
Holly Humphrey:
Hi, Toni. Thank you for inviting me and my colleagues to the podcast. I’m Dr. Holly Humphrey, former dean for medical education at the University of Chicago’s Pritzker School of Medicine and now president of the Josiah Macy Jr. Foundation in New York. The Macy Foundation is the only national foundation dedicated to improving the health of the public by advancing health profession’s education. One of the ways we do this is by holding regular conferences that explore important issues in health profession’s education. During the conferences, invited participants develop recommendations to address the issues being discussed, which we then publish and broadly disseminate. Our most recent conference focused on ensuring fairness in medical education assessment.
The commissioned papers, case studies, and recommendations report from the conference have been published in the August special supplement of Academic Medicine. The special issue features 4 academic papers that provide background on the issue of fairness in learner assessment as well as 4 case studies from medical educators who are exploring ways to create equity in assessment at their institutions. Together the 8 commissioned papers and case studies, which served as the basis for the conference discussion, detail the harmful effects of bias in medical education generally and in learner assessments specifically.
According to the papers and the case studies, and you’ll hear more about this from the authors themselves later in this podcast, harmful bias permeates learner assessment and creates unfair advantages for some medical students and residents and disadvantages for others. The papers and case studies make it clear that if we are to develop and sustain a physician workforce that can eradicate health disparities and lead an equitable health system, we must do what we can to eliminate harmful bias and optimize learner assessment. To this end, the conference recommendations report describes what an optimal, equitable system of assessment looks like and lays out a robust set of action steps that can help institutions and the broader field of medical education optimize learner assessment.
An optimal assessment system is one that prioritizes learning and promotes the development of a growth mindset in learners. Such a system relies primarily on low stakes assessment approaches, such as frequent feedback and coaching. These approaches are evidence-based as well as criterion or competency based, and their goal is to ensure that the desired learning objectives are achieved. This contrasts with the more traditional assessment approaches we’re all familiar with that rely on primarily high stakes testing, which creates competition and invites comparisons between learners. Optimal, equitable assessment recognizes and values the different strengths and needs that all learners have and nurtures them accordingly. It also involves learners by being transparent in how the assessment system works, how assessment data are used and shared, and how fairness and equity are monitored and ensured.
The recommended action steps for achieving this optimal, equitable system of learner assessment are extensive. I can’t possibly go through all of them now, but here are a few examples from the report. Develop policies that align the learner assessment system with the institution’s mission and values such as improving patient care and prioritizing equity and inclusion. Or educate faculty about the impact of harmful bias in assessment and provide them with tools and the necessary training to design and implement optimal assessment. Develop policies and mechanisms that enable faculty and learners to work together to identify and eliminate harmful bias in assessment. And avoid over-reliance on high stakes assessment data and expand use of multisource data, holistic review, competency review committees, and other similar approaches. Optimal assessment engages multiple experts in reviewing diverse sources of information at crucial points along the educational continuum such as when learners are being assessed for advancement to a new level or at the time of graduation.
I encourage everyone to read the published materials in the August Academic Medicine special supplement and to get involved with efforts to create optimal, equitable learner assessment systems. Our learners are relying on us and we owe them no less.
Karen Hauer:
Hi, I’m Karen Hauer and I’m here to discuss our paper, which is titled “‘My Assessments Are Biased!’ Measurement and Sociocultural Approaches to Achieve Fairness in Assessment in Medical Education.” I am associate dean for competency assessment and professional standards and a general internist at the University of California San Francisco (UCSF). Our expert author team also includes Yoon Soo Park and Ara Tekian, both PhD medical educators at the University of Illinois at Chicago, and Justin Bullock, a nephrology fellow at University of Washington and PhD student in medical education at Maastricht University.
The purpose of our paper is to provide an overview of assessment bias focused on clinical learners. We take a deep dive into the causes and consequences of bias from broad perspectives in the literature. We conclude with recommendations for medical educators to overcome bias and develop an ideal assessment system for all learners.
What is bias? Bias is a prejudice in favor of or against one person or group compared with another, usually in a way that is unfair. In medical education, multiple studies have shown group differences in assessments of clinical performance that favor learners who are not from groups historically marginalized in medicine and society. This bias leads to lower scores, lower grades and awards, all of which negatively affect the diversity of the future physician workforce.
We discuss bias in our paper from 2 perspectives. The measurement perspective considers bias as a measurement statistical problem that interferes with the fairness and accuracy of assessments, typically written exams. Second is the sociocultural perspective, which considers the real world clinical environment. Learners experience their supervisor’s implicit bias, microaggressions from team members or patients, and stereotype threat, all of which interfere with their ability to participate fully, learn well, and perform at their best. We wrap up with specific recommendations to guide educators in designing optimal assessment systems to avoid harmful bias.
Our 12 recommendations fall into 5 broad areas. (1) A program must articulate the values that guide assessment and then design the assessment system and policies to align with those values which should center on patients and learners. (2) Design an assessment system that defines needed outcomes focused on our patients and communities and that ensures learning through frequent formative assessments and periodic summative assessment. (3) Use best practices for assessment procedures and consider the risk of bias at all stages of assessment, design, and implementation. (4) Promote quality assurance of the assessment system by engaging individuals with diverse backgrounds and perspectives in designing the assessment system and in ongoing data monitoring. (5) Foster an equitable learning and assessment environment. Educate faculty so that all learners have opportunities to practice, ask questions, discuss feedback, and show their best performance.
So, in summary, our literature review produced recommendations that can be used to develop robust assessment systems with associated procedures and processes that are defensible, facilitate meaningful clinical learning in an equitable manner, and ultimately promote patient safety and public health.
Nientara Anderson:
I’m Dr. Nientara Anderson, and I’m a resident in the Department of Psychiatry at Yale. My coauthors are Mytien Nguyen, an MD-PhD student at the Yale School of Medicine, Kayla Marcotte, an MD-PhD student at the University of Michigan Medical School, Dr. Marco Ramos, an assistant professor in the History of Medicine and the Department of Psychiatry at Yale, Dr. Larry Gruppen, a professor in the Department of Learning Health Sciences at the University of Michigan Medical School, and Dr. Dowin Boatright, an associate professor and the Vice Chair of Research in the Department of Emergency Medicine at NYU Grossman School of Medicine.
Our paper titled “The Long Shadow: A Historical Perspective on Racism in Medical Education” explores how the history of Christian Europe, enlightenment era racial science, colonization, slavery, and racism shaped modern medicine in the United States and continues to impact medical trainees of color. Beginning with the coalescence of Christian European identity and empire, we trace European racial reasoning through the racial science of the enlightenment into the White supremacist ideology behind Europe’s global system of racialized colonization and enslavement. We then follow this racist ideology as it becomes an organizing principle of Euro-American medicine and examine how it manifests in medical education in the United States today.
Through this historical lens, we expose the macabre histories of violence behind familiar contemporary terms, such as implicit bias and microaggressions. We explore why racism is so prevalent in medical education, and we examine how racism affects admissions, assessments, faculty and trainee diversity, racial climate, and even the physical environment in medical institutions. Finally, we recommend the following 6 historically-informed steps for confronting racism in medical education.
(1) Academic medical institutions should integrate the history of racism into educational curricula and institutional awareness. Instead of ahistorical framings of implicit bias or microaggressions, racial bias should be directly attributed to European colonization, enslavement, and persecution of Black people and other people of color. In addition, medical institutions should investigate their own connections to the history of racial oppression and make their findings available to the public, because we cannot hope to address the consequences of such histories if they remain buried.
(2) Academic medical institutions must improve internal tracking and data collection on racial bias. Bias and discrimination reporting and response systems should be consolidated and centralized within academic medical institutions and should be designed to assess and improve the learning environment, and not simply serve to detect and punish only the most grievous incidents of discriminatory abuse. Medical institutions should also conduct regular internal audits for potential bias in their educational content, assessments, hiring, and admissions.
(3) We recommend adopting mastery-based assessments and eliminating the category of professionalism from trainee evaluations. Mastery-based assessments go beyond pass-fail, minimal standards of competency while avoiding the inherent vulnerability to bias in assessments that rely on nonbinary scales of scores. And professionalism should be abandoned as an evaluative category, because it is riven with historical and contemporary racial bias and is often weaponized against trainees of color.
(4) We recommend embracing holistic review and expanding its possibilities in admissions. (5) Holistic review principles should also be used in hiring and promotions in order to increase faculty diversity. And (6) the accreditation process should be used to catalyze implementation of these recommendations to combat racial bias in medical education.
These strategies will help us as a profession to acknowledge the harms propagated throughout the history of racism and medicine and take meaningful steps to address them.
Lou Edje:
We are delighted to welcome you to our podcast. I am Lou Edje, she/her, associate dean for graduate medical education and designated institutional official, which basically means I support all our residents and fellows as well as their program directors and program coordinators. I’m a professor of medical education at the University of Cincinnati College of Medicine and Westchester Hospital. I’m joined by 2 of my favorite medical education colleagues and co-authors who will introduce themselves for their section of our paper, “Strategies to Counteract the Impact of Harmful Bias in Selection of Medical Residents.”
We know that patients who receive medical care from physicians of the same ethnicity have better health outcomes. We also know that selection of learners for residency influences the demographic of our physician workforce, thus impacting health disparities. Oftentimes, we think of assessment of residents while they’re already in residency. However, the first assessment a program director does is during the interview season. As Lucy mentioned in her 2020 paper, bias and assessment is a wicked problem fraught with susceptibility to contextual influence. In our paper, we explore vulnerabilities to harmful bias and assessment of learners who are in the transition between UME and GME. We look at the residency application process as well as the interview, recruitment, and selection of learners. We define bias, explore historic inflection points of bias in the transition, and discuss ways to work toward equity and selection practices used by residencies.
Tony Casillas:
I’m Tony Casillas, and I am a second year pediatric hospital medicine fellow at Cincinnati Children’s Hospital Medical Center. I will be reviewing the section of our paper that discusses the residency selection process and the application and the different areas of where bias may exist. This process begins much sooner than one might think, beginning with the learner’s choice in specialty, and for those who are underrepresented in medicine, what factors into that decision? Do they see themselves represented within that specialty of interest? Are there opportunities for mentorship and sponsorship? For many, there is a desire to return back to their communities and serve. Do they see that model within the specialty that they’re interested in?
Then there are the different components that make up the residency application, and that includes the clerkship grades, the USMLE Step 1 and Step 2 scores, the Medical Student Performance Evaluation, and honor society induction. Looking at the standardized exam component of the application, you have the USMLE exams as well as the subject exam component of the clerkship grade, and with that has been demonstrated disadvantage and bias towards those who are underrepresented in medicine and those with disability. Within the clerkship grade is also the narrative component, which when looking at differences among gender and race have been … demonstrated clear differences showing bias [against] females and those who are underrepresented in medicine.
When you look at the Medical Student Performance Evaluation component and the honor society induction, much of what goes into those pieces of the application carry forward those standardized exam scores as well as those narratives, and so continue to perpetuate that bias. Then you get to the interview component of this process, which when interviewers are provided standardized metrics, such as the USMLE Step 1 and Step 2 scores, their scores for those applicants tend to parallel those USMLE scores. Then there is the virtual component, which has a potential for disadvantage as one is not able to fully appraise the culture of a program and to determine their fit.
Our process in its current form lacks transparency. So, as we begin to acknowledge these areas of bias, we can then move towards a process that is more transparent for applicants so that they can then make the most informed decision for their residency.
Jennifer O’Toole:
Hi, my name is Jennifer O’Toole and I’m the program director of our combined internal medicine and pediatrics residency program here at the University of Cincinnati Medical Center and Cincinnati Children’s Hospital. I’m going to be reviewing the section of our paper that provides some very practical steps to eliminate harmful bias and promote equity in the residency selection process. I do want to start with the disclaimer that this is a very dense section of our paper and I’m just going to be providing a very brief overview.
So, starting with the preclinical years, we recommend that all residency programs provide very targeted outreach to groups of underrepresented students within their specialty or their residency program, providing these students with very targeted advising, mentorship, sponsorship, and if possible, away or visiting rotations. When it comes to the application review and selection for interview process, we recommend that programs start with identifying their own program’s mission, vision, values, and commitment to diversity. We think once these characteristics are defined, programs can then incorporate them into the holistic review they conduct on residency applications.
When it comes to the actual interview day, there’s lots of different things programs can do, but some examples include recruiting a diverse group of faculty interviewers, requiring that all of these faculty interviewers undergo implicit bias reduction training, removing biased metrics from application packets, so faculty can focus on the conversation with the applicant and not the metrics on the paper before them, including behavioral-based interview questions within the interview day, and then lastly, including scored rubrics that are very easy for faculty to follow and align with their program’s mission, vision, and values.
When it comes to the final ranking and post-interview communication, we recommend that programs monitor really closely for any group think that could occur at any ranking meeting so that one voice doesn’t overpower another. We also recommend that when programs put together that final rank list and they’re using various rubrics within that ranking, they make sure that the things they’re weighing highly align with their program’s mission, vision, and values. Then lastly, we recommend that programs use caution with preference signaling, post-interview communication, and any in-person visits to their programs since we know that not all applicants have access to adequate time and financial resources to participate in those post-interview activities. Then lastly, I think as programs can be as transparent as possible with applicants about this entire process, whether it’s on any handouts or their website or their social media platforms, I think that does serve our applicants to residency very well.
In closing, we hope we provided you today with ample evidence as well as some very detailed steps on how residency programs can really decrease bias and improve equity within every step of the recruitment process.
Eric Holmboe:
Hi, I’m Eric Holmboe from the Accreditation Council for Graduate Medical Education. On behalf of my wonderful co-authors, Dr. Nora Osman, who’s associate professor of medicine at the Harvard Medical School, Dr. Christina Murphy, a recent graduate of the Perelman School of Medicine at the University of Pennsylvania, and Dr. Jennifer Kogan, a longtime friend and collaborator who’s professor of medicine also at the Perelman School of Medicine, I’d like to provide a brief synopsis of our Academic Medicine article entitled, “The Urgency of Now: Rethinking and Improving Assessment Practices in Medical Education Programs” that was produced as part of the Josiah Macy Jr. Foundation conference on ensuring fairness in medical education assessment.
Assessment integrated with robust curricular experiences is an essential component of professional development during medical school and postgraduate training. Assessment provides the information needed to give feedback, support coaching and the creation of individualized learning plans, inform progress decisions, determine appropriate supervision levels, and most importantly helps ensure patients and families receive high quality, equitable, and safe care in the training environment. In our paper, we first described 3 overarching principles for programmatic assessment that must be embraced by all training programs to optimize assessment practices and ensure all learners achieve desired medical education outcomes. They are that programmatic assessment must support professional development, use a systems thinking approach, and finally and critically important, address structural and individual bias. The issue of bias in medical education, especially in assessment, remains a pernicious problem.
In the remainder of the paper, we address key challenges resulting from bias and provide a set of 10 recommendations with tables and figures to assist in improving assessment practices. Some of these recommendations include (1) rebalancing our focus of assessment programs to emphasize more work base at the bedside assessments and reduce an overreliance on point in time high stakes assessments. (2) Take seriously and address unwarranted variation in assessment practices, recognizing that bias is a particularly unwelcome form of unwarranted variation. (3) Explicitly define assessors’ roles and responsibilities. This will also require a meaningful investment in faculty development. After all, assessment requires knowledge, skills, and attitudes that requires training. (4) Recognize the importance of coproduction with learners. Learners must be active agents in assessment programs given agency, shifting our mindset from where we have traditionally done assessments to and on learners to assessments with learners. (5) We must explicitly address bias and assessment. Through excellent research, we now have a better sense of the scope of the problem.
However, we have much less research and guidance in medical education on how to reduce bias. Pulling from other fields, we provide 5 interventions that show promise and should be explored to reduce assessment bias. Please see the paper for remaining recommendations, suggestions, and guidance, and also check out the other wonderful papers in this supplement.
I’d like to end by highlighting that our paper is by no means inclusive of all medical education assessment challenges or possible solutions. However, there is a wealth of current assessment research and practice that we believe can help medical education programs improve educational outcomes and help reduce the harmful effects of bias. We hope you enjoy the paper and we welcome your feedback.
Ben Kinnear:
Hello, my name is Ben Kinnear. I’m a hospitalist and medical educator at University of Cincinnati and Cincinnati Children’s Hospital, and I’m an associate program director for our med-peds and internal medicine residency programs.
Eric Warm:
I’m Eric Warm. I’m a primary care physician and also a medical educator at the University of Cincinnati. I’m the program director of the internal medicine residency program.
Ben Kinnear:
We were lucky enough to write with an absolutely brilliant group of authors that includes Dr. Danielle Weber, Dr. Dan Schumacher, Dr. Lou Edje, and the incomparable Hannah Anderson, who was our senior author. Each one of these authors brought a unique bit of expertise and insight to our paper and we had a great time writing it. We hope we can summarize it for you here.
The title of our paper is “Reconstructing Neurath’s Ship: A Case Study in Reevaluating Equity in a Program of Assessment.” Now, that’s quite the mouthful, and you might be wondering, what is Neurath’s ship? Well, we decided to reach way back to the past to a philosopher named Otto Neurath, and he had this analogy of a ship at sea to describe how epistemology, or how we think about knowledge and what is true, shapes what we do, the questions we ask, and the values that we hold. We thought this was a good analogy to explore how these worldviews on fundamental issues like reality, knowledge, and equity shape how we go about assessment.
Eric Warm:
The original ask was to provide a case study on equity and fairness in a program of assessment. Now, our program of assessment is often used as a case study. We spent a decade building a fairly robust system to assess learners, as well as building validity evidence for how we use that system to make summative decisions to help learners improve. However, in preparing for this case study, we recognize that our assessment system still has a long way to go towards improving equity.
We also recognize how our worldview, otherwise known as epistemology, has evolved over time and truly informed our work. We started years ago with a very postpositivist worldview. In such a worldview, there’s an objective truth that exists in the world that we’re trying to reach through quantitative assessments. And in that worldview, equity involves removing error, which we try to do using something called learning analytics. Over time, we move towards a more constructivist worldview in which truth is not an objective and fixed, but socially constructed and contextual. We build it together as a team. We leaned more into narrative assessment, understanding the context in which assessment occurs. In this worldview, equity requires co-creation of assessment systems and engagement of learners.
Currently, we are sailing into new epistemological waters, taking on a more critical theory worldview. We’re looking to critique the very foundational substance upon which our assessment system is built and seek to understand power dynamics wherever they exist. As we move through these different worldviews, our push to assessment and equity has changed.
Ben Kinnear:
So I think the most important takeaway for readers is that educators and program leaders who are designing and iterating assessment systems should really take a moment to stop and deeply consider the worldviews that inform their work. Terms like epistemology are complex and they can seem very theoretical and impractical, but our philosophical worldviews underpin everything that we do, our implicit assumptions, our decision-making, our value structures.
Before launching into debates on assessment instruments or rater training or just going straight to the work, we should ask ourselves and members of our team to reflect on their own worldviews and openly discuss how epistemologies or their assumptions about knowledge and truth inform their approach to seeking equity in assessment. We hope you really enjoy the paper. If you have any comments or feedback, please email us. We’d love to have that discourse with you.
Chavon Onumah:
I’m Dr. Chavon Onumah, and I’m an associate professor of medicine and the internal medicine residency program director at George Washington School of Medicine and Health Sciences. I’m excited to share our paper, “Strategies for Advancing Equity in Frontline Clinical Assessment.” First, I like to acknowledge my phenomenal cofirst author, Dr. Amber Pincavage and my masterful senior author Dr. Nora Osman, as well as the rest of our dynamic team that contributed to this paper, Dr. Cindy Lai, Dr. Diane Levine, Dr. Nadia Ismail, and Dr. Irene Alexandraki. I’ve been truly honored to learn and grow with this team of medical educators and leaders working to identify and disseminate evidence-based strategies to promote equity in clinical learning and assessment.
In this paper, we aim to equip frontline preceptors and assessors with practical strategies and concrete examples for real-time implementation. The foundation of this work is built on knowing that educational equity in medicine or the ability of medical students and trainees to reach their full potential in bias-free learning environments cannot be achieved without addressing assessment bias, which we know is prevalent unfortunately and has extensive implication for learners and ultimately the health care system.
So, based on our experiences as educators, we created a case study about C.M., a student early in her clinical years who wears a hearing aid and identifies as underrepresented in medicine, to illustrate ways bias affects learner assessment. Then we apply Dr. Catherine Lucy and colleagues’ organizational framework of the 3 components of equity in assessment–contextual, intrinsic, and instrumental–to offer evidence-based approaches to mitigate bias and promote equity in clinical assessment. We chose this framework because it illustrates the importance of recognizing the interconnectedness of domains in the complex learning environment when designing and implementing interventions.
Briefly, to address contextual equity, or the environment in which learners are assessed, we recommend building a learning environment that promotes equity and psychological safety, understanding the learner’s context and additional stressors such as stereotype threat and imposter syndrome that are put on some learners and undertaking implicit bias training. Intrinsic equity is centered on the tools and practices used during assessment, and this can be promoted by using competency-based structured assessment methods with flexibility and employing frequent direct observation of clinically related tasks to assess multiple domains, including things like patient advocacy.
Then finally, strategies for instrumental equity or how we communicate and use assessment includes specific actionable feedback to support growth and use of competency-based narrative descriptors and assessments. Many of us have taught and assessed some variation of C.M. Being the imperfect humans that we are, each of us has been guilty of perpetuating inequities in assessment, whether intentionally or unintentionally.
We hope this case illustration and examples to target contextual, intrinsic, and instrumental equity and assessment equips and empowers frontline clinical faculty to actively promote equity in assessment and support the growth of a diverse health care workforce. Frontline clinical faculty members may not be able to initiate widespread changes autonomously, but through a growth mindset and with intentionality, including those concrete examples seen in Table 1, they can contribute to equity and assessment, impact the learners’ experiences and their future careers.
Eve Colson:
Hello, my name is Eve Colson and I’m presenting this paper on behalf of myself and my coauthors, Maria Pérez, Stanley Chibueze, Tom De Fer, Amanda Emke, Steve Lawrence, Sherree Wilson, Nichole Zehnder, and Eva Aagaard. Our paper is entitled, “Understanding and Addressing Bias in Grading: Progress at Washington University School of Medicine.”
In this paper, we describe our approach to promoting fairness in learner assessment by addressing disparities in clerkship grading at our school. When we discovered that White students were more likely to receive honors and clerkships than those students from races and ethnicities historically underrepresented in medicine, school leadership charged a commission to propose recommendations to mitigate these disparities. Using an evidence-based approach and a quality improvement framework, the commission identified 6 areas for improvement.
The first area was ensuring equitable access to exam preparation materials, especially since the cost makes them inaccessible to some students. We also ensured that students were aware of the resources that we offer including faculty workshops, one-on-one sessions with faculty and learning specialists. Second, we developed a new curriculum, and with the new curriculum, the school revamped the assessment system. With this revamping of the system, we optimize evidence-based practices to decrease bias, including competency committees and standardized rubrics across the clerkships. Using the ACGME competency domains, we assess in multiple dimensions by various methods across the entire curriculum. Students also have real-time access to their assessments via portfolios, which they can review with their coaches at any time. Students can grow and develop before competency decisions are made.
Third, with a new curriculum, the school instituted a longitudinal health equity and justice thread. In this thread, students learn about and apply concepts of bias and antiracism and spend time in the St. Louis community with organizations that serve low income, often marginalized people. Also, to prepare for the clinical setting, students now engage in clinical immersions early on in their experience. In this way, they can learn about expectations and microcultures that they may encounter while on the clerkships. Fourth, to address the learning environment, we have training programs for students, residents, faculty, and staff, and have introduced additional intensive training programs for each department. A universal reporting system is also now available for use by all members of the Washington University School of Medicine community and the partner hospitals, so that there’s a single committee overseeing clear and consistent processes for evaluating and acting upon any incidents.
Fifth, as part of creating an optimal learning environment, the school has enhanced efforts for both recruiting and retaining a diverse workforce, and then finally, evaluating our outcomes and processes has been key to ensuring the success of these efforts.
So, while there’s no single easy fix, for us, starting with a needs assessment and creating an evidence-based plan has provided a sound start and direction for us. A multipronged approach is likely necessary to address the challenges because this issue involves not just assessment practices but also assessors and a larger learning and work environment. Having a plan in place, goals, broad community support, leadership engagement, and a way to evaluate processes and outcomes and a continuous quality improvement mindset can set the stage to create an equitable system of assessment.
Hyacinth Mason:
Hello, my name is Dr. Hyacinth Mason. I serve as assistant dean of students at Tufts University School of Medicine. The research team, Maria Pérez, Dr. Janice Hanson, Dr. Eva Aagaard, Dr. Donna Jeffe, and Dr. Eve Colson from Wash U, and Dr. Arianne Teherani from UCSF, and I are pleased to share a bit about our qualitative study, “Student and Teacher Perspectives on Racial Equity in Clinical Feedback.”
Eliminating ethnic and racial disparities in students’ clerkship grades remains a major challenge. Our previous study published in Academic Medicine in 2022 identified feedback as a contributor to racial/ethnic grading disparities. To dig deeper into the why of that result, we conducted a secondary analysis of interview data from 22 teachers and 26 students from 3 US medical schools using critical race methodology to answer 2 research questions: Do students and teachers perceive that clinical feedback is provided equitably regardless of race/ethnicity? If not, what factors do students and teachers attribute to racial/ethnic inequities in clinical feedback? We found that regardless of self-identified racial/ethnic group, teachers and students perceived race/ethnicity based inequities in clinical feedback.
Three themes emerged. Theme 1: Teachers’ racial and ethnic biases influence the feedback they provide students. One student shared, “As a student from an underrepresented background, I feel my feedback is largely focused on my personal characteristics and not my clinical abilities. Sometimes it’s very subtle. It’s hard when students, especially underrepresented medicine students, don’t feel seen. Not being seen is an endless spiral of poor evaluations.”
Theme 2: Teachers have limited skillsets to provide equitable feedback. Teachers express reticence about providing feedback to students whose race/ethnicity differed from their own, concerned that such feedback would be misconstrued and lead to personal and professional consequences for themselves and for the student. One teacher said, “I feel more nervous now about saying or doing the wrong thing. The consequences are severe.”
Theme 3: Racial/ethnic inequities in the clinical learning environment, including macro and microaggressions, created a dynamic that resulted in some students having fewer opportunities for feedback. Students’ reactions to racial/ethnic bias and aggression were interpreted by teachers in ways that resulted in teachers missing opportunities to provide formative clinical feedback. These data confirmed that clinical feedback conversations can indeed be challenging, particularly across teacher-students sociodemographic difference. We also share a participant’s perception that inequitable feedback deprive students of the knowledge and skills they need to learn and grow as physicians in training.
What can we do to promote fairness in learner assessment? On an institutional level, we believe that the time is now to create restorative, just, and learning institutional cultures committed to creating work environments where faculty, staff, and students alike feel supported to learn from mistakes and make resulting major and minor changes to policies and/or practices as needed. Administrators, teachers, and students need dedicated spaces to collaborate on designing and implementing changes needed to promote fairness in learner assessment.
This study takes a first step in centering the margins as we as medical educators grow our understanding of the dynamics of promoting fairness in assessment. Future studies should explore feedback with intentional inclusion and involvement of diverse students, teachers, and researchers at every stage of the research process from conceptualization through dissemination and application of the new learning. We thank our participants for their time and candor discussing this sensitive topic and the Group on Educational Affairs for funding our work. Thank you for your time and attention and the focus that you’ll put on this topic in the future.
Dowin Boatwright:
Hello, everyone. I’m Dowin Boatwright. I’m an associate professor for emergency medicine and population health at the NYU Grossman School of Medicine and vice chair of research in the Department of Emergency Medicine. I also serve on the planning committee for the Macy Foundation conference on ensuring fairness in medical education assessment. As a planning committee member, I work with the foundation staff to help design, implement, and now engage in follow-up activities for the conference. I can tell you that a lot of thoughtful work went into the recommendations and action steps that Holly spoke about earlier.
As she mentioned, those recommendations grew out of our discussions of the commissioned papers and the case studies that some of the authors have just summarized. As you can imagine, it’s not a simple thing for 50 people who are deeply engaged with the conference topic to draft and reach consensus around a detailed set of recommendations. We’re all very proud of the work because we sincerely believe it can help achieve fairness in learner assessment.
This is an important goal because research tells us that America’s history of racism, sexism, ableism, and other forms of oppression disadvantage learners, faculty, and patients from historically marginalized groups. It also impacts the quality of medical education and patient care. We all know this. It’s why our institutions have been adding diversity, equity, and inclusion to their mission statements for years. Previous Macy conferences have produced recommendations and reports that have been more broadly focused on diversity, equity, and inclusion in health profession learning environments, but this time the conference zeroed in on a particular component of that environment, learner assessment in medical education.
Now that the commissioned papers and recommendations are widely available through Academic Medicine, we hope several things will happen. One, we hope that those within educational institutions, medical school, and health system executives and administrators, faculty and program staff, and learners will read the recommendations and start thinking about the implications for their own institutions. This includes assessing their assessment systems, identifying needed action steps, and exploring how to take those steps. We hope that professionals and relevant roles outside of educational institutions, such as those at medical education associations, accreditation and assessment organizations, and licensing and certification boards, will read the recommendations and expand their own efforts to foster equity in learner assessment.
As Holly said, the recommendations are extensive and there are action steps directed towards nearly every role in medical education. There’s something for everyone. We all have a role to play in ensuring fairness in learner assessment. We also hope everyone will grasp the urgency that surrounds this work. Not only is it incumbent upon all of us to understand the impact of harmful bias in assessment and work to eliminate it, we’re also embarking on a technological revolution in medicine and medical education in all of society that makes immediate action imperative. Perhaps one day we’ll develop the technology to identify and correct human bias, but for now, our technologies simply perpetuate it. If we truly are committed to building an equitable health care system for the future, we must ensure that any brilliant new assessment technologies do not propagate harmful bias from the past.
Finally, as everyone reads the recommendations report, we hope they’ll think deeply about how well their own understanding of learner assessment, as well as their institution’s system of learner assessment aligns with what is optimal for producing supremely competent physicians. Traditional normative assessment is rife with harmful bias. Learner assessment can only be optimized if it’s free from this bias. Our recommendations make it very clear that fairness and equity are necessary components of competency-based medical education and assessment. We hope that everyone listening will share the recommendations report with their colleagues and help spread the awareness of these issues.
Toni Gallo:
Remember to visit academicmedicine.org to find the complete Ensuring Fairness in Medical Education Assessment supplement, including each of the papers you heard about today. You can also find the latest articles from Academic Medicine and the journal’s full archive dating back to 1926, as well as additional content such as free ebooks and article collections. Subscribe to Academic Medicine through the Subscription Services link under the Journal Info tab or visit shop.lww.com and enter Academic Medicine in the search bar.
Be sure to follow us and interact with the journal staff on Twitter @AcadMedJournal and on LinkedIn at Academic Medicine Journal. Subscribe to this podcast anywhere podcasts are available. Be sure to leave us a rating and a review when you do. Let us know how we’re doing. Thanks so much for listening.