On this episode of the Academic Medicine Podcast, host Toni Gallo is joined by Kayla Marcotte, MS, Jose Negrete Manriquez, MD, MPP, Maya Hunt, MD, Max Spadafore, MD, and Dan Schumacher, MD, PhD, MEd, to discuss the role of learners in building the future state of assessment; the importance of having a patient-focused, learner-centered, equity-based system of assessment; and the opportunities and challenges posed by new types of assessment data and AI tools.
This episode is now available through Apple Podcasts, Spotify, and anywhere else podcasts are available.
A transcript is below.
Read the article discussed in this episode: Marcotte K, Negrete Manriquez JA, Hunt M, Spadafore M, Perrone KH, Zhou CY. Trainees’ perspectives on the next era of assessment and precision education. Acad Med. 2024;99(4S):S25-S29.
Check out the complete Next Era of Assessment supplement: Schumacher DJ, Santen SA, Pugh CM, Burk-Rafel J, eds. The Next Era of Assessment: Advancing Precision Education for Learners to Ensure High-Quality, Equitable Care for Patients. Acad Med. 2024;99(4S):S1-S94.
Be sure to claim your free CME credit for listening to this podcast. You have until July 31st to claim these credits. Visit academicmedicineblog.org/cme, listen to the episodes listed, then follow the instructions to claim your credit.
Transcript
Toni Gallo:
Welcome to the Academic Medicine Podcast. I’m Toni Gallo.
In April of this year, Academic Medicine published a special supplement entitled, “The Next Era of Assessment: Advancing Precision Education for Learners to Ensure High-Quality, Equitable Care for Patients.” On today’s episode, I’m joined by Dr. Dan Schumacher, one of the supplement’s editors, as well as the authors of the paper, “Trainees’ Perspectives on the Next Era of Assessment and Precision Education.” That paper and the complete supplement are available to read for free on academicmedicine.org, and the links are in the notes for today’s episode.
In our conversation here, we’ll talk about where assessment in medical education needs to go, including the role that learners can play in building this future state. We’ll get into the importance of having a patient-focused, learner-centered, equity-based system of assessment, and we’ll discuss how data and even artificial intelligence tools can fit into this model.
So with that, let’s do some introductions. Dan, would you like to go first?
Dan Schumacher:
Very happy to be here. Dan Schumacher, peds emergency medicine faculty at Cincinnati Children’s.
Toni Gallo:
Great, welcome. Kayla?
Kayla Marcotte:
Hi, my name’s Kayla Marcotte. I’m a MD-PhD candidate at the University of Michigan.
Toni Gallo:
Welcome. Max?
Max Spadafore:
Hey, guys. My name’s Max. I’m also at the University of Michigan. I am a PGY-4 emergency medicine resident, and I’m going to come on as faculty as the director of educational informatics for the medical school next year.
Toni Gallo:
Wonderful. Jose?
Jose Negrete Manriquez:
Hi, Toni. Hi, everybody. My name is Jose. I’m a second-year internal medicine resident at UC Davis. Excited to be here.
Toni Gallo:
Thank you. And Maya?
Maya Hunt:
Hi. Thanks for having me. I’m Maya Hunt. I’m a PGY-3 general surgery resident and surgical education fellow at Indiana University.
Toni Gallo:
Thanks all of you for being on the podcast today. I’m looking forward to our conversation.
Dan, I want to start with you. Can you just tell us a little bit about this Next Era of Assessment supplement, what inspired it, and what are our listeners going to find if they look at the papers here?
Dan Schumacher:
Yeah. I took this inspiration actually from a paper that was written by Cees van der Vleuten and Lambert Schuwirth, who are really two of the people in medical education that wrote the history of assessment. And they wrote a paper about the eras of assessment that they have lived through. And it started back in the 1960s. I think something we’re all familiar with, assessment was really focused on measurement back then, the idea of standardized exams and everything. And they went through eras since then that have gone to include qualitative, written comment data, how clinical competency committees and human judgment can be used to make decisions, and the importance of different types and sources of data to really give the most complete picture of someone’s performance.
And I was really struck by how these eras are all about how we do assessment. Tests, observations, committees, lots of different types of data to give a good picture, and they didn’t really focus on why we do assessment. And so we’re really pulling on the why in this supplement. And the why is to ensure high-quality care for our patients, to do that in an equitable way, and a way, as we’re going to talk about today, that really centers learners and help prepare them to be able to meet the needs of patients. So that’s what the supplement is about.
Toni Gallo:
What are you hoping the supplement is going to achieve now that you’ve put out these papers? What do you hope listeners will or readers will take away from all of this?
Dan Schumacher:
Yeah, I think that there’s two main things. One, this supplement finds itself in the competency-based medical education domain, which really says, “Put learners in the driver’s seat for their education and help people get to achieve educational outcomes that help them be able to meet the needs of patients.” And so I think that we oftentimes do an okay job of centering learners and trainees, but I don’t know that we always have the patient in mind as much as we could, so I’m really hoping that this will draw attention to that important piece of CBME.
And the other thing the papers in this supplement do is really lean into precision education, the idea of really meeting people where they’re at in their development and helping them in an individualized manner take the next steps in their development.
Toni Gallo:
I want to turn now to a bunch of those concepts that Dan just mentioned and dig into each one a little bit. And Jose, we’ll start with you and this idea of being learner-centered. That’s one of the key tenets that you all talk about in your paper and that is woven through the whole supplement.
Can you talk a little bit about what that looks like and how learners can really be involved in building an assessment system?
Jose Negrete Manriquez:
Yeah, definitely. And I think Dan put it beautifully, “Let’s put learners in the driver’s seat.” Learners are the people that are involved in having these assessments done on them. And so I think that they know firsthand what works and what doesn’t. And so I think that the way that looks like is giving learners the ability to co-create these assessments, give them a seat at the table, give them power to say, “This actually works and this actually doesn’t work in practice.”
I think one of the things that I like to compare it to is when we think about community-based participatory research. Historically, there’s this top-down approach where researchers, they have the center of the information and they come to the community and say, “Well, diabetes is the problem. And so this is the solution and we’ll do some focus groups and this is what you do to fix it.” But I think taking a different approach and involving the community early on and saying, “We know that there’s this problem of diabetes and we know that the majority of the reasons why that happens is because there’s issues with food and there’s issues with exercise.”
And then hearing from the community firsthand, they can tell you, well, maybe they don’t have enough places to exercise. There’s no parks. The walkability index is low. Around the corner, there’s mostly liquor stores and there’s not really any grocery stores or affordable places to get food. And then you start thinking, “Well, that makes sense. That’s why there’s diabetes predominately in this community.”
And so I think that involving the people that this mostly affects is definitely a way forward.
Toni Gallo:
Have any of you been involved in thinking about assessment at your institutions, either as medical students or as residents now? I see some head nods.
Maya Hunt:
Yeah, absolutely. So I’m getting a master’s in health professions education at UIC currently, and we take a whole class on assessment, and it’s a very complicated topic that’s a lot harder and more nuanced I think than I really recognized beforehand. But knowing what our goal with the assessment is, what are we doing, why are we doing it, and how are we assessing our learners in the most equitable way is something that’s important to think about and address.
And like Jose is saying, it’s centering the patients, the people who are going to be affected by us and our learning, and it’s centering the learners who are going to be affected by the assessments. And so rather than coming from a place of what’s easiest for us to do as an assessment, what is going to be the most impactful to give us the information we need to ensure that we’re giving our patients adequate care and our learners are meeting our objectives?
Toni Gallo:
Anyone else?
Max Spadafore:
I feel like it’s kind of hard to make it through medical school these days without some sort of revamp of the assessment system. And this is what Dan was talking about with the many different eras of assessment.
I think the reason that keeps happening is because assessment is really, really hard. It’s essentially using these extremely… I often use an analogy of taking a temperature. It’s like using these really noisy temperature probes to try and figure out the temperature of a screaming baby. And so just using the back of your hand is not going to give you an accurate number of a patient’s temperature, but that’s all we have right now in terms of the accuracy and reliability of our measurement methods.
And so what we keep doing, I feel, is we keep going around in this never-ending cycle of, “Okay, we’re going to change the way we do assessment. We’re going to change the way we do assessment,” but each time we don’t actually upgrade the probes that we use to determine the competency or ability of the people we’re trying to teach to be physicians.
And so that’s really what I think this precision medical education is going to be about is about trying to find some probes that actually give us a better, more accurate number as we parse out that unknowable function of assessment, like a meta-theoretical level. And so I feel like when I was in medical school, I feel like we changed our assessment form like four times and I think that’s because we keep working with the same probes, but we don’t attempt to change those. And that’s why I’m really excited about precision medical education because it gives us the opportunity to upgrade those sensors.
Toni Gallo:
I think this is a good chance to talk about data then and how data fits into what are you collecting from learners, what does that mean, what are you sharing?
So Maya, maybe you can talk a little bit about data in this system. Dan and his coauthors in the foreword talk about data as this essential currency in assessment and in precision education. So how do you think about data here?
Maya Hunt:
So obviously, data’s important. It’s information. And in order to utilize the assessment for what we want to do, we need to have information coming in that can inform the output. Again, we really have to think about what is the goal of the assessment? Is our assessment, is it narrow, is it broad? Is it to ensure that someone has achieved a competency with a specific skillset or in a specific diagnostic criteria or that they are capable of being a safe physican and can graduate their residency program? There’s different levels of that.
And so what kind of data do you need to inform the outcome of this assessment decision and why does that data help us to achieve this goal? And how are we going to go about getting this data? Is it going to be from the electronic medical record? Are we going to be controlling for potential biases that are within the systems that we operate in, including are we measuring somebody’s clinical ability based on their patient outcomes, but not taking into account the patient-level factors that we can’t necessarily control?
So making sure that we’re thinking about the benefits as well as the risks. What is the risks of data acquisition and implementing these assessments for trainees? Is it very niche, granular data that you’re going to be getting that could be leaked or could have impacts on their career? Or is it going to be so much data that it’s unmanageable? How are we… There’s a lot of different aspects to thinking about this that I think need to be more fully thought through.
And perhaps what Max was talking about with the probes and not updating our systems is that a system is perfectly designed to get the outcome that it does. And so our current system, where we are not necessarily learning from how assessment has benefited or failed us in the past and continuing to just pivot to the next thing and try to push it forward, this is taking a step back in the supplement and looking at the field more broadly and saying, “How can we be very intentional about utilizing assessments and the data of our learners to benefit these patients and keep a narrow focus on that?” Trying to make sure that we have a plan in place going forward so that we’re not just running after something, hit a wall, pivoting, running after the next thing, hit a wall, pivoting as we continuously seem to do with these assessments.
Jose Negrete Manriquez:
And just to add to what Maya said, it’s really hard. These assessment systems, it’s really difficult. And I think I just want to reiterate this is why it’s so important to involve learners in the process. In this paper, we had our medical students, residents, and we had a fellow, and at each level of training, it can be very different. Your exposure to patients is very different as a medical student than it is as a resident, than it is in the clinics versus in the ICU versus in the emergency department.
So that’s why it’s so important to include learners in this process because like Maya’s saying, it’s so difficult. What is your ultimate outcome? That really mattered in terms of what kind of data you’re trying to collect, and so I just want to re-emphasize that.
Kayla Marcotte:
Yeah. And to build off of that, thinking about the outcomes of the data, I think for people who are designing and implementing these systems, it’s really important to think about the types of data that are being collected too. I think just collecting more quantitative data on a Likert scale of, “They did so well with this patient encounter,” or having learners complete more tests to test their medical knowledge and getting numbers from that, that is useful and there is a place for that in medical training, but there’s also a really important piece for qualitative data, feedback from people that are actually interacting with the learners that maybe is not captured in more traditional test situations.
Because there is a lot of rich data to be collected from actually seeing actual feedback from people who are watching learners learn and grow and have interactions with patients. And I think that there’s a lot of room for growth in that area of collecting data for assessment and medical education and a lot of room for thinking thoughtfully, then what do you do with the qualitative data and how do you use that for promotion or remediation decisions for a learner?
Maya Hunt:
I think one of the other things to think about, Kayla, with that is when you do have these large amounts of data, who’s managing it? Where is it going to live? Is the department going to have to pay for additional space in the cloud? How are we going to handle all this data as we get more and more and more of it? Is that actually going to be helpful in our outcomes? Or when we talk about the precision education, the training, we need to, again, be precise. It shouldn’t just be about selecting as much data as we can because we can, but actually targeting what is going to help us achieve our goals.
Max Spadafore:
Yeah, because just collecting the data carries risks. One thing we were thinking about as we start thinking about these large language models and ChatGPT and all that fun buzzword stuff is that a lot of things… We very much operate under the paradigm of, “Okay, we can collect data that we need for quality improvement.” That’s always been like, “You’re okay to do that. You can collect educational data, including identifying data as needed to try and improve the quality of your educational program.”
But when we’re talking about siphoning huge volumes of data, for example, from the EHR, if you’re taking a look at me as an emergency medicine resident and every order I put in on every patient and how fast I put it in and how accurate the orders are and how often I change them, if you’re looking at all of that, well you’re bringing in patient data, number one, and number two is the automatic, almost societal consent that you get for, “Oh, you’re doing quality improvement data,” does that change? Is this the equivalent of ChatGPT sucking up all of the New York Times and the New York Times saying, “Well, hey, humans wrote that. We own that copyright”? If my data gets mined to improve these systems and it’s mined in such a granular way, does that require a different consent than the old like, “Oh, you’re using my outcomes data to try and improve your assessments”?
And so those things, those are unanswered questions, I think, about this level of granularity and this level of data collection that I don’t think the field has reckoned with yet at all.
Maya Hunt:
Yeah, I don’t know that it’s ready for it. Also, what do you do with the information if you’re not putting in your orders a couple of seconds slower than somebody else? Again, it’s like what are these being used for and why are we collecting it and is this actually going to be helpful information or is this simply going to further reduce us to being cogs in the machine in a for-profit health care system?
Jose Negrete Manriquez:
And I think I would just like to add, building on what’s been said right now, and then something that Kayla mentioned is I think a lot of times when we think about assessment, we think about test scores, multiple-choice questions, Likert scales, all these ratings, but the qualitative data, I think it’s really important too and I think it’s important to consider how it’s going to be used and how it’s going to be used to help learners with their progression in terms of becoming the physicians, the providers that they want to be.
I also think one of the things that we talked about in the paper is thinking about structural racism, thinking about systemic bias, thinking about the way that sometimes this language can be used in a negative way for specific populations. For instance, I think there’s a lot of data out there about the Medical Student Performance Evaluation and the way that White applicants are described compared to other applicants. I think they’ve been described as outstanding, best, exceptional, whereas other applicants have not. And also, there’s gender involved in how women are described compared to men, usually more caring, compassionate versus men differently.
And so I think it’s important to consider those things too and how that’s going to impact learners’ assessment and achieving these outcomes that we want. How are we going to use these things to prepare our physicians to best equitably care for our patients that are in most need sometimes?
Toni Gallo:
I want to come back to AI and to thinking about equity and bias, but first, I want to ask you all something else with collecting all of these data about learners is what gets passed forward and how does that affect a learner over the course of their education continuum? And how are you all thinking about that? If something is collected, data is collected about you as a medical student, what gets passed on to your clerkship directors, to your residency programs? How are you all thinking about now that there is so much data collected and what gets shared and how that might affect you as you move through your education?
Kayla Marcotte:
We talk about this in the paper, but all of these decisions about what is going to be moved forward, what will people know about me before I even show up on day one needs to be really clearly communicated to trainees because of course, their educational data, it has meaning behind it. Some of it is very useful to convey. I do surgical education research, and so we work with predicting performance on a surgeon’s next procedure that they’re going to perform. And so having the attending have the ability to see, “Oh, this resident has performed this before, maybe I can give them more autonomy during the case,” or, “this is their first time to see this kind of procedure. Maybe I need to be a little bit more hands-on,” there is value in that, but also it’s important to remember that everything has a story behind it and there’s a real person who’s a real learner in that situation.
So for instance, maybe someone’s numbers are lower than you might anticipate, but maybe they had to go through that assessment during a hard time in their life. Maybe they had lost a family member or had another personal situation going on, and that’s not captured in numbers.
And so always just remembering that as data is moved forward, we just always need to remember that there’s real people behind these numbers and just being very thoughtful about how they are used and communicating clearly to learners how they’re going to be used. That way they can go into a situation prepared to talk with their supervisor about whatever they’re about to do.
Max Spadafore:
I think it’s important to recognize that data does move forward and does move at least around. There’s a joke that we have in our medical education research group that’s vibes-based medical education where you’re using Kayla’s example of a surgical procedure. Well, every time you walk into a room with an attending, not that I ever do surgeries or would ever be good at them, but you walk into a room with an attending and your reputation enters that room first. You are pretty quickly, I feel like, in residency, dropped into this box of hard-working or not hard-working, quick on the pickup or not quick on the pickup, smart or not smart. And you live with that your whole residency.
And there’s at least some degree of clean slate-ism as you move from different stages of medical education to the next, but to an extent, all of these worlds, EM especially, they’re small worlds, and so everybody knows who you are and has heard about you.
And so the question is, is it better to communicate that data in an intentional way, in a way with maybe rules or guidelines or at least a system around it if it’s a well-designed system, or… Because right now, the data’s being communicated, it’s just, “Oh, yeah, he’s great,” or, “she’s great,” or, “he’s ehhhhhh.” That type of thing. And the question is, what’s going to help our learners more? And I think a well-designed, feedforward system for performance would be advantageous to our learners versus the, like I said, vibes-based medical education that happens right now.
Maya Hunt:
Well, and you’re exactly right. I think it’s also prone to confirmation bias and reminds me also of diagnostic momentum. I remember learning in med school when somebody assumes that a patient has some sort of personality disorder but was never formally diagnosed, they write it in the chart and then every chart from then on says that they have this personality disorder, but they were never again formally diagnosed.
It makes me think about that with a reputation is if you’re getting an assessment or you have a bad day or you have something like… I was one of the people that Kayla was describing, my Step 1 score was not excellent because I was going through something, and Step 2, I did a lot better. And to me, when people ask about the story between those, that’s when I know they’re using the assessment in a way to get to their goal of understanding me.
And that’s what I think we should be doing is looking for… utilizing the data that we’re moving forward, but also looking to ask questions and look between things to look for, how can I use these numbers to understand a person rather than put them into a bucket? And so that’s what I have some fears about with precision education is how is this going to be utilized, but also is this going to be weaponized?
Jose Negrete Manriquez:
Yeah, I think that’s well said. And I think just to build on that, I think that’s why it’s so important to think about the outcome and in terms of why this entire supplement issue was done is to ensure equitable care of our patients. So all this assessment and all this data, what’s the purpose? Is it to make doctors that are going to do really well in multiple-choice questions so then when they see a patient, they’re going to have, it can either be A, B, C, or D? Patients don’t present like that. It varies.
And I think that that’s the reason why it’s so important to think about the patients ultimately as the outcome. And then these systems, I think some of the things that we’ve talked about is each person has different learning trajectories, but also I think that there’s life that happens and we’re human, and sometimes things are going to be going on in life and you’re not going to be able to perform with this multiple-choice test, but performing doesn’t mean learning. And so I think it’s important to consider that.
And I think the last thing that I think about is about this concept of the Community Cultural Wealth from Tara Yosso, which talks about learners coming in, usually it’s students that are from underrepresented backgrounds, and they bring in different types of capital. They bring in linguistic capital, resistant capital, aspirational capital, where they bring in these soft skills, these different approaches to the way that they communicate with patients, the way that they understand situations in the community. I think that that adds to the patient experience, it adds to their adherence with medications, with treatment. And I think that that is something that’s really valuable too to think about when we think about assessing learners.
Toni Gallo:
I want to stay with this idea of equity, and Kayla, we’ll turn to you now. Equity and bias come up a number of times in your paper, whether it’s thinking about mitigating rater bias in assessments or this ultimate goal of providing equitable care for patients. And can you talk about how you all thought about equity as you were writing this and maybe some of the other things that our listeners should be considering as they’re designing assessments and assessment systems?
Kayla Marcotte:
Yeah. So I think your question has really two parts. How do we put equity at the forefront of assessment for learners and how do we ensure that patients are provided with high-quality, equitable care?
And so first, for learners, I think everyone on this call and listening can agree that going through medical training, they have had the experience of experiencing an unfair assessment during their training, whether that is from a senior trainee or attending who just didn’t like them for whatever reason, or the very stringent rater that always gives everyone a two on the scale, no matter how good the learner is.
I think in order to ensure that equitable assessment occurs, we need to identify those kinds of assessments, where there are systematic discrepancies in how a faculty is rating students, where people are experiencing unfair feedback from people because small differences in assessment in medical education can really have huge impacts in the outcomes that learners have. If you get… sometimes during a rotation for a medical student, you might get two evaluations, and if one of those evaluations is very low because said attending always gives everyone a low rating, that can impact your ability to move forward in matching for that specialty. And so thinking about how those things come together is really important for ensuring equitable assessment for learners.
There’s lots of ways to think about how we can mitigate that. Going back to collecting data, we collect data from raters too, from faculty as well. And so we can identify where maybe faculty could use additional training and how to give unbiased assessments. And we can also use a lot of the different AI tools to identify where maybe there are patterns in unfair ratings, and then we can adjust those ratings to adjust them up or adjust them down depending on the need. And I think that those questions, as more data is collected, are going to be able to be answered in the future, how we can best do that in the system to make sure that everyone is on the same playing field when they’re getting assessed.
And just to return to what we’ve been talking about this whole time, I think augmenting learner voices in the design of assessment is really a great way to ensure that assessments are equitable because learners may be experiencing bias in assessments that the designers don’t realize or outcomes from the assessments that are designed could be used in ways that are unfair or not with the original intent. And so always bringing a learner into the room to give their voice or opinion is also important.
And so then the second part of your question, how do we ensure equitable patient care in all of this assessment, because as Dan said at the beginning of this, the why behind all assessment is to ensure high-quality care for our patients. And so I think there’s a lot of different ways that we can think about this. There are lots of different frameworks for considering ethics in education.
One I’m more familiar with is the Faden and Kass framework for ethics and learning health systems. And so that system, that framework really puts learning activities at the center and always looking at every learning activity and how does it affect patients, how do we ensure that the patient is receiving high-quality care from this activity? And also, thinking about the different rights of learners to learn versus the rights of patients to get equitable care, I think as we design assessments and just medical education curricula in general, we just really need to continue to think about that, balancing those needs and making sure that everyone is receiving high-quality care and receiving a quality education.
And I think to build on that too, sorry one more thought, just throughout the design and implementation of all assessments, there just need to be a lot of checkpoints and the willingness to, if something is not working, to change it. That’s a big thing in implementation science. It’s just sometimes you try it and you fail and you have to adjust on the fly. And that’s, I think, okay. And so just always keeping that in mind that getting feedback and trying to adjust is not a bad thing.
Toni Gallo:
Dan, I want to ask you, how does faculty development come into the supplement? Are there specific papers or ways that you all are thinking about ensuring that faculty are trained for this new kind of assessment?
Dan Schumacher:
Yeah, I don’t know that it comes up in a large way in the supplement. And actually, I would point back to this paper, which I think really advocates for having trainees have a meaningful seat at the table, and another paper that I think really frames and that this paper also speaks to that having a diverse group of trainees at the table I think is really important.
And I think that when you’re doing that, I think that that means… from planning to all the way through implementation to making improvements. And I think that means what you need to be doing with faculty development. I see assessment as a system. It’s not just what tools are you going to use, but how are you using them, how are you improving the use of them? And I think that having trainees at the table to give feedback about how faculty are using them in intended and unintended ways and how you continue to make improvements on that, I think is really key.
Max Spadafore:
I think there’s a lot of talk around precision education and data-oriented educational methods in general, that it’s like this monolithic thing that you set up and then you push out forward into the world. And I think it’s very unrealistic to assume that any of these systems are going to work that way. There’s no… Like Kayla was saying, you’re not going to end up with perhaps the most equitable, or the most effective even, system right out the door. You need to have mechanisms on the back end. And when I say mechanisms, it usually means money on the back end to support the maintenance and continuous quality improvement of these systems. And that costs money, that costs time, and it’s often overlooked.
And so nothing is monolithic here, and I think it’s easy to be like, “Oh, this is a catch-all. This is a paradigm shift,” and it is, and that’s amazing, and my career is riding on that, but it’s the maintenance that’s the hard part. And when things get boring, frankly, and perhaps the most important part.
Kayla Marcotte:
On that maintaining phase, I think it’s important for medical educators to also remember that there are other experts out there, that this is what they do for a living is maintain systems and look for quality-improvement work. And so bringing in outside experts who maybe aren’t practicing medicine or don’t have a background in medical education, they can still bring a lot to advancing these systems and optimizing them to make them more effective and efficient. And so I think that’s just an important thing to think about as we’re designing all these systems as well.
Maya Hunt:
Absolutely agree.
Jose Negrete Manriquez:
And just something to build on what Max was saying, and with the editors mentioned this in the foreword about… there’s only a small percentage of funds that actually go into education and medical education and these assessment systems. And so I think one of the things is how do we increase that? How do we increase the funding?
And I think one of the things I always think about too is in some of the… So I did a master’s in public policy and took a bunch of education classes there, and I think a lot of times, a lot of the professors talked about, well, promotion really for them looked like doing more research and getting grants, and it was less about the classes that they taught and less about the evaluations that they were getting. And so there was no real incentive for them to invest in the way that they taught students, the way that they delivered the information. And so I think that that’s something important to think about.
And the other thing too is involving learners. I think learners are vulnerable in a sense that going through this training, especially medical education, we talk about how there’s an increase in the information that’s available out there now. Learners are expected to know so much more, and yet we’d have technology and things to augment that and to help, but I think it does make things more difficult. And so how do we compensate learners for their time that they take away from their studies, especially in going back to the vulnerable and underrepresented in medicine where maybe for them, they can’t afford to take time away from studying, or think about maybe incentivizing them to participate would be helpful because I think that sometimes they’re having to take out a bunch of loans and their families may not be able to support their medical school journey as much as maybe other learners may be able to. And so I think that that’s also important to consider.
Maya Hunt:
Yeah, I think what you’re describing a lot of, Jose, is that assessment for the most part is reactive. And I think this precision education is trying to get a little bit in front of that and see how can we be proactive in this? And essentially what we’re doing is changing the system. Like I said, the system is optimal for the outcome that it gets.
The outcome that we’re getting is that education is underfunded, under-prioritized, and is not the currency of academia. Publications are. And that’s okay to have that as a currency, but I do think there should be other currencies available, such as teaching and mentorship. And if we are investing in people who are focused on those things, I think we will continue to have that long-term investment in education. But I think it comes with restructuring how reviewing the system of what is prioritized and valued, and if our value is in good patient outcomes, we need to have good education, which means we should probably use our resources into promoting education for that outcome.
Toni Gallo:
I want to turn back to artificial intelligence. This has come up a couple of times in our conversation so far.
So Max, we’ll start with you. Can you talk a little bit about maybe AI tools that are already being used in the assessment space or where we might think about AI helping us to achieve some of the goals that we’ve talked about, whether it’s having such a huge volume of data and needing to go through that or some of the other ways where AI might be helpful here?
Max Spadafore:
Yeah. Oh, boy. Let’s go. So the short answer is AI has actually been used in medical education for a long time, at least since the ’90s, you’ve seen applications of AI towards medical education. Now, recently with these large language models, which are the ChatGPT-type conversational models, you’ve seen really an explosion in the field of AI in medical education moving from, I’ll say, niche to buzzword phase. And along those lines, what you’re seeing is a lot of reproduction.
I want to ask a question to everybody here, which is, let’s say that I create an AI algorithm that can accurately reproduce the competency ratings of a competency committee, like a residency or medical school competency committee. It takes a look at evaluations, it reads the text, it looks at the numbers, spits out a “Hey, they’re doing good,” or “no, they need some remediation.” My question is, does that add to the system? Does that add to the current educational paradigm, or is it just replacing something that we’re already doing now? It’s cool. For me, it’s really cool. I think it’s really cool that a computer could do that, and 10 years ago, that would be absolutely unheard of. But the question is it actually pushing forward the boundaries or is it just slapping AI in front of something we already do as a system and we already know is flawed?
And I think if you look at a lot of the medical education literature, the AI and medical education literature right now, I’d say about 80 to 90% of it is about reproduction of things that we do. And now sometimes that’s great. If that makes it easier for humans to do something or cheaper because we don’t have a lot of money in medical education, that’s great and important and is not to be discouraged, but I think we need to move forward as a field and start thinking a little bit more, especially in the context of precision education, about what AI can do for us that is perhaps a little bit more transformative or a little bit more, it’s not a word, but augmentative in terms of its function.
And so I’m really interested in how can, in the context of precision medical education, AI do things that we couldn’t do before? I think Kayla mentioned looking at the quality of assessments, like narrative feedback. Everyone knows a terrible eval, a “good job, read more,” from a three, four-sentence long that describes the behavior of the learner, talks about how they could do a little bit better and links those things together. And we can tell those apart, but no one pays anybody to actually do that auditing of feedback. And computers can do that now, AI can do that now, and that’s where we need to push.
I think overall, the field is very ripe for this push into now we’ve come up with a lot of proof of concepts, a lot of cool things we can do, it’s time to make the rubber meet the road and really implement these things. And that’s why the supplement was so exciting because it’s this, “Hey, this is a thing now and we need to start working on it.”
And the last thing I’ll say is looking towards the future, I think every major health system now is looking into ambient AI scribing, and I can tell you I’m really looking forward to it where you just bring a phone with the algorithm into the room and it writes your note for you as you talk to the patient.
I want to think about ambient AI education. So this scribe follows you around, listens to your discussions on rounds, which take from 9:00 AM to 5:00 PM, I think, and creates a summary of the education that happened on that day of rounds, creates a summary of the feedback that was given. Like, “Oh, I wouldn’t necessarily think about that antibiotic. I’d think about this one.” The things that attendings, that supervisors say in passing that just disappear into the educational ether because we can’t capture them are all of a sudden now going to be capturable. And that’s the 10-year push. That’s where we’re headed is into this ambient gathering of educational outcomes and data. And that carries a lot with it, but I’m just super excited for it.
Toni Gallo:
How do the rest of you feel about that?
Kayla Marcotte:
I mean, I think it’s, just like what Max was saying, very exciting. There’s so much potential for how AI can be used in education. For example, if feedback is collected from an attending, whether it’s from their voice or from written feedback that they give, we can use these models to then identify resources, send articles then to learners so that they can learn about a disease state or something, like that antibiotic that they didn’t understand in the first place. And by using those tools that we have at our fingertips, we really could make learning easier and faster and all these things that are important in medicine. And so I’m excited about it too. I think there’s so much potential and just the world is open for this.
Maya Hunt:
I agree there’s a lot of potential. It does make me a little nervous because I wonder, is precision education then just going to become the next new assessment model that we rammed through because we didn’t think about it enough and how AI is really going to impact health care and our assessment systems?
So it makes me hope that… I want to see a lot more intentionality and thought put into how AI is being used rather than just slapping it at every problem. I think it is a really powerful technology, but I think we’re going to need to be careful with it and thoughtful with it, just like we use any other tool. So it does make me a little nervous, I’ll say.
Max Spadafore:
Yeah, I mean, who wants a microphone listening to them the entire time they’re on rounds, right? That’s a tough sell.
I think an interesting model… I think, Maya, what you’re talking about is governance. And I think a lot of health systems, maybe not a lot of health systems, but some health systems like I know U of M does, they have a clinical intelligence committee and their job is to… Every algorithm that’s going to be deployed in a patient-facing context has to go through the CIC, the clinical intelligence committee, and it gets evaluated on a usefulness of cost and an ethical potential for bias and potential for harm standpoint. And up until now, there really hasn’t been any discussion of doing that in an education realm at all, but I think we’re going to need one. We’re going to need an EIC, an educational intelligence committee that sits and governs these proposals and these ideas and also helps them come to fruition because usually it has a lot of good stakeholders on it.
And Jose, I’m thinking that’s where you can have lots of different, a very diverse group of trainees sitting on this committee representing the actual stakeholders that it’s something you can’t even really have on the clinical intelligence committee because usually the patients don’t sit on those committees. But you can have that with the educational intelligence committee as you develop a model of governance that includes the right stakeholders and runs the idea past people. Is it okay having a microphone on for four hours a day during the entire learning process?
And so I think being, like Maya said, being really intentional about governance is going to be critical to keep this from turning into a dystopian nightmare scenario.
Jose Negrete Manriquez:
Definitely. And I think the thing that I’ll say is that in addition to what everybody said about the checks and balances that we need in order to really implement this well, I mean, I like the thought of having something like this clinically. And I think about Atul Gawande’s book on checklists where maybe AI can have a role in doing more of the mundane tasks that we sometimes do from the day-to-day. Sometimes for me personally, writing the notes is not as helpful. So if I can have something that can do the notes as I’m talking to the patient, it’s the one less thing that I have to do that I don’t feel I really value in my education, and then focus that time on doing something more productive, doing more complex care, doing something else with my time. So I think that that is an avenue where it can be helpful too.
Toni Gallo:
All right. We’re just about at the end of our time, so I want to give everybody a chance. If you have any final thoughts that you want to share with listeners or anything that we didn’t talk about yet, I’ll give you each a chance. Dan, I’ll start with you, and then we’ll go around.
Dan Schumacher:
Yeah, this has just been such a rich discussion, lots of things to think about as we plan for the future. I still am really stuck on the analogy that Max put forth of the person putting the hand to the back of a child to determine the temperature.
And the reason I’m stuck with this is for something that he… I agree with everything that he said in that analogy, but I also think that about this through the lens of how we sometimes can be too deeply rooted into what we think and that we think we know. So I’m imagining the parent that’s like, “Well, I know they had a fever because I felt it with the back of my hand. So why would I do a different way of doing this? They obviously had a fever.”
And I think that this analogy plays for sometimes… Sometimes we get too stuck with our current way of doing things and we think it’s telling us what we need to know, and sometimes we’re like, “I know it when I see it,” and I don’t think we know it when we see it. And I think that’s a reason that we need better assessment tools, a reason that we need to be not rooted to things we’ve done in the past, but maybe be a little bit more blue-sky, open-minded thinking for the future.
Toni Gallo:
Max, any final thoughts?
Max Spadafore:
Well, I mean, we just need more of this. What a cool group of people and what a excellent and diverse set of opinions. And I think there’s a lot of isolationism in this field where everyone’s trying to work towards their own thing. And if anybody wants to start a conference on precision health, hit me up and pay me some money.
Toni Gallo:
Jose?
Jose Negrete Manriquez:
I just want to thank everybody. Yeah, just echo what Max said. I feel really lucky to be part of this discussion, work with this wonderful group of individuals, and I’m excited to move forward and see how we can integrate this and how it can be a fruitful thing for learners, for patients and make it in an equitable way.
Toni Gallo:
Maya?
Maya Hunt:
Toni, Dan, thank you so much for having us. I really appreciate the opportunity to be able to speak about this and to participate in writing the perspective piece. It was really lovely to come together with so many different specialties, levels of training, institutions, and all come together to create a collective voice. And Kayla was a fabulous leader as she was our med student, and we really wanted to make sure that she got to see a group of individuals who are all in education and saying, “Well, you not only have time, but also we want you to have a first-author paper,” and wanting to promote and bring the next generation up with us, I think is really demonstrating us in action how education can change things and how we can do things a little bit differently.
So it was a fantastic time to be able to write it, and thank you all for having us.
Toni Gallo:
Kayla?
Kayla Marcotte:
Yeah, I want to echo just what everyone else is saying that this was just a really great opportunity to learn from a lot of different people and just building community around these topics is just so important because as Max said several times, AI in education, precision education, big buzzwords right now, but there’s so much good thinking going on and great intentions on how to use this in a medical education. I think the opportunities abound and the more we work together and really talk through some of these harder concepts that are maybe not easy to talk about, but so important as we’re crafting these systems, I think as we build that community of practice around medical education, just I think this could be really great for everybody. So just looking forward to what the future holds.
Toni Gallo:
If you all want to keep writing about these topics, submit your work to Academic Medicine. That goes for all the listeners out there too. I think these are all important things that the field is going to have to grapple with. So send us your scholarship. We want to know what everybody’s talking about. We want to hear about your good ideas.
And the paper we talked about today, as well as the whole Next Era of Assessment supplement, is available to read for free on academicmedicine.org. Definitely go check all of that out. Thanks very much. Thank you all for being here.
Maya Hunt:
Thanks.
Max Spadafore:
Thanks, guys.
Kayla Marcotte:
Thank you.
Toni Gallo:
Be sure to claim your free CME credit for listening to this podcast. You have until July to claim these credits. All you have to do is visit academicmedicineblog.org/cme, listen to the eight episodes listed, then follow the instructions to claim your credit.
From the journal’s website, you can access the latest articles and our archive dating back to 1926, as well as additional content including free eBooks, other article collections, and other supplements that have been published in Academic Medicine. Subscribe to the journal through the subscription services link under the journal info tab, or visit shop.lww.com and enter Academic Medicine in the search bar. Be sure to follow us and interact with the journal staff on X, formerly Twitter, @AcadMedJournal, and on LinkedIn at Academic Medicine Journal. Subscribe to this podcast anywhere podcasts are available. Be sure to leave us a rating and a review when you do. Let us know how we’re doing. Thanks so much for listening.