On this episode of the Academic Medicine Podcast, guest Heeyoung Han, PhD, joins hosts Toni Gallo and Research in Medical Education (RIME) Committee members Javeed Sukhera, MD, PhD, and Andres Fernandez, MD, MEd, to discuss new research into the different methodologies used in health professions education research and how rigorous, or not, the descriptions of these methodologies are in published studies. Also covered is advice for researchers who want to more creatively and rigorously conduct and write up their work.
This is the final episode in this year’s 3-part series of discussions with RIME authors about their medical education research and its implications for the field.
This episode is now available through Apple Podcasts, Spotify, and anywhere else podcasts are available.
A transcript is below.
Read the article discussed in this episode:
- Han H, Youm J, Tucker C, et al. Research methodologies in health professions education publications: Breadth and rigor. Acad Med. 2022;97:S54-S62.
Read the complete collection of articles included in the 2022 RIME supplement.
Transcript
Toni Gallo:
Hi everyone. I’m Toni Gallo, host of today’s episode. Every year Academic Medicine publishes the proceedings of the annual Research in Medical Education sessions that take place at the AAMC’s Learn Serve Lead meeting. This year, the RIME papers, including the one we’ll be talking about today, were presented throughout the Learn Serve Lead meeting, which took place in person earlier this month. The RIME papers are also available to read for free on academicmedicine.org.
As in previous years, I’ll be talking to some of the RIME authors on this podcast about their medical education research and its implications for the field. In October, I spoke to Luca Petrey and Dr. Laura Weingartner about their research into the inclusion of standardized patient characters and actors with diverse gender identities. You can find that episode in our archive.
For the final of this year’s RIME conversations, I’m joined by two members of the RIME Committee, Dr. Javeed Sukhera and Dr. Andres Fernandez. And we’ll be talking to Dr. Heeyoung Han, who co-authored the paper, “Research Methodologies in Health Professions Education Publications: Breadth and Rigor.” And I’ll put the link to that paper in the notes for this episode. Let’s start with some introductions. Javeed, could you get us started please?
Javeed Sukhera:
Thanks so much, Toni. Hello everyone who’s listening. My name is Javeed Sukhera, I’m a child and adolescent psychiatrist and an MD-PhD education scientist, currently the chair and chief of psychiatry at Hartford Hospital and the Institute of Living in Connecticut. And I’m an associate clinical professor at Yale University’s School of Medicine, as well as a RIME Committee member.
Toni Gallo:
Great, thank you. Andres?
Andres Fernandez:
Hi everyone. Nice to be here. My name is Andres Fernandez. I’m an assistant professor in the Department of Neurology at Thomas Jefferson University in Philadelphia. And I’m also an intern in the internship program at the RIME Committee.
Toni Gallo:
Thank you. Heeyoung?
Heeyoung Han:
Hello everyone. My name is Heeyoung Han, I’m associate professor of medical education at Southern Illinois University School of Medicine. I am the director of postdoctoral program and medical education research fellowship program at the school. Thank you so much for inviting me to this podcast session.
Toni Gallo:
Thank you to all of you for joining us today. Our discussion is really going to focus on the work that Heeyoung and her co-authors have done to explore the different methodologies that are described in health professions education publications and what the authors of these studies said and didn’t say about the methods that they used for their research. I want to turn it over to Javeed and Andres to give us some context and background for our conversation today.
Javeed Sukhera:
This paper is really a gift, I think, to the medical education community. When we were reviewing it, I was very struck by the elegant and thoughtful way in which the authors chose a topic that isn’t an easy one to throw out into the literature. There’s lots of sensitivities and tensions around different ontologies, epistemologies, and axiologies in the field, but what the authors have done is really synthesize in a meaningful way what the current state of the science is. And I think it has some fascinating implications for not only our current state, but where we might be going. Andres, what are your thoughts?
Andres Fernandez:
I think it is very interesting that today we’re going to talk about the research process and rigor, and I think that’s something that is not talked about enough. And really what strikes me how that process really and the choices that we make, it can shape the research in health professions education. I’m very much looking forward to an exciting discussion.
Javeed Sukhera:
Maybe I can kick us off with the first question. There’s so many fascinating things about this paper, but I guess we should start with the fact that you open up the paper with a discussion about how the methodology of a study really shapes the kind of knowledge that can be learned. I’d love to know why you chose to explore this topic in particular.
Heeyoung Han:
Well first of all, I want to thank you for the very, very nice comments about the paper. I really appreciate that. Actually I came up with this research topic from my professional and personal reflections. I’m an educator, PhD educator, researcher, and looking back on my publications and my practice of research, I have been very proud of my publications. But at the same time I thought to myself that, oh, I can’t help thinking that I am limited to the set of methodologies that I am comfortable with. It feels like I just keep making the same stories based on the limited tools that I have. That was a kind of personal and professional reflection. And one day I came across several articles in linguistic psychology where the authors argue that language can shape the thoughts rather than language is a media to convey your thought. I know it’s a little bit far analogy but just trust me.
And I thought to myself about my identities. I was born and raised in South Korea and my mother language is Korean and I had to pick up English to study and work in the United States. And there were numerous cases where I had some thoughts, but I couldn’t really convey the thoughts in English. So I had to give up in conveying some meaning. Anyway, just like that, I just connect all the thoughts to my professional reflection about research. Just like that, if the methodology is just tool or media to convey the idea or thought or knowledge, it should be not that way. Anyways, I thought to myself that I … connecting that reflection to my reflection to research methodology, if that is a case, the research methodologies that I’m comfortable with may be shaping the nature of the knowledge that I’m making in the field.
And then I connect the thought to the organizational level, the field level. If that is a case, if the group of researchers that we are in the field are doing the similar practice, then as a collective entity, we may be limited to the set of tools or methodologies that we are comfortable with and maybe unconsciously consciously shape the field of knowledge in a certain way that we didn’t really know of. Anyway, that’s background and I talk about this idea with my AAMC GEA MESRE colleagues, and I know it is … I thought it’s crazy idea, but they were really, really excited and they love the idea and they really, really agree with the issue and they just jump in. And that’s how we started this project.
Andres Fernandez:
The article describes in detail what you found in some of these papers, but also you reflect and shed light on what you did not find, some of those absences. Can you talk to us a little bit about those absences and what you think drives those absences?
Heeyoung Han:
Yeah, that’s very, very exciting discussion point that we actually had. We wanted to observe the rigor, we wanted to observe the breadth of the existence of things, existence of rigor, existence of breath. But what we found is some really kind of obvious absences. For example, we found no discussion about epistemological assumptions in each paper, lots of authors just jump in right away methodologies, which is common practice, somewhat understandable. But because of that we don’t really check our assumptions about reality, assumption about the fact, or assumption about the knowledge. That’s something that we might want to sit back and think about. Why is that? Do we need to do that? Anyway, that’s one of the absences. Because of that maybe lots of methodology approaches like oral ethnography and postmodern approaches, critical realism, those different methodologies or epistemological approaches are scarce.
And another absence that we actually talk about is the population sample. As I said earlier when we summarized what we found is most the HPE empirical research papers were from North America, Europe, from students, trainees, collected one year using survey or interview. Other than that, it’s missing there. Why not some patient impression or performance data from non-US or -European countries? Are we comfortable with the collected set of knowledge that are generated by those people or by those certain types of research data collection integration? Why do we not have a longitudinal understanding of phenomena? If we think that … if we assume the reality changes constantly, shall we see the longitudinal understanding of the phenomena? That kind of things that we saw as missing in the field. I’m not sure what do you mean by missing, but are we comfortable with this absence? That’s the question that I would draw in the field.
Toni Gallo:
I actually want to open it up to all three of you. You’re all education researchers. I know the RIME Committee is thinking about how to foster high quality scholarship. What do you think are some of the … Heeyoung mentioned there’s this one type of research that we see most often and I’m curious what you all think about why that is the most common type of research that we see. Why some of these other types of research haven’t really made their way into HPE work?
Javeed Sukhera:
I think what this paper catalyzes is a conversation that’s been happening but needs to happen in a different way. There’s frequent tensions between qualitative and quantitative approaches, positivist versus constructivist epistemological positions. But many of those conversations happen in a way that’s often polarizing without embracing the nuance that this paper addresses. For example, learning that overall in this review, quantitative studies were actually less rigorous, highlights a lot of assumptions that are made about whether or not one type of study is inherently maybe more rigorous than the other. I often see this as a primarily qualitative researcher where, when people try to do mixed methods, they don’t really describe the rigor needed when it comes to the qualitative arm of their study. Yet your paper highlights that just because you think you might be doing quantitative work doesn’t mean that you should not think about the necessary rigor. That’s definitely one thing I appreciated.
Andres Fernandez:
And on the other end, the qualitative research had more rich descriptions, so to the point of those assumptions that were made in the quantitative articles perhaps of the methods that were used. I agree with you Javeed, there’s a larger conversation here that needs to happen.
Heeyoung Han:
Javeed, I really appreciate your point of the ironic finding in this paper that quantitative study has lesser, I mean lower, rigor scores than other types of methodology. I reflect on that a lot actually. And one of my … I know it’s a little bit maybe crazy thing, but in another study where attending physician and resident when they communicate for surgical consult, when attending trust the resident then the resident doesn’t need to report the details of the patient case because the attending assumes that the resident is good enough and they don’t want to hear all the details, just the succinct report. I wonder if this is like that. It’s organizational psychology in the field, that we have a big trust but maybe unproven trust in quantitative study where reviewers and editors, I don’t know, maybe some people even including authors, just assume that they don’t need to report the details of the rigor because they thought that it’s a proven, accepted legitimate methodology.
Ironically, they are not really good at importing the rigor evidence and you never know how they did it. Versus qualitative studies, especially emerging methodologies and not many people are familiar with, they constantly get challenged to show the evidence of the rigor, which is good actually as a researcher. That’s a good practice. But we don’t seem to be doing that for quantitative study. I’m not saying that quantitative study is not rigorous, but what I’m saying is that applying the rigorous standard should be consistent for all different kinds of methodologies. And also one thing that I would like to include is the issue that we bring up in this paper may sound like individual researcher’s fault, but I would say maybe but maybe not. It’s a collective effort.
The elephant in the room is that the publication is a very structured social practice. When you have a reviewer who are familiar with certain methodology, you can get easily published, I guess. But if you have some novel approach that nobody could really understand, you have to go for extra miles to make that happen. I think that’s a lot of work, but I think it’s meaningful so that’s something that we can think about. It’s worthwhile to do as a collective set in the field, as an author and then reviewer and editor, to leave some space for intellectual flexibility at the journal, where some people actually try out some different ways of knowing, conveying the different types of reality, conveying different methodologies. It may be awkward, it may be novel, it may be unfamiliar, but if we believe that it’s worthwhile to try, we may save some space for that intellectual playground.
Javeed Sukhera:
That’s a really important point because one thing that’s neat about medical education is that we do have a lot of epistemological diversity in terms of different folks doing different kinds of research. But your points and your work highlight that if we don’t adapt in terms of how we review, accept, or reject studies that may incentivize a creativity, a methodological mismatch or methodological borrowing as Lara Varpio and Tina Martimianakis describe, we’re not really going to move the field forward. What do you think we should be asking of our editors and reviewers in this regard?
Heeyoung Han:
Well, that’s a really big question. I think that I don’t… I have no answer, but I can just move the thought a little bit moving forward. As I said earlier, it’s a social practice. The reviewers and editors and authors move the wheel together. I would probably challenge more experienced researchers or senior researchers to take the risk, I think. I know it’s a little bit crazy, but if there’s breaking the status quo and then going beyond the boundary, which is risky, early career researchers may not be able to do that because their job is to get their work published and then they tend to follow the proven trail.
Some researchers may not, but in general, and then that kind of risk taking should be happening. And I wonder if there is some collective effort at the leadership level to make that initiative safer and meaningful. Because otherwise we may just keep creating the knowledge of HPE within the set of the tool that already we are familiar with and can ignore … not really ignoring the kind of … not really understanding that part of the meaning. Anyway, it’s as Javeed said earlier, it’s a very sensitive topic, but I think that’s something that we need to keep talking about and then do something as a pilot, even pilot at a journal or collective set of journals as well.
Andres Fernandez:
Going back to your comments on mixed method studies, you mentioned that most researchers combine quantitative and qualitative methods, but without really considering mixed methods at a unique paradigm unto itself. How do you think researchers should treat mixed method studies going forward?
Heeyoung Han:
Well, again, this is really big question. I would say probably rather than focusing on mixed method studies or methodologies, the fact that we have more mixed method approaches in the field. When I look at other field, I see more mixed method approaches happening in the field of HPE, which is very, very reasonable because medical education itself is a very complex practice, complex social phenomena. We can not only possibly react the reality in one dominant way. We really wanted to have an improvised method … collection of methodology which is based … the mixed methods philosophy which is methodology of pluralism. You collect whatever making sense to represent your reality, just like critical realism. Just like that, researchers, I think, including me, we need to really focus on the reality. What is it? Do we really cut them off to make the research practice easier?
No, we just embrace the complexity and then to look out for methodological process, epistemological process, that makes sense. Mixed method is one of them, and when we look at the papers using mixed method, they didn’t really fully embrace that approach fully. It’s more technical level where they use the survey basically, quantitative measure, and then several open ended question and then using thematic analysis. That’s it. And then, I feel like it’s fine, but there is more than that. When you try to grab the reality, the entity that you are trying to study on. Rather than just focusing on methodology itself, just try to embrace the complexity of the object of research and then have a collection of all different kinds of methodology, which end up probably in mixed method research methodology.
And another area actually is that probably for those people who are not used to or familiar with mixed methods, we can actually look into some guideline or what are the tips for mixed method or what are the things that they have to consider in mixed methods. That guideline and development of process at the editorial level and reviewers level would be very, very beneficial.
Toni Gallo:
For listeners who are trying to figure out how to write up their research in a more rigorous fashion, do you have recommendations for them based on some of the findings from your study? If any of you have recommendations just from your own work as researchers or your work as reviewers, I think it would be really helpful for listeners to get that feedback so they can think about it as they’re writing up their next paper.
Heeyoung Han:
Sure. I guess I just promote our paper here. We came up with the coding structure to review the article and we provided that in a table in the paper. A list of the coding question was really helpful for us to think about our research rigor process as well. I think that can be really quick check boxes that researchers can check, for example, like study population, study epistemology, study design, population sampling, validity, and so on. Very basic. But there is some guideline that they can use for their own study. The other way that actually I would recommend for the researchers to ensure their research quality, rigor quality, methodological quality is that they can do the kind of audit process themselves. For example, in our study we have all the artifacts that we actually created along the way.
For example, like randomized sampling, creating the literature review process, creating coding structure. All the artifacts are in our shared drives, so anybody can take a look. Oh, that’s how they did a sampling, how they decide the sample size and stuff like that. That kind of leaving the audit trail for themselves can be helpful so that they can actually fill the gap if there is some gaps. Again, they can do their own audit process and then our table can be a easy check box that they can do.
Javeed Sukhera:
Well. I think what we need to really send a message for junior researchers is to be thoughtful and deliberate and to not take things for granted. I think many people who tiptoe into medical education research might be coming from more of a positivist orientation and be really oriented more towards biomedical research. That isn’t the same orientation that one can automatically have when doing medical education research or scholarship. And to develop one’s self I think what that requires is to get outside of your own head and work with others. This is a very supportive community that we’re part of and whether there are people at your site or others, I think it’s important to reach out and assemble a team of folks that understand and appreciate this field, this audience and the conversations that are happening, particularly in educational research and scholarship.
Andres Fernandez:
And that comment on teamwork that you just mentioned. And then the practical approach really resonated when you mentioned using the scores and guides, not after the fact, but really when you’re thinking about what you’re doing, using it as a practical tool and guide to really see what approach you’re going to take with your research. I think that’s a very good application that perhaps wasn’t the intention of the paper, but it’s a product that I think would be very practical for junior researchers like myself.
Heeyoung Han:
One thing that I’d like to add as a final thought is that research publication or research practice itself is a very collective social practice. And then that also means that there are some certain rules that you have to follow and some rules creates some rules. But sometimes we just kind of take advantage by the rules, but the rules itself confine our practice. Again, we can actually reflect on our current system and current practice and then think about what else we can do better. I think that’s the methods that I wanted to convey. The journal editors, journal reviewers and authors can actually think outside of the box and then move the stone a little bit further to the close proximity of the reality as a collective set.
Toni Gallo:
You mentioned using this paper as a guide as researchers are writing up their work. You all mentioned the resources in your colleagues and fellow researchers. Are there other tools that are out there that you would recommend to junior researchers or types of tools that folks can look for if they need some help in this space? I know there’s tons of things out there, so sometimes it can be a little overwhelming to find exactly what you need, but any recommendations for how to find those guidelines or tools to help?
Heeyoung Han:
You mean guideline for research methodological rigor?
Toni Gallo:
Yeah, sure. You know how to think about the design of your study or the way you write it up or anything like that.
Heeyoung Han:
Yeah, sure. I guess there are two ways you can figure out on your own by literature review. There are tons of literature out there, how to do mixed method, for example, how to do action research, how to do a qualitative phenomenology study beyond the medical education literature. You can go outside to look it up. And then the other way is the team members, you actually ask some colleagues who are familiar with those methodology or some educational research practice. Then you can actually have some key literature from the colleague. Again, it’s a team sport to me so when we did a study for this paper, we have nine people, nine members. Some are really, really experts in quantitative studies, some are really good at qualitative studies, some are really good at mixed method, program evaluation.
We just share our expertise all together. Just like that, you can do it on your own, but at the same time you can actually get some wisdom and insight from your colleague so you can actually take some easier step … to give some heavier step when you begin the process. That’s what I would do, just ask people.
Toni Gallo:
Okay. I want to give you each a chance, if you have any final thoughts you want to leave us with, anything you really want listeners to take away from our conversation today. I’ll give everybody a chance to just share anything else they want to say before we end.
Andres Fernandez:
Yes, thank you so much for inviting us to this podcast. This was a very stimulating discussion. I learned a lot and it’s really going to … a lot of thoughts on next time I have an idea or I want to start research to really slow down, step back and really be very intentional about those initial steps that I’m taking. That’s a big message I get from reading this paper.
Javeed Sukhera:
And I think the main message is just a lot of humility. Don’t take anything for granted. If you’re going to do research in the field and you look at your research methodology, it’s important to have rationale, to be thoughtful about what you’re choosing and why. It’s important to also create room for creativity, but anchor yourself to the experts who have done the research and can support more nuanced understanding of what you’re trying to do and why you’re trying to do it.
Heeyoung Han:
One thing that I want to add to today’s conversation is that our study may sound like you need to do a rigorous study, right? That’s true. But at the same time I wanted to emancipate the medical education researchers in terms of methodology. I know it’s a little bit ironical, but just knowing that there are some more space, more room, more area that we have not fully explored, which means that you can actually have a freedom to explore some other way of knowing, if that makes sense to your reality. Yeah, rigor rigorous study is one thing, and at the same time I challenge our colleagues, including me, to explore some unexplored area to advance our field. That’s ironical, but very tightly linked messages that I wanted to convey today. Thank you.
Toni Gallo:
Well, I want to thank you all for being on the podcast today and I want to encourage our listeners to look for the paper that we discussed here as well as the complete RIME supplement. Everything is available on academicedicine.org and if you look in the archive for this podcast, you’ll find the other discussions we had with RIME authors this year and in past years, so thanks everybody.
From the journal’s website you can access the latest articles as well as our complete archive dating back to 1926. There’s also additional content like free eBooks and article collections. You can subscribe to Academic Medicine through the subscription services link under the Journal Info tab, or you can visit shop.lww.com and enter Academic Medicine in the search bar. Be sure to follow us and interact with the journal staff on Twitter at @AcadMedJournal and subscribe to this podcast anywhere podcasts are available. Be sure to leave us a rating and a review when you do let us know how we’re doing. Thanks so much for listening.