Below is the transcript of the following Academic Medicine Podcast episode:
Advice from a Master Peer Reviewer
Originally posted August 2018, Reposted September 6, 2021
Read more about this episode and listen here.
Toni Gallo:
Hi all. This week I am reposting a discussion I had with Academic Medicine’s former Editor-in-Chief David Sklar, and one of the journal’s Master Reviewers, Carl Stevens. David and I spoke to Carl back in August of 2018 about the practice of evaluating a scholarly article as a peer reviewer. Carl is a longtime reviewer for Academic Medicine and a 10-time winner of our Excellence in Reviewing Award. I hope you all will find Carl’s description of his process for evaluating submissions, including some very practical advice about the logistics of completing a review and what he looks for in a submission, valuable for your own work.
Toni Gallo:
If you’re looking for other resources to expand your skills as a reviewer, check out academicmedicine.org. Under the Journal Info tab, there’s a For Reviewers link. There, you’ll find reviewer policies and guidelines as well as advice and recommendations from the journal’s other Master Reviewers. There’s even a practice exercise so you can try reviewing a research report manuscript that was submitted to Academic Medicine, and you can compare your review comments with those of the actual reviewers.
Toni Gallo:
I want to say a quick thank you to our current peer reviewers for their thoughtful comments and their recommendations, which really help us maintain the highest standards of publication and help improve the integrity and the quality of the articles that we publish. If you’re interested in becoming a reviewer for Academic Medicine, you can email your inquiry and your CV to academicmedicine@aamc.org. Your CV will be vetted, and if there’s alignment between your expertise and your experience and the journal’s mission and focus, we’ll invite you to become a reviewer. We always need reviewers who are skilled in both quantitative and qualitative research methods as well as in literature review methods. With that, I’ll turn you over to my conversation with David Sklar and Carl Stevens from 2018. Here’s Advice From a Master Peer Reviewer. Thanks for listening.
Toni Gallo:
Welcome to the Academic Medicine Podcast series. You’ll meet medical students and residents, clinicians and educators, health care thought leaders and researchers. Some episodes will chronicle the stories of these individuals as they experience the science and the art of medicine. Other episodes will delve deeper into the issues shaping medical schools and teaching hospitals today. Listen as the conversation continues in this Academic Medicine podcast.
Toni Gallo:
Welcome to the Academic Medicine Podcast. I’m Toni Gallo, a staff editor with the journal. Joining me today to discuss the peer review process, specifically the practice of evaluating a scholarly article as a peer reviewer, are Academic Medicine Editor-in-Chief, David Sklar, and Carl Stevens, the Medical Director at Care Oregon, a Master Reviewer for the journal, and a 7-time winner of our Excellence in Reviewing Award. Today’s discussion is meant to be a resource for new and seasoned reviewers alike who want to improve their reviewing skills. While we’ll be talking about reviewing for Academic Medicine, many of the principles and recommendations we’ll discuss really apply to reviewing submissions to other journals, other publications, conference abstracts, and more. So let’s get started. Carl, we wanted to ask you, why do you review?
Carl Stevens:
I enjoy writing and so I just naturally enjoy reading the writings of others. That’s probably one primary reason. Another reason for me is the papers that I review for Academic Medicine are generally in topic areas that are of really lifelong interest for me. Most of what I look at and review relates, in one way or another, to health care policy, innovations in the delivery system, quality improvement, and to a lesser extent, some more technical education interventions in those areas. Because the topics are important to me, I find a lot of interest in looking at them.
David Sklar:
Carl, this is David Sklar, and I want to thank you for spending some time with us. Usually, what we try to do is send articles out to people that we know are more or less in their area of interest and strengths. Just for people who might be listening, we don’t send out every article that gets submitted to us. We actually do an initial review and narrow that number down to about maybe a third of the articles that get submitted, do go out to reviewers. So we’re trying as much as possible to get those articles that are the highest quality to our reviewers, so that the time that they spend is spent on articles that are most promising.
David Sklar:
Typically, what we’re trying to look for when we send out an article to one of our reviewers is an article that is a good fit and that addresses problems that we think are important and does it in a way that offers some new insights or new directions for us to improve how we think about the problem. So I’m just wondering as you receive these articles from us, whether those are some of the main areas that you’re thinking about when you’re looking at the article or what are some of the main issues that are going through your mind as you’re trying to decide “Is this the type of article that we should continue to pursue and maybe improve?” Or what kinds of thoughts are going through your mind when you’re reviewing an article? Maybe also if you can just tell us how do you read an article? Do you read it once, twice, three times? How do you go through that process?
Carl Stevens:
Yeah, I always take a quick trip through the entire article first, and I have to discipline myself not to start making marks on it, which is almost irresistible at times. That I feel like until I’ve read through the whole thing and gotten to the end, I really don’t have a good sense of where the authors are going. So I take a fast trip through the entire manuscript. I usually look pretty carefully at the references, and what I’m doing there is looking at the topic that they’re talking about and seeing if I know of important literature that’s not cited. Because one of the main interventions that I try to make when I review a paper is, consider looking at this particular reference that you haven’t listed, which addresses the topic that you’re writing about. So that’s my method, is to go through once quickly. Then on my second trip through, I try to mark each of the major areas that I know I’m going to comment on in the review that I submit. I’ll write in some marginal notes about what I want to put in relating to the sections.
Carl Stevens:
The other main questions that come up for me in looking at our paper are, has this work actually already been done and reported already? Am I aware of literature that covers the same ground so that it’s, in its current form, not that much of a new contribution? I also try to look at, as you know, the clarity of the language itself. I look for opportunities where things could be worded differently and have more impact. Then, I guess, the third factor is the … I pay a lot of attention to who in Academic Medicine‘s readership, what audience is going to find this useful, and how could the paper be changed to draw in a larger audience. Because I think sometimes that papers are very specialized and would attract a smaller subset of faculty than they could.
Carl Stevens:
So if I’m getting a paper on quality improvement and how to include a quality improvement component in a residency program, what I try to do is not have that be of interest only to people who are already converted, preaching to the choir, but write it in a way that general faculty who otherwise wouldn’t look at it would find it interesting.
David Sklar:
So Carl, this sounds like as you’re reading the paper, you’re already thinking of how to improve the paper. I know some people, when they’re doing reviews, they give themselves a little bit of time between the first reading and the second reading to just develop their thoughts about even whether or not the paper should be a reject versus a revision. Is that something you do or do you just keep going forward all the way till you get to the end of your review?
Carl Stevens:
I usually do it in one-
David Sklar:
One sitting?
Carl Stevens:
… concentrated period of time, although that may last over a couple of days to do. I may do it over a weekend. Just for the mechanics of it, I happen to use Dragon NaturallySpeaking when I’m producing the review itself, because I’ve written marginal notes in about what I want to say. What I find is that if I speak my … the text of my comments, I can put in a lot more content and a lot more nuance. When you do that, of course, Dragon puts in a bunch of the wrong words all together. So I will put in more text than I’m going to ultimately send, and then I’ll go in and pull out sections that I think are either wrong, or too much, or don’t really add that much. That’s the mechanics of how I actually produce my reviews.
David Sklar:
How about things like deciding whether or not the paper is fixable or a reject, or how do you go through that process?
Carl Stevens:
Yeah, as you know, I rarely, rarely have an out and out reject recommendation because I think… I’m a strong believer that there is no such thing as a perfect manuscript, and that every manuscript is flawed to a greater or lesser extent. If it’s really, really badly flawed, or for example, if I just simply can’t understand the points that the author is making, usually because of the way the language is written, that’s a situation in which I would say this is not really salvageable in its current form. Or sometimes I get a paper that I feel is of interest in such a narrow audience within the academic health center that it really belongs in one of the more specialized medical education journals that really are niche … focus on various niches such as methodology and some more statistical analysis type work.
Carl Stevens:
I feel like for Academic Medicine, the readership extends across the continuum from undergraduate medical education, all the way to continuing medical education, and lots of readers who really aren’t primarily educators at all. So I’m usually going to be somewhere between reconsider with major revisions, or reconsider with minor revisions, or accept more or less as is. I’m going to be in one of those 3. If I feel that the nidus of the idea, the essence of the idea is a good one, but the way the story has been put out is entirely unconvincing, then that’s going to be a major revision for me. If I think just some tuning up of the wording is all that’s needed, then that would be a minor revision for me. Does that make sense?
David Sklar:
Yeah, I think that’s good. Now, have you over … You’ve been reviewing, I know, for quite some time, and I know you’ve also won many awards for your … the qualities of your reviews. Have you noticed much change over the time that you’ve been doing reviews for Academic Medicine?
Carl Stevens:
I don’t think so. Not not in the topic areas that I review in. In a way, that’s a surprising thing because there was all the excitement around the Flexnerian centennial that, “Oh, we’re going to completely transform medical education.” Because I think the delivery system itself is in desperate need of transformation, it was pretty exciting to me that we would make some major changes in medical education. I’ve actually been surprised by how little has changed.
David Sklar:
Why do you think that is?
Carl Stevens:
Okay, so here’s my theory on why there’s been relatively little change: there’s been a massive change in technology that’s available in academic health centers. So actually the delivery of medicine in terms of imaging and therapies that are available has shifted dramatically, but the day-to-day behavior and how decisions get made has not changed that much, which is surprising, right? There’s an enormous literature on value-based purchasing and efforts on the part of the government to shift to value-based purchasing models. But we still actually live in what’s a pretty plain vanilla fee-for-service world. I think because that really hasn’t transformed, medical education has sort of stuck where it was.
David Sklar:
We’re kind of dealing with that system. Do you think the articles that we’re publishing are helping to move us to transform care delivery? Or do you think that it’s pretty tough to make those changes without reforming the payment system?
Carl Stevens:
No, I think they do. One of the reasons it’s important for me to review is that I think we need to prepare the next generation to advocate for change. I’m a very strong believer in physician-led transformation of the delivery system. Obviously, physician behavior alone will not accomplish the fix that’s needed, but I think we’re in a very strong position to lead. If you look at successful efforts around the country, like Intermountain and Partners in Boston, those systems were transformed under physician leadership. So to me, it’s critically important that more medical students and more residents get really a deep understanding of both some of the drawbacks of our current system and some ideas that are out there for improving it.
David Sklar:
Now, one question I’m a little curious about, in terms of the whole review process, is we try to send people when they review articles, the other reviewer comments. Does that help make you a better reviewer, or do you find that useful?
Carl Stevens:
No, I think it’s always, I do find it helpful. I generally find that I’ve written … my word count on my reviews tends to be higher. That’s just because I speak them, not keyboard them. But it’s very helpful to see what other reviewers have commented on. The lens that I bring to reviewing the papers that are sent to me for review is probably fairly different from typical reviewers because although I was a professor or on the faculty at the UCLA School of Medicine for 30 odd years, I have also always had some kind of a policy or industry role going on at the same time.
Carl Stevens:
I’ve had the good fortune to work with Brent James at Intermountain on the improvements that he made there and worked with people from RAND on some of the health services research they’ve done, and really kept one foot in the quality improvement, value-based care world. So when I’m reviewing, I’m not looking purely from a medical education perspective. I’m really looking more broadly from a health care delivery system perspective, and how would this educational intervention play out in that larger context of the delivery system. How could learners positively influence the delivery of care in their institutions? Does that make sense? Do you see what I’m talking about?
David Sklar:
Yeah, it does. Let me ask you another question that I’ve often thought about, is what do you do when you get a paper that uses a methodology that you’re not familiar with? The topic may be one that you know pretty well, say quality improvement or patient safety or something, but they’re using a methodology, maybe a phenomenology or some other qualitative method maybe that is not something you personally have that much experience with, or maybe there’s another quantitative methodology that they’ve used that you’re not familiar with. So how do you deal with that part of it?
Carl Stevens:
Well, I’m a pretty heavy user of PubMed. So what I will generally do is first, I’ll check and see if the methodology that’s being used is specifically referenced in the references. If it is, I’ll open that reference and read it. Let’s take qualitative research as an example. I know what that is. I’ve done some of it, but I’m by no means an expert in it. I might pull a paper specifically on that and try to get a sense of where the major steps in that methodology followed here.
Carl Stevens:
If it’s too exotic and arcane, a statistical test like that, I might call a colleague who, or go chat with a colleague who, is a professional statistician in essence. I’ve been lucky to always, and this continues to be true in the job I have at Care Oregon now, we some very expert statisticians and health policy analysts that I can turn to and say, “Help me understand what’s being done here. Does this look like it’s basically a kosher application of this, or did somebody just file this to just go package and plug it in?”
David Sklar:
Of course, we do have some statistical and other expert support at the journal for different kinds of methodology, so that is available. I think we look to our reviewers to say, if this methodology is not one you’re familiar with, that you would let us know so that we would make sure that we did send it to one of those people who could give us more of an in-depth analysis of the methodology.
Carl Stevens:
Yeah, I think it’s nice to know that you have that backup in place. The topics I review on are not generally extremely quantitatively intensive, just by virtue of their subject matter. I taught a bit of statistics at UCLA of the very most basic type. Mainly focused on looking at study design, not focused on trying to understand whether the observed effect was random or not. Because I think the biggest problem with papers nowadays that come out isn’t the statistics that were done on them, it’s really study design in the first place. So I don’t often find myself just plain stumped by the analytic methodology, but I think that may reflect the type of papers that are sent to me.
David Sklar:
Mm-hmm (affirmative), that you’re familiar with. Toni, questions you have, you’d like to ask?
Toni Gallo:
Yeah. So over your career as a reviewer, what have you found to be helpful in developing your reviewing skills? Is there some sort of resource, or working with a mentor, plain practice, anything that kind of helped you develop as a reviewer?
Carl Stevens:
I think the more you do, the better you get at it, certainly. I have always tried to write a consultative review. In other words, my mindset in addressing a paper is not to judge it acceptable or not acceptable. I try to look for ways that it would be a stronger paper really, and I’ve pretty much stuck with that over the course of the years that I’ve done reviews. Again, I have the advantage of applying some of the different lenses from my other non-academic jobs to a lot of the work that I do. I have a pretty strong understanding of how dollars flow through the delivery system and the incentives that affect physician practices.
Carl Stevens:
So often I’ll be looking at things from that lens. That’s very often the perspective from which I’m going to be looking at the paper itself, is yeah, what if students learned this or did this, and what’s going to be the broader impact as it goes out? In terms of resources, I don’t know. I myself have not really … Other than having had manuscripts reviewed by other journals, I have not really looked for mentorship. But then I started doing this at mid-career, not early on in my career. That may be a factor.
David Sklar:
So you already had a lot of experience with your own scholarship to bring to bear to the review process?
Carl Stevens:
Yeah. I’d already done quite a bit of writing and I had taught quite a lot of problem-based learning and read a lot of resident notes and participated in a ton of journal clubs. And I’d already taught interpretation of the medical literature, so I had walked many, many students through the interpretation of randomized clinical trials as an example, so I feel like that was good background for me. If you teach the reading and interpretation of the medical literature, I think that’s a great way to build up your own critical appraisal skills. Really, a review, a peer review is sort of a critical appraisal of a journal article. So I think there’s an overlap between the sort of orthodox McMaster Gordon Guyatt methodology for analyzing a clinical trial or any type of trial of a diagnostic test and doing a review of a paper. You’re looking for some of the same types of issues.
Toni Gallo:
I was just going to go back to your comment about being familiar with the process as an author and how that helped you as a reviewer. Was there anything specific that as an author you really appreciated when the reviewers told you, or alternately, something that was just not helpful?
Carl Stevens:
Yeah. On the helpful side, statements that I’ve made in a paper that are unconvincing to the reviewer and would need to be covered more thoroughly or more clearly, usually in the discussion section of the article – those are always very helpful. In other words, if a reviewer says, I really don’t either get this point that you’re trying to make here, or it’s being made in a way that is ineffective and unconvincing, that gives me something that I can then work with and improve. What is less helpful is if the author happens to be an expert in the area that I’m writing about and simply disagrees with my position, and expresses as much, that’s not particularly helpful to me. Then again, because there’s no such thing as a perfect paper, and there will always be flaws in each manuscript, comments that are too nitpicky either about language or methodology, whether it’s a disagreement on, you should’ve said X where you said Y here, I don’t find those all that helpful usually.
Carl Stevens:
So the more conceptual and higher level advice on improving an argument is much more helpful than the extremely detailed comment on a particular point I’ve made.
Toni Gallo:
I guess, just one final question to end the discussion. You’ve talked a little bit about things that have been helpful for you. But is there any advice that you would give to a junior colleague who is just starting to review? Something for them to keep in mind or just general advice that you would offer?
Carl Stevens:
I think the first piece comes back to not being excessively demanding and critical and that you want to cut the author some slack. Another piece of advice is if you’re looking at a paper that either has medical student authors, and especially if there’s a medical student lead author, then I try to be very positive and consultative in my review. The less experienced the author, the more sort of didactic approach I try to take to improving the work rather than judging it.
David Sklar:
Carl, that leads to another issue that sometimes comes up where we will get reviews, at least in some cases, where they’re rather … I don’t know if insulting is the right word, but they are personally critical sometimes of an author or their ideas. My sense is that that’s not obviously something that you would support, but maybe you might have a word or two in terms of how you … even if you find something you don’t like, how you can say it in a more supportive way.
Carl Stevens:
Yeah. I mean, I think I avoid ad hominem attacks scrupulously, if I can. I would never want to be in a position of appearing disrespectful. That’s why I end each one of my reviews by thanking the authors or complimenting the authors on the effort that they’ve put in and thanking the journal for the opportunity to review it. I always open my review with a statement on the importance of the work, even if the second sentence in the review is, “However, I don’t think it’s appropriate for publication in Academic Medicine.” So I try to start with good news and I try to end the review with good news. To the extent that I can, my review centers on the suggestions I would have for making that into a better manuscript, of interest to a broader audience, and more likely to succeed.
Carl Stevens:
I think it’s really important not to come across too critically of the work. So I do not strike that kind of tone. I don’t think it’s appropriate. I think it’s similar to the way that you have to approach residents and students in the sense that even if you really need to give some fairly firm feedback, and you need to see a change in behavior, that has to be done in a way that is not going to cause the learner to shut down immediately, right? So yeah, no, I really try to stick with a very positive tone and phrase my suggestions in such a way that they come across as in the spirit that they’re offered, as constructive criticism, not as judgment.
David Sklar:
Well, I think we sometimes forget how much work it is and how much energy people put into these papers. It’s very easy to find something that is not quite right. We’ve all been there before where we’ve had our own work reviewed critically, and it can be very painful. So I think being mindful of how that feels, and having been through it, and being respectful as you say, of the person, even if the work needs a lot of revision, I think is a great comment to leave us with. So I think … Yeah, go ahead.
Carl Stevens:
I was just going to throw in just a couple of thoughts I’d had while I was sitting here. In terms of the mentorship and maybe of a younger reviewer trying to get started, and the people that exist on a medical school campus or in an academic health center, the mentors really that have been most useful to me, I’ve been incredibly fortunate to be able to take Brent James’s health care delivery improvement course in Salt Lake City. I consider him among my most important mentors. That has nothing directly to do with reviewing but having contact with thought leaders on campus. I was fortunate to be able to take the quality of care course from Bob Brook at RAND and work with some people over there.
Carl Stevens:
Seeking out individuals that are just doing extremely high quality, innovative work in general is important mentorship. Then of course, to the extent that you can, spending time with the more prolific and more articulate writers on campus that publish a lot is another good resource.
David Sklar:
Well, that all helps you kind of get a sense as to whether a submission is really making a contribution, because you can see it in the broader context of scholarship in that area.
Carl Stevens:
Yes, and just the quality of intellectual analysis and the quality of thinking.
David Sklar:
Yeah. Well, good. Well, I think we’ve probably run out of our time here.
Carl Stevens:
Okay.
David Sklar:
So I want to thank you and Toni for putting together, I think, a great session, and I hope all of our reviewers have an opportunity to hear your comments. We’ve really appreciated your contributions to the journal, and we hope that they’ll continue. Again, we also want to thank all of our other reviewers who are out there, who aren’t on the call right now, but are contributing to us on a daily basis. So Toni, you have any final parting words?
Toni Gallo:
I don’t. Just thanks again, Carl, for all of your work as a reviewer and for joining us today.
Carl Stevens:
Okay. It’s been my pleasure talking with you.
Toni Gallo:
For more from Academic Medicine, including to access other resources for reviewers we offer, find us online at acdemicmedicine.org. Check out our blog at academicmedicineblog.org. Follow us and interact with the journal staff on Twitter at @AcadMedJournal and subscribe to this podcast through iTunes, the Apple Podcast app, and SoundCloud. While you’re there, leave us a rating and let us know how we’re doing. Thanks so much for listening.
