Is evidence-based medicine easy?

Welcome back to Depth of Anesthesia. This is a podcast that critically ex I'm David Hao, and I'm an anesthesia resident at the Massachusetts General Hospital. A quick shout out to Anesthesiology News. They have two new podcasts called On the Case and Ask the Experts. On the Case is a show that offers a behind-the-scenes look into popular case reports published in their magazine And Ask the Experts is a show that offers insights from some of the leading experts in the field of anesthesiology, where they'll be asking questions submitted by their listeners to those experts on a variety of topics.

You can listen to both these podcasts on the Anesthesiology News Presents channel. If you'd like to support our mission here at Depth of Anesthesia, consider contributing to our Patreon. Your donation will help offset the cost of hosting and producing this material. I'll include a link in our episode description. I'm really excited today because we're going to be doing an episode that's truly unlike any depth of anesthesia episode in the past.

No claims, no cases. We're just going to be talking about a simplified approach to evidence-based medicine. Let me start by saying that I know reading primary literature can feel stressful. And starting out, it certainly felt like that for me as well. To make matters worse, we often have guests on the show who are incredibly skilled at reviewing literature, often picking up some of the most subtle details and nuances.

And I know this can make approaching literature seem even more overwhelming So part of our mission here at Depth of Anesthesia is to encourage critical thinking about our practices and learning to read and critique the literature yourself is a big part of that. So today we're going to break down a simplified six-step approach that might make the whole process a bit less daunting. Our guest today is Dr. Justin Morgenstern. As some of you may remember, Dr. Morgenstern joined us back in April of last year to discuss whether COVID-19 is spread by aerosols and aerosol generating procedures. He's a community emergency medicine physician and creator of First 10EM, an exceptional free and open access medical education project.

Dr. Morgan Stern, welcome back to the show. Yeah, it's such a pleasure to be back, especially to talk about uh some some of my favorite nerdy topics and especially my contention that evidence-based medicine at its base is actually really, really easy. I'm so glad to have you back on our show. Can you start us off by just telling us a little bit about your own journey with evidence-based medicine? Yeah, well I have to admit, when I was back in medical school, I really didn't care about EBM at all.

You know, I found the way that it was taught just honestly boring. It didn't seem to matter to the patients that I was seeing. So in medical school, I just went about memorizing every guideline that I possibly could. And that's how I practiced in residency. I followed the guidelines to a T, and I sort of thought that I was practicing perfect medicine Uh and then I happened to find a few podcasts that actually went through the evidence uh in a way that showed how it mattered to patients.

Uh and they showed the studies and more importantly pointed out their flaws and it turned out that many of the things that I was doing were not just useless, but actually had been previously proven to be harmful to patients. So they they were known to be harmful based on evidence, but they were still in the guidelines. And so my word was actually a little bit shattered at that at that point I honestly now have a little bit of a hard time trusting guidelines. And unfortunately, I think the more that anyone digs into evidence, the more they'll find that there are a large number of our guidelines that actually still suggest things that Are actually harmful to patients.

And so that's why I've ended up spending so much of my time reading studies and trying to summarize them in a way that I think is a little bit more transparent and hopefully more understandable than most of the guidelines we still have available. I certainly feel like I've had some of the same experiences as you with evidence-based medicine. When it comes to guidelines specifically, I've always wondered who came up with this and how do they arrive at these conclusions. As depth of anesthesia listeners can certainly attest to, when you actually dive into the evidence, things are not so black and white anymore. So I'm looking forward to learning more about your approach because I think it's a great framework to not only start building comfort with reading literature, but actually learning how to critique it.

This is something I certainly wish I had when I first started. So I'm glad that you're here to share this with our listeners. Starting now at the beginning, how do you even find a paper to read? This sounds like it should be the easiest part, but this can actually get a little bit complex because it depends a lot on what you're trying to accomplish here. Uh, if you want to do a really exhaustive search, like what they do in systematic reviews, you're probably gonna need a medical librarian, because honestly, that involves looking at thousands of papers just to find the you know eight or ten that are relevant.

So for most practicing clinicians that's just going to be too much, but there are still lots of reasons that we can do less thorough searches. The the most common reason would just be that you have a specific question, you know, that like does ketamine cause hypotension in trauma RSI And my my first piece of advice would actually be to check if anybody else has already done the search. You know, if there's a big systematic review or a Cochrane review on the topic. It's worth finding that because even though I'm going to want to read the papers for myself rather than just trusting the conclusions of the systematic review, there's a pretty good chance that they will have already found all of the most relevant papers.

If nobody's already done this review, the honest truth is it's just gonna take some time. I do I use a combination of Google Scholar, PubMed, the Trip database to pull out any articles that seem relevant. But you know, this can't be done on shift in just a few minutes. It's gonna take some some time. So that's the reason number one to find a paper would just be you have a burning question.

Another reason is you know you just want to keep up with the latest literature. And again, I think, you know, there's so much being produced, it's really hard for the average person to keep up, to do all of this. And just like off in option number one, I think your best answer is actually letting somebody else do the work for you So in emergency medicine, we have a fairly large number of podcasts and and blogs. I've throughout my career used something called emergency medicine abstracts, and I just let them do a lot of the work for me. They comb through all the literature and I get to hear about it every single month.

And that's Brilliant. So you can use Journal Watch or InfoPoems or whatever it is in your specialty. Find a service that does most of the work for you. And then in addition to that, it's probably a good idea for everybody to pick, you know, the two or three biggest journals in your area of uh uh specialty and you know get the abstracts emailed to you you're not going to read every single paper but you want to see what is being published uh in that area There's one other topic that's worth covering, and that's if you have a niche. If there's a one specific topic that you really care about.

So for example, if you do like PE research or you're an ECMO specialist. you probably not want to know about all the papers that are being published in that specific area. And so in that case, you really want to know about something called PubMed alerts. They're basically the same as Google Alerts, which are also pretty useful. And basically you have to create an account on PubMed. It's free.

And then you tell the system that basically any time a new paper gets published on that area, just to send you an email. And you can bunch the email so they only come in once a week or once a month. But that's a great way to stay on top of a specific topic. Maybe I'll I'll mention one last reason why you might want to find a paper. It's getting long-winded, but you might want to just find some papers for practice. this to get more comfortable with critical appra appraisal.

And I think that's a really, really good idea. Uh but the best way to go about this is to find papers that have already been critically appraised by some other experts. So find a blog or a podcast or a journal that has already reviewed a paper. Download that paper and read it for yourself and then go and listen to the podcast and see whether the experts agree with your opinions about the paper. So that was a really long-winded answer for you.

And it winds up being the classic EBM answer, right? It depends. How you go about finding papers is really going to depend on what you want to use them for. When I go through the process of researching a depth of anesthesia episode, I go through a very similar process of first looking to see if anyone has done the legwork already to answer the question I have in mind, basically letting someone else do some of the work If not, then I'll have to certainly do the literature review myself, and that takes a lot more effort. Now, to stay up to date, I have email alerts set up for a few high-impact journals in our field.

For the journal anesthesiology, for example, I'm subscribed to the featured articles and online first alerts. I'll then skim my email notifications and save the email if I see something that I want to read later I think the PubMed Alert strategy that you mentioned is also very valuable because it alerts you to articles by topic as opposed to publisher. We'll send out an article and maybe a short video tutorial on how to set up a PubMed email alert. Okay, so now that I'm getting emails possibly with hundreds of publications, most of which are probably still about COVID, how do I now decide if the paper is worth reading? Yeah, this takes a little bit of work to call through all of the papers that you see.

And this is really the only time where you would ever want to read the abstract. Most of the time the abstract is way too short to be valuable, but this is it's there to help you cull through these papers. And I actually just jump right to the conclusion of the abstract. Because if the conclusion is boring or just not helpful, then I can throw the paper out and not waste any more time. If the conclusion is interesting or relevant to what I'm looking for, then I'll glance at the methods section of the abstract.

And again, if the methods are clearly awful, like you know, this is an uncontrolled rat study, again, I just throw the paper in the garbage and I don't waste any more of my time. And then it's only if the conclusion is interesting and the methods seem reasonable, then I'll download the paper and I can read it when I find time I think this step is probably one of the most important because there's so much literature out there and you want to spend your time reading and ultimately critiquing papers that might actually change your practice. So now that we've picked the paper we want to read, is there an efficient and strategic manner to approach these inevitably lengthy documents?

Yeah, I think this is so important. And I was never taught this myself. When I started reading papers, I would just read them like a book. You pick it up, you start at the front, you end at the back, and it would take forever. But not all the sections of a paper are equally valuable.

So if you just focus on the good stuff, you can read a paper without missing any of the important details in like five or maybe 10 minutes. So let's just quickly run through the sections that usually make up a paper and what they're good for. So right at the top you're going to find the title and the abstract. We already said those are good useful maybe when you're searching for papers, but now that you've actually sat down to read a paper, you can skip them. The introduction.

You know, if you know absolutely nothing about a topic, you might want to read the intro, but 95% of the time I just skip it entirely. Because the background info in the intro is not the result of a systematic review. It's essentially just the author's opinion on a topic, and there's a lot of room for bias there And we're not sitting down to read a scientific paper to hear about people's opinions. We want the science. We want to form our own opinions.

So I just skipped the intro. The method section is probably the most daunting part of the paper, but it is by far the most important. It's literally where all of the science lives. The results are meaningless unless you have a good methods section The methods section might be the most confusing section because it does contain often a lot of esoteric language or jargon, but a simplified approach is possible. And I think we'll get back to that in a minute.

But I always start by reading the methods section, because again, you can save yourself a lot of time. If the methods are garbage, stop reading now. Just throw the paper out because no matter what you find in the results section, if there's poor methods, the paper should not change your practice. Then the results section, that's the second most important part of a paper after the methods section. It's literally why you picked up the paper in the first place, right?

The only problem with the results section most of the time is that there's often a ton of different results listed. And it can be a little bit overwhelming sometimes. So I tend to try to just focus in on the one Primary outcome. And then maybe one other outcome if you don't think that the primary outcome was actually the most important outcome Now, I actually find that most of the value doesn't come from reading the text, but actually looking at the graphs and tables. The text contains what the authors want to draw your attention to.

But again, that's another potential source of bias. I just tend to look through the tables and graphs primarily to see what numbers really stand out at me. Are there big differences between the groups? The next section down is the discussion section. This is another non-systematic review of the literature.

The authors will generally compare their results to prior studies where they might talk about potential weaknesses with their studies, but a lot like the introduction. This section represents the opinions of the authors. That can be really interesting sometimes, but it can also be biased. It's no longer pure science, so usually I end up skipping the discussion section. And then the conclusion is the author's opinion about what their results show.

But it did by this point, you should have already read the methods and the results, so you should have already formed your own conclusions about the papers. So you don't need the author's conclusions. So although most of these papers seem really long, they're like 20-page PDFs, it's really only the results and the method sections that really matter. So you can throw the rest, and now you're down to two or three pages, and this is super easy. So it sounds like despite the tendency of these papers to really seem overwhelmingly long, we can approach them strategically by focusing on the important sections and understanding that the actual objective aspects tend to live in the methods and results.

As you pointed out in your blog post, Sketchy EBM outlines a very similar approach to reading paper, so I'll be sure to link our listeners to that video as a quick review. Yeah, I think again, being really explicit about what each section provides you is really important. And don't get me wrong here, I have learned a ton from intro sections and discussion sections in my life. The people writing these are very, very smart. But I think you want to divide it out.

If your goal is to judge the science for yourself, to be as objective as possible, stick to the science section Methods and results. If you want, just want to learn something, if you want to hear the opinions of world experts, but then by all means, read the intro and read the discussion. Just remember that you're reading opinions and not necessarily science in those sections. Perfect. So we've made it now through the first three steps.

Step one, how do I find a paper to read? Step two, is this paper worth reading? And step three, actually reading the paper. I think the next step is probably where people often feel like they need additional training or expertise. I've often thought this myself as our guests here on Depth of Anesthesia often have a fantastic grasp of statistics.

And are able to point out some of the nuances that I completely missed. Do you have any special training or background in statistics or epidemiology? So no, I I don't have any special training at all. And so there are often, you know, stats concepts that I either don't understand or that I have to look up. For the average clinician, you're right, the stats are the sort of overwhelming part of evidence-based medicine, but honestly, I'll tell you, I think they are by far the least important part of EBM In order for stats to really matter, everything else needs to be perfect already.

The analogy I like to use is a race. So imagine two people are running a race. One per person is running straight uphill with 50 pounds on his back. The other person is running downhill and uses steroids. Getting obsessed with stats is like trying to measure the results of that race down to the thousandth of a second Measuring microseconds is silly if the race wasn't fair.

And I think honestly in medicine we do this too often. We spend a lot of time arguing about p-values, but that's a lot like staring at the stopwatch when the race wasn't unfair The stats are completely irrelevant when the race isn't fair. So I think as clinicians, our job when reading a paper is to consider. Was this race fair? And that can mostly be done with common sense.

I think most people are really good at that. And then we can leave the stats to the statisticians I think maybe one thing we really want to highlight is that when defining evidence-based medicine, David Sackett made sure to point out that literature is only one part of the picture. He said there were three essential components: there was the literature. But there was also clinical judgment and the values of our patients. So even if you feel a little bit weak in stats, every doctor is already an expert in two of those three domains.

And that's why I think it's so important that doctors read the literature for themselves, because our clinical judgment and our interactions with patients bring something to evidence-based medicine that a statistician never can. I think that's an important observation. As clinicians, we have many of the tools we need to engage in evidence-based medicine, and statistics is often just one component of the big picture. Now, when it comes to actually critically appraising the literature, do you have a framework that you like to use? Yeah, so like anything in life, the more you do critical appraisal, the better that you'll get at it.

But when you're beginning out, having a structured approach or relying on checklists can really help. So let's just run through one tool that I really like that I think is a pretty useful guide to get you through reading a paper. And again, this is nothing about stats. This is all about trying to make sure the race was fair. So the tool I use is the Rambo mnemonic.

And Rambo, spelled with two M's, stands for recruitment. allocation, maintenance, and then measurement, was it blind or objective? Now you're never gonna memorize all that, but you can pull it out while reading a paper. And basically it's just a reminder of the various aspects of a trial that we really want to think about going. through the trial. So recruitment is all about who got into that trial. You want to paint a pretty decent picture of who was included in a trial for two major reasons.

First, You really should only be applying the results of a study to similar patients to those who were included. And second, you want to make sure that the population seems fair You know, if somebody was studying a new pregnancy test and they only studied it in males, that wouldn't make a lot of sense. Now that's a ridiculous example, but there are a lot of subtle things that we could think about Say we were looking at whether vitamin C improves survival in sepsis. Well, if you only enrolled sepsis patients with normal vital signs and negative lactates, the survival might look amazing. But the study isn't necessarily looking at true sepsis patients.

So the key questions here are, do the study patients look like my patients? Is the study size appropriate? And hopefully that should be pretty easy. The researchers should just tell you their sample size calculation. And then were there any important exclusions that could really affect the results?

The next thing is A allocation, and it's very similar to recruitment. Again, we're looking at who the study participants were, but now instead of looking at the overall population. We're trying to look at how they were broken up into groups because that's really the basis of all of our trials. We break patients up into groups that are different in a way that we know about. And so then the key questions here are, were the groups similar at the beginning of the trial?

You know, if for example you were doing a stroke study, you really wouldn't want the group getting placebo to have significantly worse NIH stroke scores than the group who got TPA. Which for any astute members of our audience they might know is an exact description of the NIN study. Next, you would want to know if the assignment to the groups was randomized. If assignment isn't randomized, it's worth considering what factors might have made the groups different. These are called confounders, but honestly, it's almost impossible to identify all confounders.

And then yeah, I I like to get a little bit more nerdy. I think about something called allocation concealment. Basically, you don't want the treating doctors to know what group the patient's going to be in, because it might change how they act, and that could potentially screw up the trial. So we had recruitment, who got into the trial, allocation, what groups were they put into? The next step up is maintenance.

And this is really just about asking now that we have multiple groups that we think are fair, Were the groups treated the same throughout the trial, aside from obviously the intervention of interest? You know, we can't have one group running uphill while the other group is running downhill. And then the worst part of the Rambo mnemonic is that there is a double M. So next we come to measurement. And this is just a reminder we want to think about how the outcomes were actually measured.

And then the BO comes in. We either want our measurements to be blinded or objective, or ideally both. So were the patients, clinicians, and researchers all blinded to the treatment? Because bias is much more likely when people are aware what groups patients were assigned to, or at least were the outcomes objective and standardized Because in an unblinded trial, biases going to be a lot less likely if we have an objective outcome like mortality than it is with a subjective outcome like satisfaction with treatment. And we also want to know whether the outcomes were measured for all or at least most of the patients in the trial.

You know, were there people who lost to follow-up? And then finally, I like to add, did they measure harms adequately? Because unfortunately in medicine, we often get so focused on benefits that we forget that our treatments can also cause harms. So Although there's still a lot to think about there, there's no shortcut, I think you can see that critically appraising papers in this framework is not about talking about statistics. These simplified Rambo questions help me distill that methods section down into some what I think are pretty common sense questions that I can understand.

And then the key is really just stepping back and trying to decide, do you think this trial is fair? Got it. So the Rambo approach is really a strategy to break down the method section to common sense questions. That then help you to better assess the validity of the results. And as you mentioned, it does seem like very little of the critical appraisal through this approach is statistics-oriented.

Now are there any other questions you ask yourself to help place a trial in its appropriate context? Yeah, so the Rambo framework Is great as a way to think about the methods of a single trial. But one trial is essentially never going to change practice because science is all about the accumulation of evidence. So when I'm done reading a paper, I like to pause and think about a few questions to put the trial into its overall context. Maybe the most important question overall is just why was this study done?

You know, is somebody trying to sell me something? Was the trial actually looking at a really important question? People will have spent a lot of time on the trial. The question is, why? I also like to pause and think, you know, is the benefit big enough?

Because too often our trials don't actually consider harms. They don't consider opportunity costs. They don't consider overall costs to society. And you know, or the fact you know that investing in one area of medicine often takes time or money away from another area of medicine. So I think thinking explicitly about net benefit and whether it's worth the cost and harms is really important.

And then finally, it's pretty important to know how does this one trial fit with the existing research? You might not know off the bat, uh, you might have to go looking for other research. And actually, this is one of the things that I find most exciting about EBM. Somebody publishes a paper and it surprises you. And then you end up going and looking for past studies and it turns out that it's something that we knew all along for decades.

An example from emergency medicine in the last year. There was an RCT last year that showed that you can manage most pneumothoraces, even really large ones, without a chest tube, just conservative management. Well, that surprised a lot of people, but it turns out if you go and search through the literature, there are literally decades of observational studies saying that these people do fine if you just leave them alone. I find that fascinating. So again, I think the answer to these questions are way more important than stats.

And that's really good news because the average clinician can answer these questions, even if you don't know what like an I squared test is. I want to highlight a point that you just made, which is that we need to place any single trial in the appropriate context. Oftentimes there's a historical arc and trials are created to answer questions raised by other trials. Putting that all together, I think, is a big part of it all. You've definitely set the stage for an approach to basic critical appraisal of literature that I think is manageable for any practicing clinician.

With more practice and familiarity, I do think this process becomes easier. Now, what if the paper is particularly complex or if you're tackling a more important question? Are there other tools that you use to then help guide your appraisal? Yeah, I think even with lots of practice, people who have tons of experience, some research can get really complex. And You know, actually I think this is a lot like clinical practice.

Most patients I can manage just based on the knowledge that's in my head. But the more complex the patient, or the more critical the action for a patient, the more important a checklist becomes. So if you're just reading a paper for fun, you don't need a checklist. But if you're doing a Cochrane review, it's essential that you don't miss anything, so use a checklist for sure. You just have to decide how important this question is or how complex the paper is, whether you're going to pull out one of these validated checklists.

And there are a ton of really good resources out there, and we could probably just put some links in the show notes for people. I think that's perfect. And I know you've done a lot of work in terms of aggregating some of these resources on your website first 10 EM. So I'll include some of the links to those resources as well as to your website. Now, I personally like the next step, step five, because it's something that I continue to do in almost every domain of clinical practice.

What's step five? Yeah, maybe we're gonna get a little philosophical here and move beyond evidence-based medicine. You know, one of the best ways to grow in life is just to admit that you don't know and to ask for help. And that so I include that as part of my EBM strategy. My experience is that the EBM community is incredibly open to helping.

Maybe it's just because we're all so nerdy and we just need a little bit of human contact in our lives, but when I was learning this stuff. I would send emails out to what I consider giants in the field, people who I thought were way out of my league, and they essentially always responded. They were incredibly kind. They were really helpful. So maybe the best advice here is if you're stuck on some EBM point or any part of life at all, just don't be afraid to ask for help.

Yeah, and that's something I've learned through the work we do here at Depth of Anesthesia. People are willing to share what they know if you just ask. Dr. Morgenstern, we've made it to the last step, and this is probably the most complex step. Certainly something we could go on and on about. Yeah, so the last step in my post is about applying the research to the patient in front of us.

This is in s in one sense the most complex thing that we're going to do, but in another sense, as clinicians, this is the most natural thing we're going to do. I expect some listeners might have to go back and listen to the Rambo technique again, but everybody who's listening is already eminently familiar with the process of trot taking complex research and applying it to the research in front of them. We may not all be stats or methods experts, but I guarantee you already All of us are evidence-based medicine experts. And the reason I say that is because evidence-based medicine is not just about stats or methods. It's not just about science.

I've already said this, but this is so important. EBM, as it was defined by one of its originators, David Sackett, has three major components. Evidence-based medicine is the integration of the best available research evidence with clinical expertise and patient values. Evidence-based medicine requires clinical expertise. Clinicians have that.

And we also know how to talk to our patients about their values. So even if you don't know stats at all, And I've already said stats is only a tiny part of understanding the literature, but still, even if you don't understand stats, you are already an absolute expert in two-thirds of EBM. Now that's not to say that this is easy. But we all do need to spend some time to learn how to take the results of these studies and to integrate it with the expertise that we already have And then to learn to talk to our patients in language that they can understand about the results of these trials so that together we can make the best possible decision for that patient in front of us.

But I think ultimately, this is why I think it's better for practicing clinicians to be reading the literature than even expert methodologists. Although a statistician will have incredible insights into the mathematics and methodology of a paper, it's really only the practicing clinician who can adequately filter the information through their own clinical expertise. And then explain it in simple terms to their patients, and then use that to make decisions that mesh the best available evidence with the values of the patient. And to me, that is evidence-based medicine Yeah, these discussions are complex, but we have them every shift. In comparison to this, reading the literature is actually really, really simple, which is why I suggest everybody gives it a try I just spent the whole time nodding because you've hit so many key points and I hope you've empowered some of our listeners to feel like they can tackle the literature as well.

At depth of anesthesia, we know that clinical dogma is everywhere. And one of our core tenets is that we believe that clinicians should know what is supported or refuted by evidence. I can't count the number of times that I found literature that was either contradictory or outright misquoted. And I think that speaks to the importance of reading literature for yourself. It can definitely be a daunting task, but I think your framework, Dr. Morgensen, really sets the stage for anyone to give it a shot.

Thank you so much for joining me on the show today and really talking about a topic that I think will provide a lot of value for depth of anesthesia listeners. Yeah, this was a ton of fun for me. Yeah, absolutely. I just hope your listeners found some value in my endless ramblings today, but I'm happy to come back anytime and chat about these topics more. To our listeners, thank you so much for joining the show today and for your continued support for our work here.

I'm excited because we have a ton of great episodes in production, but we're always looking for more ideas. If you want to work together on a topic, shoot me an email at depth of anesthesia at gmail. com. Don't forget to leave a five-star rating on Apple Podcasts. to help other listeners find our work. We'll see you next time. Stay hungry and keep asking questions.