Quality Metrics – Don’t Make Assumptions Based on Headlines
I recently came across a study dealing with “evidence-based medicine” in primary care (Ebell MH, Sokol R, Lee A, et al. How good is the evidence to support primary care practice? BMJ Evidence-Based Medicine 2017; 22:88-92). A quick scan of the article indicated the authors reviewed recommendations for treatment in a popular, primary care-oriented reference frequently consulted by physicians (i.e., Essential Evidence). Having worked in health care administration and health policy for most of the past 30 years, I am familiar with Essential Evidence and its widespread use, so naturally I wanted to learn more.
The study looked at the quality of the evidence used in Essential Evidence to develop its 3,251 “bottom line” recommendations for physicians. Each bottom-line recommendation was classified into one of three categories based on the quality of the evidence used in making the bottom-line recommendation: (A) Consistent and good quality patient-oriented evidence, (B) inconsistent or limited quality patient-oriented evidence or (C) consensus, usual practice, opinion, disease-oriented evidence, or case series. The study found that only 18% of the 3,251 bottom line recommendations were based evidence in the A category, and 51% were based on either A or B categories of evidence combined. The percentage is a bit better (26% in level A and 65% for levels A or B combined) when you consider only the treatment recommendations made by Essential Evidence. However, the percentages are much lower (4% and 7%) for the diagnosis and prognosis recommendations in the highest category, respectively.
To be clear, the study was a bit dated (2017), and frankly its conclusions were not all that revolutionary. Afterall, it’s not surprising that there is more complete literature for treatment recommendations than diagnosis and prognosis recommendations. The study caught my attention because it popped up on social media, and the way it was portrayed in the post is an excellent cautionary tale about understanding the motivations of those posting information are often marketing their business under the guise of expert opinion. The misuse of this study feeds into a pretty consistent theme we have seen with the fear tactics related to quality.
So, what does all of this mean, and maybe more importantly, what does it NOT mean? For starters, of course the nexus between evidence-based medicine and quality of care is well established, and the study builds on this assumption. It is reassuring that experts evaluate evidence used in making care recommendations to physicians, even if this evidence still needs improvement. But the biggest take away from this study is that even the best physicians in a primary care setting have limited, high-quality evidence upon which they can rely in caring for patients. This reinforces the notion that despite all of the efforts of research and study, much of medicine is still an art form which requires strong relationships between patients and providers, especially in primary care.
Nobody should take the findings of this study to conclude a majority of the care delivered by primary care providers is below average quality. This study clearly looked only at the bottom-line recommendations in Essential Evidence and the evidence used to come up with those recommendations. The study did not look at a single episode of care or a single patient outcome, and as a result, any suggestion that care quality is somehow poor in most cases is simply a bridge too far. The frustration of this misguided narrative is compounded even more with the irony of the fact that most of the quality reports and scorecards typically rely almost exclusively on category C evidence (i.e., payor claims data) which is the lowest category in this study.
I’ve written in previous issues of Industry Perspectives (2023) about the shortcomings and challenges in using provider report cards and star rankings to direct members toward specific providers. As I noted back in April, scorecards and quality metrics should be part of the overall picture, but payors should be leery of services that direct care based on a scorecard or similar metrics exclusively. Payors need to consider lots of factors such as local reputation, facilities, patient experiences and yes, overall costs when its recommendations to members. It’s the difference between health care quality in theory and in practice.
This study shows that while the reliance on evidence-based medicine is growing in our system (and that’s a good thing), there is still much to be done to improve the evidence. That’s why it’s important to have partners that see the whole picture rather than just reading the headlines.