U.S. News & World Report recently issued its 2023-24 Best Hospitals ratings. The ratings come at a time of increased scrutiny and skepticism of ranking systems. In very public ways, many colleges, law schools, and medical schools have declined to participate in the U.S. News process, while some hospitals have expressed their desire to abandon the rankings as well. These stakeholders’ concerns center on the purpose of these ranking systems and whether their methodologies serve the best interests of consumers.
That second point is particularly crucial when it comes to medicine. Now is the time for health care ratings to offer patients transparent and useful measures of quality that enable informed choices.
We have previously written about how flaws in data and methods for quality measures lead to biases for high volume and high transfer hospitals. Notably, like many others that seek to evaluate health care, U.S. News uses publicly available Medicare fee-for-service claims that lag the present by 18 months or more. It does not include more robust and personalized sources of information such as data in electronic records for clinical services provided, or more detailed data like patient-reported symptoms and outcomes.
U.S. News has modified its approach over time. For instance, it now includes more measurement of outpatient care and less focus on reputation. But it does not go far enough to help patients understand where to find the best care. Some of the changes have resulted in less transparency and unaccountable shifts in scoring for many hospitals and specialties. The net effect is a system that is confusing for providers, health systems, and patients.
One change — the inclusion of emergency department transfers to hospitals — is especially detrimental to validity and transparency. In the past, patients transferred from emergency departments to tertiary care hospitals were excluded. Their inclusion now has caused broad changes in hospital mortality rates. We have observed a bias in this method: Hospitals with large volumes of high-risk transfers may see disproportionately higher mortality rates than hospitals with lower transfer volumes. In modeling we have conducted, high-volume transfer centers for neurosurgery and orthopedics specialty care were most adversely affected.
An illustrative example is stroke care. High-performing stroke centers invest in their communities through outreach and networks to community hospitals. Creating streamlined transfer programs from community hospitals can enable the best outcomes and allow seamless transfers of the sickest of patients. Excluded in prior years, these cases are now included without additional risk adjustment, and effectively penalize hospitals for early transfers, even though the practice is better for patients.
The changes also emphasize a larger issue: relevance. We are moving to a time of greater consumer awareness of health care options. Studies have shown patients do not use the U.S. News rankings (or other systems) to select care and that systems can be confusing. It is our experience that it is largely health systems, marketing departments, and hospital boards that attach importance to these ratings. When hospital or specialty ratings can fluctuate wildly from year to year based on methodologic changes rather than true improvement, or when outcome metrics are based on partially representative samples of the overall population, how can we expect consumers to make sense of the ratings? While U.S. News publishes their methodology, the average consumer (and health care worker) is left confused by both the opaqueness and irrelevance of some of the factors being measured as a surrogate for quality.
To be useful to patients and providers, rating systems must move past monolithic systems that only focus on hospital care and use older and limited data. Patients and providers deserve quality rating systems that are easy to understand, use contemporary data, and measure outcomes that will drive real improvement and that will recognize the great clinical work that is delivered every day to the diverse communities they serve.
Brian Stein, M.D., M.S., is chief quality officer at RUSH University Medical Center. Bala Hota, M.D., MPH, is senior vice president and chief informatics officer at Tendo.