Usefulness Questioned for Transplant Center Report Cards

Public report cards for heart transplant centers are under fire for not being meaningful — potentially even misrepresentative of care quality — as a given center’s ratings may vary wildly from year to year, a study found.

Since 2016, the Scientific Registry of Transplant Recipients (SRTR) has scored centers by their performance, on post-transplantation graft failure (GF) and other metrics, in five tiers ranging from “worse than expected” to “better than expected.”

From analyzing years of these public performance ratings, investigators showed that around 39%-52% of adult centers and 28%-40% of pediatric centers had a new rating for GF compared with the previous report card 6 months prior. There was a median of four GF rating changes per adult center and three per children’s center from 2017 to 2021, according to researchers led by Shahnawaz Amdani, MD, of Cleveland Clinic Children’s Hospital, reporting in JACC: Heart Failure.

Across adult and pediatric heart transplant centers, the most volatile rating was waitlist survival (WS). Only the faster transplant (FT) metric for adults did not show poor consistency from one biannual report card to the next.

“The current 5-tier reporting of transplant center performance is highly volatile and has poor reliability and consistency. Given the unintended and significant negative consequences these reports can have, critical revision of these ratings is warranted,” the study authors urged.

The SRTR uses program-specific data on transplant centers nationwide for their public reporting. The subsequent ratings are used by CMS to identify underperforming centers needing course correction, for public and private payers to identify sites that may lose funding, and for patients and families deciding where to seek care.

SRTR reports had a three-tier rating system until December 2016, when the five-tier system was implemented for greater granularity. This may have resulted in too-narrow margins of differences between the five tier ratings, such that any small differences are magnified — the difference in actual GF rates at 1 year after transplantation between tiers being only 1.5%, according to Amdani and colleagues.

“This would mean that a difference of 1-2 graft losses over a 2.5-year period may change a transplant center’s ratings. Such minor differences in GF rates, may in most cases have very little to do with the center’s performance and more likely to have occurred from myriad other reasons,” they wrote.

“While the rationale for such public reporting is to promote transparency, accountability, and best practices among surgical programs and allow patients to make informed decisions as to where to pursue care, such measures have unintended negative consequences,” Amdani’s group warned. “Concerns have been raised as to the validity of such reports that often do not adequately risk-adjust for patient complexity and have the potential to limit access to care for sicker patients.”

Already, many transplant professionals have expressed dissatisfaction with the current SRTR report system. One survey had fewer than 14% report any positive effects of the five-tier system.

“To improve the utility of SRTR metrics, stakeholders from the transplant community must clearly define what it is we want to see in these metrics and how we can best go about measuring them in the face of variability in outcomes that is unrelated to center quality,” wrote Josef Stehlik, MD, MPH, of University of Utah School of Medicine in Salt Lake City, and two colleagues in an accompanying editorial.

“In reality, quality metrics are only as useful as their ability to reveal real underlying attributes related to a program’s processes and outcomes and their ability to differentiate programs accurately along a gradient of quality that is meaningful and comprehensible to patients. The data should ideally also provide practical information that underperforming programs can act on to improve,” they added.

Amdani and colleagues suggested switching to “ratings that have a more robust association with prospective candidate outcomes and is more aligned with care delivery and less susceptible to random events.” They also advocated for longer-term ratings and assessment of survival after listing.

For the purposes of the SRTR scores, GF currently covers 2.5 years of transplants, WS 2 years, and FT just 1 year.

Heart transplant centers are already bracing for change, as the United Network for Organ Sharing and Organ Procurement and Transplantation Network are preparing to move donor heart allocation to a continuous distribution system in the near future. Each transplant candidate is expected to have a medical urgency score that factors largely in determining listing priority.

Amdani’s team had accessed SRTR report cards for 112 adult and 55 pediatric centers for the present study.

Approximately 80% of centers were rated between 2 and 4 on every metric. Nearly all centers had at least one change in rating in at least one of the tiers, and this change usually took 12-18 months on average. A two-unit change in SRTR scores was observed for 42% of pediatric and 37% of adult centers.

The researchers said they were open to the possibility that these unstable SRTR ratings reflect real changes in care quality over time.

“Of course, programs do experience real changes in quality over time, but these changes are likely to be rather gradual in most programs,” Stehlik and colleagues commented.

Please enable JavaScript to view the

comments powered by Disqus.