Background
The numerous currently available public hospital quality rating systems frequently offer conflicting results, which may mislead stakeholders relying on the ratings to identify top-performing hospitals. Given that there is no gold standard for how a rating system should be constructed or perform and no objective way to compare the rating systems, we evaluated the strengths and weaknesses of four major public hospital quality rating systems based on our experience as physician scientists with methodological expertise in healthcare quality measurement. Objectives >>
Objectives
Compare differences in four major public hospital quality rating systems
Focus on potential for misclassifying hospital performance
Identify strengths and weaknesses of each system
Identify issues limiting all systems
Identify opportunities to advance the field of hospital quality public reporting Results >>
Results
No rating system received an “A” or an “F.” The highest grade received was a “B” by U.S. News & World Report. The Centers for Medicare & Medicaid Services’ Star Ratings received a “C.” The lowest grades were for Leapfrog, C-, and Healthgrades, D+. Each rating system had unique weaknesses that led to potential misclassification of hospital performance, ranging from inclusion of flawed measures, use of proprietary data that are not validated, and methodological decisions. More broadly, there were several issues that limited all of the rating systems we examined: Limited data and measures, lack of robust data audits, composite measure development, measuring diverse hospital types together, and lack of formal peer review of their methods. Opportunities to advance the field of hospital quality measurement include the need for better data subject to robust audits, more meaningful measures, and development of standards and robust peer review to evaluate rating system methodology. Conclusions >>
Conclusions
In this Rating the Raters initiative, we found that the current hospital quality rating systems should be used cautiously as they likely often misclassify hospital performance and may mislead stakeholders. These results can offer guidance to stakeholders attempting to select a rating system for identifying top-performing hospitals.
The Ratings
 |
 |
 |
 |
|
|---|---|---|---|---|
| Overall Grade | C | D+ | C- | B |
| Potential for Misclassification | D | D | C- | B |
| Importance/Impact | C+ | B | C+ | B |
| Scientific Acceptability | C+ | D+ | C | B |
| Iterative Improvement | C- | C | B- | B+ |
| Transparency | B | D+ | C | B |
| Usability | B | C | C+ | B |
Background
The numerous currently available public hospital quality rating systems frequently offer conflicting results, which may mislead stakeholders relying on the ratings to identify top-performing hospitals. Given that there is no gold standard for how a rating system should be constructed or perform and no objective way to compare the rating systems, we evaluated the strengths and weaknesses of four major public hospital quality rating systems based on our experience as physician scientists with methodological expertise in healthcare quality measurement.
Objectives
Compare differences in four major public hospital quality rating systems
Focus on potential for misclassifying hospital performance
Identify strengths and weaknesses of each system
Identify issues limiting all systems
Identify opportunities to advance the field of hospital quality public reporting
Results
No rating system received an “A” or an “F.” The highest grade received was a “B” by U.S. News & World Report. The Centers for Medicare & Medicaid Services’ Star Ratings received a “C.” The lowest grades were for Leapfrog, C-, and Healthgrades, D+. Each rating system had unique weaknesses that led to potential misclassification of hospital performance, ranging from inclusion of flawed measures, use of proprietary data that are not validated, and methodological decisions. More broadly, there were several issues that limited all of the rating systems we examined: Limited data and measures, lack of robust data audits, composite measure development, measuring diverse hospital types together, and lack of formal peer review of their methods. Opportunities to advance the field of hospital quality measurement include the need for better data subject to robust audits, more meaningful measures, and development of standards and robust peer review to evaluate rating system methodology.
Conclusions
In this Rating the Raters initiative, we found that the current hospital quality rating systems should be used cautiously as they likely often misclassify hospital performance and may mislead stakeholders. These results can offer guidance to stakeholders attempting to select a rating system for identifying top-performing hospitals.
Major Issues Identified across All Rating Systems
• Data and Measurement Limitations
• Considerable Use of Administrative data
• Composite Measure Development Methods
• Measuring Diverse Hospitals Together
• Failure to publish research of methodology changes with peer review
• Potential Financial Conflicts (selling emblems)
Opportunities to Advance the Field of Public Hospital Quality Ratings
• Better Data
• Better Measures
• Meaningful Audits of Data
• External Peer Review of Methods
Major Issues Identified across All Rating Systems
- Data and Measurement Limitations
- Considerable Use of Administrative data
- Composite Measure Development Methods
- Measuring Diverse Hospitals Together
- Failure to publish research of methodology changes with peer review
- Potential Financial Conflicts (selling emblems)
Opportunities to Advance the Field of Public Hospital Quality Ratings
- Better Data
- Better Measures
- Meaningful Audits of Data
- External Peer Review of Methods