Lisa M. Lines, PhD, MPH; Wayne L. Anderson, PhD; Harper Gordek, MPH; and Anne E. Kenyon, MBA
We recently reviewed the paper by Chen et al titled “CMS HCC Risk Scores and Home Health Patient Experience Measures” in the October 2018 issue of The American Journal of Managed Care® and would like to comment on aspects of the methods and findings.1 We are researchers at RTI International, which is the CMS contractor responsible for conducting national implementation activities for the Home Health Care Consumer Assessment of Healthcare Providers and Systems (HHCAHPS) Survey.
Chen et al performed risk adjustment using agency-level, not person-level, CMS Hierarchical Condition Category (HCC) risk scores. In theory, for use in national implementation, we believe that an agency-level HCC risk adjustment measure presents at least 2 construct validity issues. First, HCC scores are known to be subject to upcoding, which likely contributes to their agency-level coefficient having upward bias, consequently contributing to the magnitude of their findings. Second, the random sampling method used in HHCAHPS does not consider all of an agency’s patients to be eligible for sampling. The survey inclusion criteria are that patients must have had at least 2 skilled visits in the past 2 months, be 18 years or older, have received care other than routine maternity care, not be receiving hospice care, and not be receiving care for a condition for which the state prevents the release of patient information. Moreover, the survey is not answered by all eligible patients. Because the authors used agency-level HCC risk scores that are based on the entire patient population of an agency, there were likely to be outliers at the high end that may have also contributed upward bias in their agency-level HCC score measure. Random samples such as those used in HHCAHPS are, theoretically, less affected by outliers.
We identified 2 additional methodological issues. First, the authors did not report that they tested for excessive correlation. Because Chen et al used state fixed effects and race variables together, there may have been excessive correlation because racial composition varies by state. Their inclusion of additional variables beyond those currently used in national implementation of HHCAHPS may also result in some excessive correlation for which testing results should be reported. Second, the authors controlled for agency profit/nonprofit status and the number of years that an agency had been certified by Medicare, both of which may reflect aspects of an agency’s care orientation that are rightfully reflected in the agency’s publicly reported scores.
With respect to the paper’s results, effect sizes for care experience measures can be categorized as small (difference of 1 point), medium (3 points), or large (≥5 points).2,3 The findings on agency-level HCC scores reported by Chen et al indicate small to moderate effects on HHCAHPS measures with just 1 exception (effect on willingness to recommend the agency). Further, a 2017 paper by Smith et al4 found that effects of race categories were of the same magnitude as some of the existing risk adjusters currently used that are associated with race (eg, education level, health status). The race effects reported by Chen et al were also relatively small (0.1 coefficient or smaller), suggesting that our current adjusters generally control for race effects.
RTI International staff conducted rigorous diagnosis-related risk adjustment testing as part of a mode experiment before national implementation began, using more than 100 separate diagnosis-derived condition categories underlying the CMS HCC score methodology. We found that only schizophrenia and dementia had sufficiently large, statistically significant coefficients to merit inclusion as risk adjusters. In addition, the HHCAHPS publicly reported case-mix–adjusted scores already include adjustments for self-reported general and mental health status, as well as selected sociodemographic and other adjustments. Self-reported information on health conditions has been previously shown to be reasonably reliable and accurate—sometimes more accurate than claims data.5-8
In addition to the issues identified above, an agency-level HCC measure is impractical for use in national implementation, given that the CAHPS surveys collect deidentified data. Even if we did have patient identifiers, individual-level HCC scores would not be available on a timely basis for use in regular public reporting.
Chen et al conclude that “findings indicated that current risk factors insufficiently adjust for the variation in beneficiaries’ clinical and functional conditions that affects patient experience.”1 We thank the authors for bringing their concerns forward, but given the methodological issues presented here, the relatively small size of their results, the extensive testing we conducted prior to national implementation using hundreds of claims-derived diagnoses, and the impracticality of using agency- and patient-level HCC scores, we think that the current risk adjustment strategy used in HHCAHPS, although not perfect, is effective.