Lisa M. Lines, PhD, MPH

We recently reviewed the paper by Chen et al titled “CMS HCC Risk Scores and Home Health Patient Experience Measures” in the October 2018 issue of 

 and would like to comment on aspects of the methods and findings.

 We are researchers at RTI International, which is the CMS contractor responsible for conducting national implementation activities for the Home Health Care Consumer Assessment of Healthcare Providers and Systems (HHCAHPS) Survey.

Chen et al performed risk adjustment using agency-level, not person-level, CMS Hierarchical Condition Category (HCC) risk scores. In theory, for use in national implementation, we believe that an agency-level HCC risk adjustment measure presents at least 2 construct validity issues. First, HCC scores are known to be subject to upcoding, which likely contributes to their agency-level coefficient having upward bias, consequently contributing to the magnitude of their findings. Second, the random sampling method used in HHCAHPS does not consider all of an agency’s patients to be eligible for sampling. The survey inclusion criteria are that patients must have had at least 2 skilled visits in the past 2 months, be 18 years or older, have received care other than routine maternity care, not be receiving hospice care, and not be receiving care for a condition for which the state prevents the release of patient information. Moreover, the survey is not answered by all eligible patients. Because the authors used agency-level HCC risk scores that are based on the entire patient population of an agency, there were likely to be outliers at the high end that may have also contributed upward bias in their agency-level HCC score measure. Random samples such as those used in HHCAHPS are, theoretically, less affected by outliers.

We identified 2 additional methodological issues. First, the authors did not report that they tested for excessive correlation. Because Chen et al used state fixed effects and race variables together, there may have been excessive correlation because racial composition varies by state. Their inclusion of additional variables beyond those currently used in national implementation of HHCAHPS may also result in some excessive correlation for which testing results should be reported. Second, the authors controlled for agency profit/nonprofit status and the number of years that an agency had been certified by Medicare, both of which may reflect aspects of an agency’s care orientation that are rightfully reflected in the agency’s publicly reported scores.

With respect to the paper’s results, effect sizes for care experience measures can be categorized as small (difference of 1 point), medium (3 points), or large (≥5 points).

 The findings on agency-level HCC scores reported by Chen et al indicate small to moderate effects on HHCAHPS measures with just 1 exception (effect on willingness to recommend the agency). Further, a 2017 paper by Smith et al

 found that effects of race categories were of the same magnitude as some of the existing risk adjusters currently used that are associated with race (eg, education level, health status). The race effects reported by Chen et al were also relatively small (0.1 coefficient or smaller), suggesting that our current adjusters generally control for race effects.

RTI International staff conducted rigorous diagnosis-related risk adjustment testing as part of a mode experiment before national implementation began, using more than 100 separate diagnosis-derived condition categories underlying the CMS HCC score methodology. We found that only schizophrenia and dementia had sufficiently large, statistically significant coefficients to merit inclusion as risk adjusters. In addition, the HHCAHPS publicly reported case-mix—adjusted scores already include adjustments for self-reported general and mental health status, as well as selected sociodemographic and other adjustments. Self-reported information on health conditions has been previously shown to be reasonably reliable and accurate—sometimes more accurate than claims data.

In addition to the issues identified above, an agency-level HCC measure is impractical for use in national implementation, given that the CAHPS surveys collect deidentified data. Even if we did have patient identifiers, individual-level HCC scores would not be available on a timely basis for use in regular public reporting.

Chen et al conclude that “findings indicated that current risk factors insufficiently adjust for the variation in beneficiaries’ clinical and functional conditions that affects patient experience.”

 We thank the authors for bringing their concerns forward, but given the methodological issues presented here, the relatively small size of their results, the extensive testing we conducted prior to national implementation using hundreds of claims-derived diagnoses, and the impracticality of using agency- and patient-level HCC scores, we think that the current risk adjustment strategy used in HHCAHPS, although not perfect, is effective.

 Center for Advanced Methods Development, RTI International (LML), Waltham, MA; Quantitative Health Sciences, University of Massachusetts Medical School (LML), Worcester, MA; Aging, Disability & Long-Term Care (WLA) and Survey Research Division (HG, AEK), RTI International, Durham, NC.

 Funding for this research was provided by CMS to RTI International under contract HHSM-500-2014-00447G.

 The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (LML, WLA, HG); acquisition of data (LML, HG); analysis and interpretation of data (LML, HG); drafting of the manuscript (LML, WLA, HG, AEK); critical revision of the manuscript for important intellectual content (LML, WLA, HG, AEK); statistical analysis (HG); administrative, technical, or logistic support (AEK); and supervision (WLA, AEK).

 Lisa M. Lines, PhD, MPH, Center for Advanced Methods Development, RTI International, 307 Waverley Oaks Rd, Ste 101, Waltham, MA 02452-8413. Email: llines@rti.org.

1. Chen HF, Tilford JM, Wan F, Schuldt R. CMS HCC risk scores and home health patient experience measures. 

2. Paddison CA, Elliott MN, Haviland AM, et al. Experiences of care among Medicare beneficiaries with ESRD: Medicare Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey results. 

. 2013;61(3):440-449. doi: 10.1053/j.ajkd.2012.10.009.

3. Quigley DD, Elliott MN, Setodji CM, Hays RD. Quantifying magnitude of group-level differences in patient experiences with health care. 

. 2018;53(suppl 1):3027-3051. doi: 10.1111/1475-6773.12828.

4. Smith LM, Anderson WL, Lines LM, et al. Patient experience and process measures of quality of care at home health agencies: factors associated with high performance. 

. 2017;36(1):29-45. doi: 10.1080/01621424.2017.1320698.

5. Kwon A, Bungay KM, Pei Y, et al. Antidepressant use: concordance between self-report and claims records. 

. 2003;41(3):368-374. doi: 10.1097/01.MLR.0000053019.79054.B6.

6. Schenck AP, Klabunde CN, Warren JL, et al. Evaluation of claims, medical records, and self-report for measuring fecal occult blood testing among Medicare enrollees in fee for service. 

. 2008;17(4):799-804. doi: 10.1158/1055-9965.EPI-07-2620.

7. Jylhä M. What is self-rated health and why does it predict mortality? towards a unified conceptual model. 

. 2009;69(3):307-316. doi: 10.1016/j.socscimed.2009.05.013.

8. Williams JS, Egede LE. The association between multimorbidity and quality of life, health status and functional disability. 

. 2016;352(1):45-52. doi: 10.1016/j.amjms.2016.03.004.

Articles

The authors disagree with previous research concluding that the Home Health Care Consumer Assessment of Healthcare Providers and Services (CAHPS) publicly reported data are insufficiently adjusted for patient comorbidities.

Lisa M. Lines, PhD, MPH

Articles

Risk Adjustment in Home Health Care CAHPS