The Mis-Measure of Physician Performance
Published Online: October 22, 2013
Seth W. Glickman, MD, MBA; and Kevin A. Schulman, MD
The United States is poised to embark on one of the largest measurement efforts in the history of medicine—the assessment of patient satisfaction with physician performance using the Clinician and Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS) surveys.1 The Affordable Care Act (ACA) directs the Secretary of Health and Human Services to compare individual physicians using patient experience measures.2 This initiative will impact over 700,000 eligible physicians and will be tied to reimbursement and the Centers for Medicare & Medicaid Services’ (CMS’) Physician Compare reporting feature starting in 2015 for provider groups and for all individual physicians by 2017.3
While acknowledging the role of CMS in catalyzing an increasing focus on the patient experience in provider organizations, we believe that the CG-CAHPS construct for evaluating individual physician performance is flawed. The final regulations to implement this section of the ACA for individual providers have not yet been written, thus the importance of raising our concerns about this process to the physician community now. Our concerns about tying individual physician performance to CG-CAHPS data center around 3 areas: 1) intrinsic versus extrinsic approaches to assessing the patient experience, 2) measurement issues, and 3) unintended consequences.
Intrinsic Versus Extrinsic Approaches to Assessing the Patient Experience
Since the inception of CAHPS, we have seen profound improvement in our understanding of the concept of service operations in healthcare.4 In a learning healthcare system, an information feedback loop to clinicians on their activities and outcomes is critical in achieving the best possible care.5 Service transformation is a dynamic process that requires actionable data for physicians and managers and the flexibility to focus on operational issues unique to each clinical environment. However, CG-CAHPS was not designed for such a framework. In fact, the explicit goal of CAHPS is to “develop standardized patient surveys that can be used to compare results across sponsors and over time” where priority is on public reporting rather than facilitating improvement at the practice level.1
By analogy, one of the key methodologies in the service operations field is the Toyota Production System.6 Here, information is developed and assessed in real time by all members of the production team. This approach cannot work when data points are 1) determined by an external group, 2) not designed for the specific environment at hand, and 3) not available in real time. Additionally, Toyota does not wait for the annual J.D. Power survey of consumers to decide how to improve its automobiles. Contrary to the Toyota model, CMS prevents physicians from taking ownership of measurement of the patient experience. For example, the surveys are generated by an external body in a standardized fashion which does not allow modification of the core measures. Further, in an effort to preserve the integrity of the external measurement process, CMS requires that its surveys be administered before any internally informed patient survey.7 This complexity makes it very difficult for a provider organization to collect its own patient experience data. Of note, the delay in CAHPS survey administration may be up to 6 weeks and responses cannot be tracked to individual patients, thereby limiting the utility and actionability of the data. There is no reason to believe the guidance on this issue will be any different for CG-CAHPS than it is for CAHPS Hospital Survey (HCAHPS).
This is not to argue the potential benefits to extrinsic measurement of the patient experience, such as greater transparency and accountability. However, we believe that a preferred approach to measurement would be intrinsic, with a more limited approach to external evaluation that does not inhibit internal innovation within provider organizations. In fact, CMS recognizes the importance of this intrinsic approach, stating “HCAHPS survey items complement the data hospitals currently collect to support improvements in internal customer services and quality related activities.”8 However, current regulations do not allow sufficient flexibility around this framework. Later in this paper we will discuss ways in which CMS could consider implementing pay for performance concepts focused on enhancing internal performance efforts.
Measurement Challenges and Methodologic Issues
Specific constructs within CAHPS are indeed evidence-based and in cross-sectional studies patient experience measures are associated with improved clinical performance and outcomes.9,10 However, there is a gap in effective strategies to translate solutions to improving the patient experience into clinical practice. The creation of financial penalties and public reporting programs without a more robust understanding f the science of improving the patient experience in provider organizations may fail to achieve the policy goal of enhancing patient care or may have unintended consequences.11 Previous work suggests that substantial change within provider organization around the patient experience will only occur with significant changes in organizational strategy, leadership, and culture.12 These organizational challenges require actionable data to help managers lead this transformation.
CG-CAHPS proposes to capture data on at least 300 patient encounters per practice and 45 surveys per physician. Currently, participation in the program is the sole criterion established by CMS for receipt of payment updates, but in the hospital realm, these data have been used to “incentivize” performance as well. Without a specific construct of how these data will be used in physician payment, we envision several major challenges. In measuring performance of 700,000 physicians, minute differences on a 10-point likert scale will be statistically significant. If the data are used to rank physicians (assuming a score distribution similar to HCAHPS), a change of 100,000 places in ranking would result from a difference of only 0.1 on an overall score. Moreover, ceiling effects could exacerbate these measurement issues within and across practices, especially as they develop strategies to improve scores to try to achieve payment updates.
PDF is available on the last page.