Kaylyn E. Swankoski, MA

 To determine how the multipayer Comprehensive Primary Care (CPC) initiative that transformed primary care delivery affected patient experience of Medicare fee-for-service beneficiaries. The study examines whether patient experience changed during the 4-year initiative, whether ratings of CPC practices changed relative to ratings of comparison practices, and areas in which practices still have an opportunity to improve patient experience.

 Prospective study using 2 cross-sectional samples of more than 25,000 Medicare fee-for-service beneficiaries attributed to 490 CPC practices and more than 8000 beneficiaries attributed to 736 comparison practices.

 We analyzed patient experience 8 to 12 months and 45 to 48 months after CPC began, measured using 5 domains of the Consumer Assessment of Healthcare Providers and Systems Clinician and Group survey with Patient-Centered Medical Home items, version 2.0. A regression-adjusted analysis compared differences in the proportion of beneficiaries giving the best responses (and, as a sensitivity test, mean responses) to survey questions over time and between CPC and comparison practices.

 Patient ratings of care over time were generally comparable for CPC and comparison practices. CPC had favorable effects on measures of follow-up care after hospitalizations and emergency department visits.

 Practice transformation did not alter patient experience. The lack of favorable findings raises questions about how future efforts in primary care can succeed in improving patient experience.

The 4-year Comprehensive Primary Care (CPC) initiative aimed to transform primary care delivery. 

Patient ratings of care over the 4 years were generally comparable for Medicare fee-for-service beneficiaries in CPC and comparison practices. 

The only differences were the higher proportions of beneficiaries in CPC practices who reported receiving follow-up care from their primary care practice after either a hospitalization or an emergency department visit (by 11 and 8 percentage points, respectively); such follow-up is a focus of CPC. 

Results suggest that transforming care during the 4-year CPC initiative did not alter patient experience.

As CMS and other payers test the patient-centered medical home (PCMH) and similar models, and as they increasingly pay for care through alternative payment models that reward quality and value,

 it is important to measure how these efforts affect patient experience of care.

In 2012, CMS launched the Comprehensive Primary Care (CPC) initiative, a unique collaboration with 39 other payers to improve primary care delivery. The 4-year initiative helped practices implement 5 functions in their delivery of care—(1) access and continuity, (2) planned chronic and preventive care, (3) risk-stratified care management, (4) patient and caregiver engagement, and (5) coordination of care across the medical neighborhood. CMS selected 502 practices in 7 US regions to participate. To help practices improve care delivery, CPC provided enhanced payment, a robust learning system, and data feedback.

CPC was expected to improve costs, quality, and patient experience of care. Patient-centeredness was a core tenet underlying CPC, and several features of this initiative aimed to improve patient experience of care. Practices were expected to improve access to care, engage patients to guide quality improvement, integrate culturally competent self-management support and shared decision making into usual care, and coordinate care across the patient’s providers. Also, CMS and some of the other participating payers considered patient experience when determining practice eligibility for shared savings payments.

Prior literature examining the effects of primary care transformation on patient experience, including a study examining the first 2 years of CPC, found few, generally small, statistically significant effects during the first 1 to 2 years of transformation.

 This paper examines the full 4 years of CPC to understand whether the lack of effects on patient experience for CPC and other primary care transformation models was due, in part, to short follow-up periods that did not allow adequate time for practice transformation to affect patient experience as intended. We examine how patient ratings from more than 25,000 Medicare fee-for-service (FFS) beneficiaries attributed to 490 CPC practices compare with those from more than 8000 beneficiaries in 736 comparison practices (selected using propensity score matching), in 2013 (8-12 months after CPC began) and in 2016 (5 months before CPC ended).

For each CPC region, we used propensity score matching to select comparison practices from a pool of potential comparisons containing practices (1) in the same regions as CPC practices that had applied to CPC but were not selected and (2) in nearby areas with similar demographic and market factors that had enough practices for matching (ClinicalTrials.gov number NCT02320591).

We selected up to 5 comparison practices per CPC practice using “full matching” to form matched sets that contained 1 CPC and multiple comparison practices or 1 comparison and multiple CPC practices. The evaluation included the 497 CPC practices participating at the end of the first quarter of CPC and their 908 comparison practices. The comparison group had similar patient, practice, and market characteristics to the CPC practices before CPC began.

We administered an annual survey to a cross-sectional sample of Medicare FFS beneficiaries attributed to all open CPC practices (regardless of whether they still participated) and comparison practices. Using claims data, Medicare beneficiaries were attributed to practices where they had the largest share of selected evaluation and management visits to primary care clinicians over the prior 2 years. We invited about 60,000 of the roughly 300,000 Medicare FFS beneficiaries attributed to CPC practices, and 20,000 of the approximately 600,000 beneficiaries attributed to comparison practices, to respond to each survey round (

). We expected responses from 40 beneficiaries per CPC practice and 14 beneficiaries per matched set of comparison practices. We selected larger CPC samples to support CPC practice—level estimates used in practice-level feedback and CMS’ shared savings calculations.

Our survey instrument contains items from the Consumer Assessment of Healthcare Providers and Systems Clinician and Group survey with Patient-Centered Medical Home supplemental items (CAHPS-PCMH) version 2.0 and several additional questions that we developed about aspects of CPC.

 We measured 5 dimensions of patient experience during the prior year using 17 questions from the CAHPS-PCMH composite measures (shown in 

 Although CMS and some other payers used these measures when determining whether practices received shared savings, CPC did not explicitly focus on each item.

In addition to the 17 questions in the 5 composites, the survey contained 30 questions about patients’ experiences with care emphasized by CPC. These include timely access to care and information; providers’ communication with patients, attention to patients’ behavioral health needs, coordination of care with specialists, and follow-up after hospital stays and emergency department (ED) visits; patient engagement in caring for chronic conditions; comprehensiveness of care; and patients’ overall rating of care received from the provider. A sixth CAHPS composite measure—providers’ knowledge of the care that patients received from other providers&mdash;was also used, but the 2 questions within that domain were examined separately because the composite had low reliability.

 Among the 30 questions not from the composite measures, 2 were asked only in 2013 and 2 only in 2016.

We administered 4 rounds of the survey by mail during the 51-month initiative (June-October 2013; July-October 2014, 2015, and 2016). These occurred 8 to 12, 21 to 24, 33 to 36, and 45 to 48 months after CPC began, respectively.

Although we collected data from 4 survey rounds, we decided to compare ratings between CPC and comparison practices in 2013 (or the first year the question was asked) and in 2016 to observe where patient experience differed between the 2 groups early in and near the end of CPC. Using an intent-to-treat design, we surveyed beneficiaries in CPC practices regardless of whether the practice was still participating in CPC. We did not administer surveys to beneficiaries in practices that had closed more than 6 months before the survey.

Because we were not able to collect data before CPC, differences in any of the years may reflect pre-existing differences between CPC and comparison practices. In case CPC affected patient experience before the first survey, we did not calculate difference-in-differences estimates.

Our main analyses examine the proportion of respondents who answered each question with the best response. To test the sensitivity of these findings, we also analyzed the mean response.

We calculated the predicted probability of answering the best response and the mean responses using logistic and ordinary least squares regressions, controlling for baseline beneficiary and practice characteristics, as well as education level reported on the survey. Because most questions were answered by more than 95% of survey respondents, we calculated findings among nonmissing data and did not adjust for question nonresponse. For all regressions, we weighted estimates using beneficiary-level nonresponse weights (to make the sample similar to all attributed beneficiaries) and practice-level matching weights (to ensure similar CPC and comparison samples). We adjusted standard errors to account for clustering of respondents within practices and matched sets and for respondents answering in multiple rounds.

We tested for subgroup effects in 2016 based on the beneficiary’s (1) practice’s affiliation with a healthcare system, because systems may have more resources to transform and coordinate care among their providers; (2) practice size (measured by the number of primary care clinicians in the practice in 2012, before CPC began: 1, 2-3, 4-5, or ≥6), because larger practices might provide more access and services, but there might be less direct contact between a patient and their specific clinician; and (3) relative health risk, because sicker patients might have more interactions with their practice and more need for varied types of services. Risk is measured by whether the respondent’s 2012 Hierarchical Condition Category score, a measure of risk for subsequent expenditures, was above or below the median for respondents across all survey rounds.

Using 2-tailed tests at the 5% significance level, the analysis had 80% power to detect effects of 1 to 3 percentage points over time and between CPC and comparison practices for the composite measures and for most individual questions. Exceptions were for questions that applied to a small proportion of respondents, such as beneficiaries who had phoned the provider’s office after hours, where we could detect differences of 7 to 12 percentage points.

 To limit multiple comparisons leading to false positives, we only considered responses between beneficiaries in CPC and comparison practices to be statistically different and of substantial importance if (1) the 

 value was less than .05 and (2) the difference between the 2 groups was larger than 5 percentage points (selected in consultation with CAHPS experts, because literature did not define what size difference would be substantively important for CAHPS or other patient experience measures). We took this approach rather than adjusting for multiple comparisons based on the American Statistical Association’s caution against overreliance on 

We excluded 7 or fewer CPC practices from the analysis in each round because they had closed more than 6 months before the survey. Depending on the round, we received completed surveys from 25,318 to 26,362 Medicare FFS beneficiaries in 495 to 497 CPC practices and 8088 to 9922 Medicare FFS beneficiaries in 811 to 908 comparison practices. Using survey responses, we excluded respondents who had not visited the practice in the year before they responded. The analytic sample included more than 25,000 beneficiaries attributed to 490 to 496 CPC practices and more than 8000 beneficiaries attributed to 736 to 818 comparison practices, depending on the round (Table 1). Response rates were between 44% and 48%. Sixteen percent of respondents answered in multiple rounds; fewer than 1% answered in all rounds.

Most practices implemented the intervention and participated throughout the 4-year initiative. Practices were required to report Milestones quarterly to CMS and were put on a corrective action plan for not meeting Milestones. Between 5% and 15% of participating practices were put on a corrective action plan each year; most of these practices had a deficiency in 1 of 8 Milestones and were able to correct it within 2 quarters. Over the 4-year initiative, 60 (12%) of 497 practices withdrew or were terminated from the initiative. Of the 60, 29 left to become accountable care organizations, 40 left in the last 2 years of the initiative, and 9 were terminated.

We tested the internal consistency reliability of the 4 composite measures that combine multiple questions; they demonstrated adequate reliability with McDonald’s omega values of 0.76 to 0.96.

For both CPC and comparison practices, ratings across the composites varied in 2013 (the first segment of each bar in the 

). Three composites had more room for improvement (timely appointments, care, and information; providers support patients in taking care of their own health; and providers discuss medication decisions with patients); between 46% and 63% of beneficiaries gave their practices the best ratings. Beneficiaries’ ratings of the other 2 composite measures—providers’ communication with patients and patients’ overall rating of the provider&mdash;were already fairly high in 2013, with more than 75% of the responding beneficiaries providing the most favorable responses.

Regardless of opportunities for improvement in the 2013 scores, improvements in beneficiaries’ ratings between 2013 and 2016 were minimal (less than 3 percentage points) for 4 of the 5 composite measures. The second segment in the Figure shows the changes over time for each composite measure; because changes were small, the second segment is barely visible for most composites. The exception was the composite measure for providers support patients in taking care of their own health. Between 2013 and 2016, both CPC and comparison practices experienced a statistically significant and meaningful improvement in beneficiaries’ ratings of this composite of 5 to 6 percentage points (

Overall, CPC did not improve beneficiary ratings for the 5 composite measures, with CPC and comparison practices reporting comparable ratings for each measure in 2013 and in 2016. Results for mean responses were also comparable for CPC and comparison practices (

). There were no differential effects of CPC on beneficiaries’ ratings in 2016 for the 3 subgroups (

Individual Questions Not in the Composite Measures

Responses to the 28 questions asked in 2013 or 2014 and the 28 asked in 2016 that were not in the composite measures also indicate that beneficiaries’ experiences with care were generally comparable in CPC and comparison practices over the 4-year initiative (

). There were no meaningful differences in beneficiaries’ ratings for 26 of the 28 questions in 2013 (or the earliest year the question was asked) and 25 of the 28 questions asked in 2016.

The notable exceptions were that more beneficiaries in CPC than comparison practices reported receiving follow-up care after hospital and ED visits both early in and near the end of CPC. In 2013, more beneficiaries in CPC than in comparison practices who stayed in a hospital overnight or longer in the previous year reported that they saw a doctor, nurse practitioner, or physician assistant in the provider’s office within 2 weeks of the most recent hospitalization (70% vs 65%, respectively; 

= .002). In 2014, the first year the question was asked, more beneficiaries in CPC than in comparison practices who visited the ED for care in the previous year reported that they were contacted by their provider’s office within 1 week of the most recent visit (53% vs 48%, respectively; 

In 2016, more beneficiaries in CPC than in comparison practices reported that they were contacted by their provider’s office within 3 days of a hospital discharge (60% vs 50%, respectively; 

<.001). Beginning in 2014, to align with the 2014 CPC requirements, we shortened the follow-up window in the 2013 question and no longer limited the follow-up to occur in the practice. More beneficiaries in CPC than in comparison practices reported that they were contacted by the provider’s office within 1 week of the most recent ED visit (59% vs 51%, respectively; 

There was 1 unfavorable difference. Fewer beneficiaries in CPC practices than in comparison practices reported in 2016 that they always received an answer to their medical question as soon as needed when emailing their provider (69% vs 75%, respectively; 

 = .039). However, more than 92% of beneficiaries in both CPC and comparison practices reported that they did not email their provider with a medical question in the past 12 months and therefore did not answer this question.

Findings were similar for mean responses (

Despite responses indicating opportunities to improve care delivery, beneficiaries were generally pleased with their providers. Roughly 80% of beneficiaries in both CPC and comparison practices rated their provider as a 9 or 10 out of 10 in 2013 and 2016. In 2014, the survey began asking beneficiaries to compare the care they received in the past 12 months with the care they received at the practice in the previous year. In each of the 3 years this question was asked, about 17% of beneficiaries in CPC and comparison practices reported that the care they received from the provider was much better than in the prior year; about two-thirds reported that the care compared with 1 year ago was about the same (data not shown).

These findings suggest that although CPC practices were undergoing substantial changes to improve care delivery, CPC did not alter patient experience. The areas in which there were effects—the larger percentages of beneficiaries in CPC practices who reported that their provider followed up with them after hospital stays and ED visits&mdash;reflect CPC’s emphasis on increasing primary care involvement after acute care.

Prior studies have found mixed effects of PCMH adoption on patient experience. These studies examined patient experience after a shorter exposure (1-2 years) of their practices to transformation. Four studies that looked at the impact of medical home transformation on patient experience of care found no statistically significant effects on patient experience 1 to 2 years after the intervention began.

 Four other studies (2 of which focused on the same intervention) found favorable and statistically significant, but generally relatively small or isolated, effects in some dimensions of patient experience.

Articles

Four years of practice transformation toward comprehensive primary care had little effect on patient experience.

Patient Experience During a Large Primary Care Practice Transformation Initiative

To determine how the multipayer Comprehensive Primary Care (CPC) initiative that transforms primary care delivery affects the patient experience of Medicare fee-for-service beneficiaries. The study examines how experience changed between the first and second years of CPC, how ratings of CPC practices have changed relative to ratings of comparison practices, and areas in which practices still have opportunities to improve patient experience.

Prospective study using 2 serial cross-sectional samples of more than 25,000 Medicare fee-for-service beneficiaries attributed to 496 CPC practices and nearly 9000 beneficiaries attributed to 792 comparison practices. 

 We analyzed patient experience 8 to 12 months and 21 to 24 months after CPC began, measured using 6 domains of the Consumer Assessment of Healthcare Providers and Systems Clinician and Group 12-Month Survey with Patient-Centered Medical Home supplemental items. We compared changes over time in patients giving the best responses between CPC and comparison practices using a regression-adjusted difference-in-differences analysis.

 Patient ratings of care over time were generally comparable for CPC and comparison practices, with slightly more favorable differences—generally of small magnitude&mdash;for CPC practices than expected by chance. There were small, statistically significant, favorable effects for 2 of 6 composite measures measured using both the proportion giving the best responses and mean responses: getting timely appointments, care, and information; providers support patients in taking care of their own health; and providers discuss medication decisions. There was an additional small favorable effect on the proportion of patients giving the best response in getting timely appointments, care, and information; there was no effect on the mean.

During the first 2 years of CPC, CPC practices showed slightly better year-to-year patient experience ratings for selected items, indicating that transformation did not negatively affect patient experience and improved some aspects slightly. Patient ratings for the 2 groups were generally comparable, and both faced substantial room for improvement. 

		Two years into CPC, Medicare patient ratings of care over time were generally comparable for CPC and comparison practices. 

		There were statistically significant favorable effects in the proportion of patients giving the best responses for 3 of 6 composite measures of the Consumer Assessment of Healthcare Providers and Systems Clinician and Group 12-Month Survey with Patient-Centered Medical Home supplemental items: getting timely appointments, care, and information (2.1 percentage points); providers support patients in taking care of their own health (3.8 percentage points); and providers discuss medication decisions with patients (3.2 percentage points). 

		Results suggest that transforming care during the first 2 years of CPC did not negatively affect patient experience but did generate some small improvements.

CMS is seeking to tie 50% of payments to quality or value through alternative payment models by 2018

 by working with payers around the country to test the patient-centered medical home (PCMH) and similar models to improve primary care delivery and pay for value instead of volume.

 Thus, it is important to measure how this transformation is affecting the way in which patients experience care and to identify opportunities to continue to improve patient experience.

In a unique collaboration, CMS and 39 public and private healthcare payers launched the Comprehensive Primary Care (CPC) initiative in October 2012 to improve primary care delivery in the United States. CPC helped practices implement 5 key functions in their care delivery—1) access and continuity, 2) planned chronic and preventive care, 3) risk-stratified care management, 4) patient and caregiver engagement, and 5) coordination of care across the medical neighborhood&mdash;supported by continuous data-driven improvement, enhanced accountable payment, and optimal use of health information technology. CMS selected 502 practices in 7 US regions to participate. To help participating practices improve their care delivery, CPC provided them with enhanced payment, a learning system, and data feedback during the 4-year initiative.

CPC aimed to improve cost, quality, and patient experience of care. This paper focuses on patient experience, examining how the ratings of more than 25,000 Medicare fee-for-service (FFS) beneficiaries attributed to 496 practices participating in CPC at the time of the first survey changed between the first and second years of CPC. This paper also identifies how ratings of CPC practices changed relative to the ratings of comparison practices, selected using propensity score matching,

and areas where practices could still improve. 

Patient-centeredness was a core tenet of the model, and several aspects of CPC aimed to improve patient experience of care. Practices were expected to provide better access to care, engage patients in order to guide quality improvement through surveys and/or a patient and family advisory council, integrate culturally competent self-management support and shared decision-making tools into care, coordinate care across the medical neighborhood, and use a personalized plan of care for high-risk patients. In addition, patient experience was used to help determine eligibility for shared savings payments.

We conducted a repeated cross-sectional study using a large sample of Medicare FFS beneficiaries attributed to CPC practices and to a set of comparison practices selected using propensity score matching to have similar market-, practice-, and patient-level characteristics before CPC began. We examined changes in patient ratings and used difference-in-differences (DID) to evaluate how CPC practices’ ratings improved relative to comparison practices between 1 year (8 to 12 months) and 2 years (21 to 24 months) after CPC began. We did not draw inferences about effects from tests of each hypothesis separately, but rather from the findings across the set of questions and composites, particularly the summary composites. 

CPC practices are primary care practices in 7 US regions: 4 states (Arkansas, Colorado, New Jersey, and Oregon) and 3 geographic areas (Cincinnati—Dayton [Ohio and Kentucky], Capital District–Hudson Valley [New York], and Greater Tulsa, Oklahoma). We drew comparison practices from: 1) those that had applied to CPC in the same regions as the CPC practices but were not selected, and 2) those in areas near the CPC regions that had reasonably similar demographics and market factors and had enough practices for matching.

Using Medicare claims data, we attributed Medicare FFS beneficiaries to practices where they had most of their evaluation and management visits to primary care clinicians over the prior 2 years; using survey data, we identified attributed Medicare FFS beneficiaries who had visited the practice at least once in the 12 months before the survey round began. 

In each survey round, we mailed questionnaires to a random sample of an average of 119 attributed Medicare FFS patients from each CPC practice and an average of 24 attributed Medicare FFS patients from comparison practices. These sample sizes aimed to yield completed surveys with at least 40 attributed Medicare FFS respondents per CPC practice and 14 respondents per matched set of comparison practices (the larger sample in CPC practices supported practice-level feedback). We followed the National Committee for Quality Assurance’s sampling guidelines for the number of patients to sample in each practice; more patients were sampled in practices with more clinicians.

 The average number of completed surveys was 53 per CPC practice and 18 per comparison practice set, exceeding our targets of 40 and 14, respectively.

In 2013, we obtained response rates of 45% and 46% for CPC and comparison practices, respectively. We then excluded patients from 2 of the 497 CPC practices and their comparison matched sets because the calculated weights of the patients in those practices—which combined matching weights and nonresponse weights&mdash;were large outliers and would have unduly influenced the results. This left samples of 25,843 Medicare FFS patients in 495 CPC practices and 8949 Medicare FFS patients in 818 comparison practices. For the 2014 survey, we sampled patients from 496 CPC practices: 2 of the 497 total CPC practices in 2013 closed in summer/fall 2013 and 1 split into 2 practices in 2014. Similarly, the number of comparison practices in our sample fell from 818 in 2013 to 792 in 2014. In 2014, response rates were 48% and 47% for CPC and comparison practices, respectively. The final sample for the 2014 survey contained 26,356 Medicare FFS patients in 496 CPC practices and 8865 Medicare FFS patients in 792 comparison practices. About 15% of respondents replied in both survey rounds.   

Our patient survey instrument contains items from the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Clinician and Group 12-Month Survey with Patient-Centered Medical Home supplemental items.

 The survey asks patients about their experience of care over the previous 12 months across 6 dimensions of primary care: 1) patients’ ability to get timely appointments, care, and information; 2) how well providers communicate; 3) providers’ knowledge of the care patients received from other providers; 4) if providers support patients in taking care of their own health; 5) if providers discuss medication decisions with patients; and 6) patients’ overall rating of their primary care provider. To summarize patient experience of care, we created 6 summary composite measures using 19 questions following the CAHPS Clinician and Group survey scoring instructions.

 details the specific patient care experiences that the 6 summary composite measures evaluate. Although CMS and some other payers used these composite measures to help determine whether practices received shared savings, CPC did not focus explicitly on each item. In addition to the 19 questions in the 6 summary measures, 25 other questions gauged patient experience of care, yielding 44 total questions (listed in 

We administered 2 rounds of the survey: 1) June through October 2013, 8 to 12 months after CPC began; and 2) July through October 2014, 21 to 24 months after CPC began. All surveys were administered by mail, following the CAHPS Clinician and Group survey instructions, with slightly modified timing of mailings. 

We analyzed both the proportion of patients who gave the best (most favorable) response (response scales varied from 2-point [yes/no] to 11-point [0 to 10 rating scale]) and mean response. Our main analysis is on the best responses. Examples of these responses are: 1) the provider always explained things to the patient in a way that was easy to understand; 2) in the last 12 months, between visits, yes, the patient did receive reminders about tests, treatment, or appointments from the provider’s office; and 3) the patient got an appointment for care needed right away that same day. 

We first calculated the likelihood that patients responded to a question with the best response using logistic regressions, controlling for baseline patient and practice characteristics and education level reported on the survey. We calculated predicted probabilities for each of the 44 questions (eAppendix Table A reports results from each question).

In addition to analyzing responses to individual questions, we looked at the 6 summary composite measures containing 19 of the 36 questions asked in both rounds, following the CAHPS Clinician and Group survey scoring instructions.

 We first calculated patient-level composite measures by averaging nonmissing binary indicators for whether the patient’s response was the best option across each question in the composite. (That is, if the composite contained 4 questions and the respondent answered all 4 and gave the best response for 3, the patient’s score was 0.75.) Ordinary least squares regressions controlled for baseline patient and practice characteristics and the respondent’s education level. 

For the 36 questions in both survey rounds and the 6 composite measures, we used DID to compare the changes over time between CPC and comparison practices in the proportion of patients who gave the best response. For the 8 questions that were asked in only 1 survey round, we tested the within-year difference between CPC and comparison practices. 

For all regressions, we weighted estimates using practice-level nonresponse and matching weights (to ensure that CPC and comparison samples were similar) and clustered standard errors at the practice level to account for practice-level clustering and respondents answering in more than 1 round. We considered 

 <.10 to be statistically significant (but relied on combined findings across related measures to draw inferences about whether the results were likely to be true effects or chance differences). The analysis had 80% power to detect small differences of 1 to 2 percentage points between CPC and comparison practices. To test the sensitivity of our findings, we repeated the analysis using mean responses (standardized on a 0-to-1 scale). (See the eAppendix for full description of methods.)

We tested the internal consistency reliability for 5 of the 6 composite measures that combine multiple questions (the composite measure for patients’ rating of the provider contains only 1 question). Four of the 5 composite measures had adequate reliability with McDonald’s omega values between 0.80 and 0.96. One composite—providers’ knowledge of the care patient received from other providers&mdash;had less reliability (McDonald’s omega = 0.54).

There were small, statistically significant, favorable effects of CPC on the percentage of respondents choosing the best responses for 3 of 6 composite measures: 1) getting timely appointments, care, and information (2.1 percentage points; 

= .046); 2) providers support patients in taking care of their own health (3.8 percentage points; 

 <.001); and 3) providers discuss medication decisions with patients (3.2 percentage points; 

). These were driven by small (less than 2 percentage points) year-to-year improvements for CPC practices and small declines (less than 2 percentage points) for comparison practices. We did not adjust for multiple comparisons; however, estimated effects for half of the 6 composite measures are statistically significant at the .05 level and 2 are significant at the .01 level, so we are confident concluding there were small effects. 

Despite CPC practices showing improvement over comparisons in these 3 areas of care, only about half of CPC and comparison patients gave the best responses for getting timely appointments, care, and information, and for providers supporting patients in taking care of their own health, indicating room for improvement. There was less room for improvement in the other 3 composites where differences between CPC and comparison practices were not statistically significant—how well providers communicate, providers’ knowledge of the care patients received from other providers, and patients’ rating of providers&mdash;as more than 75% of CPC and comparison patients gave the best responses (Table 2).

To understand the factors driving the composite measure results, we turned to the 19 questions in the composite measures (eAppendix Table A). There were statistically significant and favorable effects of CPC for 5 of the 19 questions—a result of small year-to-year improvements for CPC practices coupled with small declines over time for comparison practices. For 2 of these 5 questions, changes from 2013 to 2014 in the proportion of CPC patients providing the best responses increased between 2 and 3 percentage points: 1) someone in the provider’s office asked the patient during the last 12 months whether there were things that made it hard for the patient to take care of his or her health (increased 2.8 percentage points to 35.6% in 2014), and 2) if a patient talked about starting/stopping a prescription medicine, the provider asked what the patient thought was best (increased 2.3 percentage points to 78.1% in 2014). For the other 3 questions that showed favorable effects of CPC, the year-to-year increases among CPC patients were only 0.2 to 1.1 percentage points. The year-to-year changes in CPC practices for the remaining 14 questions were not statistically different from comparison practices.

Mean responses yielded results similar to the best responses. There were small favorable effects of CPC in 2 composite measures: providers support patients in taking care of their own health (0.04 points on a 1-point scale; 

<.001) and providers discuss medication decisions with patients (0.02 points; 

= .006). Unlike the analysis using the best responses, the favorable difference on the composite measure for getting timely appointments, care, and information was not statistically significant (

= .117). The other 3 composite measures showed no statistically significant CPC-comparison differences (Table 2). (See 

Looking at all 36 questions asked in both survey rounds, the change in patients giving the best ratings of care over time was generally comparable for CPC and comparison practices, with slightly more—generally small&mdash;differences favoring the CPC practices than we would expect by chance. There were no statistically significant CPC-comparison differences in 75% of the 36 comparisons. Twenty-five percent&mdash;versus the 5% expected to occur by chance&mdash;showed more favorable ratings for CPC than comparison practices. However, year-to-year improvements for the CPC practices were small (5 percentage points or less) for 8 of these 9 measures, and CPC practices experienced a small decline for the other measure (eAppendix Table A). 

Reflecting larger issues with healthcare delivery, CPC and comparison practices both had sizable opportunities for improvement. In 2014, for example, just 43% of patients in both groups answered that when they phoned their provider’s office for care needed right away, they usually got an appointment that same day and only 29% of patients answered that they always saw the provider within 15 minutes of appointment time when they had an appointment—2 measures of access. About a third of patients reported that practice staff spoke with them about a personal, family, mental, emotional, or substance abuse problem in the past year, suggesting the need for more screening for mental health issues. Another third of patients reported that someone at the practice had asked whether there are things that make it hard for the patient to take care of his or her health, suggesting room for more patient engagement (eAppendix Table A). Ratings were higher, but with room for improvement, for many other questions, including some on which CPC practices had statistically significant improvements relative to comparison practices.

Despite giving responses that indicate opportunities for improvement in many aspects of care, patients were pleased with their providers. Overall, three-fourths of both CPC and comparison patients rated their provider as a 9 or 10 out of 10 in both survey rounds.

There were 8 questions asked in only 1 round: 2 in 2013 and 6 in 2014. CPC patients were more likely than comparison patients to give the best responses for 4 of the 8 questions and equally likely to give the best responses for the other 4 questions. Three of the 4 favorable differences relate to transitional care. CPC required practices to improve coordination of care across the medical neighborhood, including patient follow-up after hospital stays and emergency department (ED) visits. CPC patients were more likely than comparison patients to report that someone in the provider’s office contacted them within 2 weeks after their most recent hospital stay in 2013 (or within 3 days after the most recent hospital stay in the 2014 survey) and that they were contacted by their provider’s office within 1 week of their most recent ED visit in 2014. These favorable differences for CPC, although statistically significant, were modest: between 3 and 5 percentage points (eAppendix Table A). 

In 2014, CPC patients were also slightly more likely than comparison patients to report that they always received a copy of their treatment plan when receiving care for a chronic condition (46% vs 43%). But comparable proportions of CPC and comparison patients reported that they were always asked their ideas or goals when making a treatment plan (36%) (eAppendix Table A). 

Two years after CPC began, CPC practices showed small improvements in the proportion of patients giving the best ratings for 3 of 6 CAHPS Clinician and Group composite measures relative to comparison practices, driven by slightly better year-to-year changes in the proportion of patients giving the best ratings of care on selected questions in the composites than expected by chance. Comparing the changes in mean responses over time between CPC and comparison practices also demonstrated small favorable effects of CPC in 2 of these 3 composite measures. Considering all 44 questions, there were slightly more favorable effects than expected by chance, again small. Generally, the proportions of patients giving the best ratings, as well as mean responses, were mostly comparable for CPC and comparison practices. These results suggest that the significant changes in care delivery during the first 2 years of CPC made minor improvements in patient experience and did not negatively affect it. 

Prior studies found mixed effects of PCMH adoption on patient experience, measured using different patient survey instruments. Four studies of the impact of medical home transformation on patient experience found no statistically significant effects 1 to 2 years after the intervention began.

 Similar to these CPC findings 2 years into the initiative, 3 studies found statistically significant, favorable, but generally relatively small or isolated, effects in some dimensions of patient experience with care.

For example, Kern et al (2013) followed up with patients 15 months after practice transformation and found statistically significant improvement at the 5% level in the proportion of respondents giving the best rating in the access to care domain (from 61% while the practices transformed to 69% at follow-up) and statistically significant improvement at the 10% level in experience with office staff (from 72% to 78%). The proportion of respondents giving the best ratings in the domain for follow-up with test results showed a statistically significant decline at the 10% level over time, from 76% to 69%. There were no effects in the other dimensions of patient experience that they measured: communication and relationships, disease management, doctor communication, and overall rating of the doctor. However, the study did not have a comparison group to net out any secular trends that may have affected patient experience.

The main limitation to this study is that the comparison group was not chosen experimentally; therefore, differences between patient ratings over time for the CPC and comparison practices might reflect unmeasured preexisting differences between the groups of patients, in addition to the effects of CPC. Further, we could not obtain a list of patients to sample in time to survey patients before the initiative began. Therefore, the DID estimates might understate the true effects of CPC because CPC practices may have already made some improvements between the start of CPC and the first survey round that began 8 months later. Alternatively, these estimates might overstate the extent that changes (and possible disruptions) during the first year of CPC led to short-term negative effects on patients in CPC practices. Indeed, the proportions of patients giving the best responses to CPC practices were generally 1 to 3 percentage points lower than for comparison practices for 35 of 38 questions in the 2013 survey and for all 6 composite measures. 

Despite these limitations, these findings are a timely and important contribution, as CMS launched the new CPC Plus model, the largest investment in advanced primary care to date.

 As the US healthcare system moves from traditional FFS to new models that reward value instead of volume, maintaining and improving patient experience of care will be critically important. Our results allay concerns that the disruptions inherent in the early stages of primary care transformation and payment reform impair patient experience of care. The small improvements in patient experience also suggest that future efforts in primary care may be a path toward improved patient experience.&ensp;

Practice transformation toward comprehensive primary care slightly improved patient experience in 3 of 6 domains of care: access, provider support, and shared decision making.

Kaylyn E. Swankoski, MA

Articles

Patient Experience During a Large Primary Care Practice Transformation Initiative

Patient Experience Midway Through a Large Primary Care Practice Transformation Initiative