Denise D. Quigley, PhD

 To assess whether Medicare patients’ reports of denied care, appeals/complaints, and satisfactory resolution were associated with ratings of their health plan or care.

 Retrospective analysis of 2010 Medicare Advantage Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey data.

 Multivariate linear regression of data from 154,766 respondents (61.1% response rate) tested the association of beneficiary ratings of plan and care with beneficiary reports of denied care, appeals, complaints, and complaint resolution, adjusting for beneficiary demographics.

 Beneficiaries who reported being denied needed care rated their plans and care significantly less positively, by 17.2 points (on a 100-point scale) and 9.1 points, respectively. Filing an appeal was not statistically significantly associated with further lower ratings. Beneficiaries who filed a complaint that was satisfactorily resolved gave slightly lower ratings of plans (—3.4 points) and care (–2.5 points) than those not filing a complaint (

 Lower ratings from patients reporting complaints and denied care may notably affect the overall 0-10 CAHPS ratings of Medicare Advantage plans. Our results suggest that beneficiaries may attribute the actions that lead to complaints or denials to plans more than to the care they received. Successful complaint resolution and utilization management review might eliminate most deficits associated with complaints and denied care, consistent with the service recovery paradox. High rates of complaints and denied care might identify areas that need improved utilization management review, customer service, and quality improvement. Among those reporting being denied care, filing an appeal was not associated with lower patient ratings of plan or care.

Patients who reported denials of needed care gave low ratings to their care, and especially their plan, regardless of whether they appealed the denial. Beneficiaries satisfied with complaint resolution provided only slightly lower ratings than those with no complaints.

Medicare beneficiaries may largely attribute the actions that lead to complaints or denials to their plans more than to the care they received.

Medicare plans should focus on successful complaint resolution, as it can substantially mitigate the low ratings patients give their plan that result from whatever motivated them to file a complaint.

Patients’ complaints about their health insurance plan and denials of care are important aspects of the patient experience. The extent to which specific care is necessary can be complex and contentious.

 Denials to cover care may reflect barriers to accessing needed care or appropriate curtailing of unnecessary care. Throughout this article, we will refer to a health plan’s decision to not cover care as “denial of care” as an imperfect shorthand for the patient’s report, recognizing that patients can choose to pay out-of-pocket or out-of-network for their care. Similarly, complaints about health insurance plans may indicate a misunderstanding, a lack of plan responsiveness, or appropriate voicing of patient concerns.

Managed care organizations have delivery systems that greatly influence interactions between physicians and patients.

 Their rules and restrictions on choice of providers, network coverage, review processes, preauthorization, and drug formularies may create conflicts with patients and providers. Through the denial and appeals process, patients, physicians, and health plans communicate and determine what care is medically necessary.

As required by CMS, all Medicare Advantage (MA) contracts (health plans) must provide beneficiaries a means to file a complaint or appeal a denial of coverage.

 Denied coverage for care may adversely affect MA enrollees’ perceptions of plans and care. Nineteen percent of 2010 Medicare beneficiary disenrollments were prompted by denials of coverage for medical services,

 and over one-third of callers to the Medicare Rights Center’s national helpline expressed difficulty managing coverage denials and appeals.

Health plans may benefit from how they respond to grievances, complaints, and service failures, known as service recovery.

 found that it is not the service failure that reduces customer ratings of their plan or care, but the service provider’s actions (ie, recovery) following the failure that matter. This phenomenon is the “service recovery paradox,” wherein those who complain and see their problems resolved by the offending organization may rate the organization higher than those who experience no problems.

Service recovery in healthcare differs from other settings, like retail, banking, restaurants, or hotels, because it may not be possible to “make things right” or “undo the situation” in healthcare. A service provider’s apology also could be construed as an admission of a mistake and become the basis for litigation.

 The few articles about healthcare service recovery in the literature primarily offer managerial suggestions to healthcare administrators rather than empirical evidence.

 One study of a single health management organization (HMO) found a strong positive relationship between satisfaction with the complaint handling process and overall satisfaction with the HMO.

In general, however, relatively little is known about how denials of coverage for needed care, subsequent appeals of the denied coverage or complaints, and resolution status affect a patient’s rating of their plan or care. The question remains whether patients attribute the denial or complaint to their plan or to their providers. These attributions may also differ by beneficiaries’ health status, given that a denial of care may have greater consequences for those in poor health or with multiple chronic conditions.

Previous research on patient complaints and plans’ reported denials of care has focused on whether a patient has negative or positive experiences with care coverage and identifying any contributing factors. For example, kidney transplant recipients’ interactions with their insurers

 suggest that patients have difficult interactions with insurers and limited skills in navigating insurance options, thereby curbing their access to needed medications and health services. More than one-third of Medicaid enrollees reported denials of care for their children in the previous year, with families of children with chronic conditions more likely to report a denial of care.

Among the few studies that have focused on patient complaints and plans’ reported denials of care, only 2 appear to examine the relationship between denial of care (whether it was appealed and its resolution) and the patient’s subsequent rating of their health plan or care. Denial of needed care was associated with lower ratings of health plan quality and significantly less trust in patients’ primary care physicians.

 Similarly, many patients who are denied authorization for emergency department (ED) care are dissatisfied with their health plan but still intend to return to the ED.

 These results follow Folger’s fairness theory,

 which states that judgments of fairness are related to people’s “would/could/should” reasoning about a decision by a powerful other. Fairness theory predicts that after being denied care, the patient, in assigning responsibility for the injustice, would choose the powerful other with the most “could” authority, here the health plan with authority over authorizations, not the care provider.

This article uses a nationally representative sample of MA enrollees to assess the relationship between patient-reported denials of care, appeals, complaints, and resolution and their association with overall patient ratings of plans and care. Information on plan disenrollment or health plan ratings prompted by coverage denials could be a valuable tool for consumers who need to select a plan or identify an important missed opportunity for Medicare and other health plans for service recovery with patients. Furthering understanding of complaints and reported denials of care is especially pertinent for Medicare recipients, whose average healthcare needs exceed those of the general population and who, in some cases, may have difficulty managing the complaint or appeal process due to age or infirmity.

In 2010, the MA Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey provided a unique opportunity to study these issues, as it included items that asked about complaints and denial of care. Under the Affordable Care Act, patient experience measures from this survey play a role in the determination of quality bonus payments for MA plans.

This study used additional items from the 2010 MA CAHPS survey to assess the association of MA enrollee ratings of their plan and care with whether they reported a denial of needed care, appealed the denial, or filed a complaint, along with the resolution status of the complaint. We first established the extent to which MA beneficiaries reported filing complaints with their plans or being denied coverage for needed care. Then we tested whether those with complaints/denials provided lower ratings of their health plan or care. Based on fairness theory, we hypothesized that members would assign less blame to the care they received because caregivers did not have the power  (ie, “could” authority) to accept or deny the appeal. In contrast, health plan management did have the “could” authority, and so responsibility was attributed to health plans. Thus, plans received lower ratings. Next, we tested whether the patient ratings of their plan and care differed for those filing an appeal and those with successful resolution of an appeal. Based on the service recovery paradox, we hypothesized that members with successful resolution would rate their health plan nearly as high or even higher than those who experienced no problems. Finally, we tested whether those who are in poorer health rated their plans or care more negatively following denied coverage or complaints.

We used data from the 2010 MA CAHPS survey, which included items on complaints and denial of care. This survey was a mail survey of MA beneficiaries, with telephone follow-up on nonrespondents.

Our population of interest was MA enrollees 65 years or older, excluding those in US territories. The analysis was restricted to responding beneficiaries (61.1% response rate for this population), excluding the 6% not completing items about complaints or denial of care, which left 154,766 cases. We examined responses to the survey items listed in 

 and estimated health contract—level standard deviations of denial, appeal, complaint, and resolution responses as the square root of contract variance components.

Second, multivariate linear regression predicted the responses to 2 ratings (health plan and care) from the denial, appeal, complaint, and resolution responses and several covariates: gender, age, race/ethnicity, education, self-reported health and mental health status, Census division, self-reported chronic conditions, Patient Health Questionnaire-2 depression symptoms,

 current smoking status, functional limitations, proxy assistance, dual eligibility for Medicaid, Part D or MA only, Special Needs Plan status, and whether the respondent had a personal doctor. We ran 4 sets of regression analyses to calculate 1) population estimates, 2) within-contract estimates, 3) associations, and 4) differences in these associations. Data were weighted to account for the sample design and nonresponse.

not shown) assessed whether those with poorer health had a stronger negative association with plan ratings than those in better health via an interaction term between overall health status and denial of service. As the average associations were similar for those who did and did not subsequently file an appeal of the denied care, we combined these groups in this analysis.

Characteristics of the MA beneficiaries 65 years and older appear in 

. Forty-three percent were male, 72% were non-Hispanic white, and 26% were 80 years or older. Eighteen percent had a 4-year college degree, and 14% did not have Part D coverage through their MA plan.

Prevalence of Denials, Appeals, and Complaints 

 shows the prevalence of denials, appeals, and complaints. Six percent of respondents reported that their plan denied care or services they believed they needed. Among beneficiaries who reported a denial of needed care or services, 49% filed an appeal. Overall, 8% of respondents filed a complaint, 54% of whom indicated that their complaint was resolved to their satisfaction.

The contract-level standard deviation of satisfactory complaint resolution from a linear mixed model was 9.9%, suggesting that approximately two-thirds of health plans had rates of satisfactory complaint resolution between 44% and 64% (after accounting for sampling error) and that most contracts had a rate of satisfactory complaint resolution ranging from 34% to 74%. Complaints, which had a contract-level standard deviation of 3.8%, involved a variety of issues and did not necessarily concern denials of needed care. Contract-level standard deviations were 3.7% for reports of denial of care and 5.4% for appeals.

 show multivariate linear regressions results that tested the association of denials, appeals, complaints, and resolution with overall ratings of health plan and healthcare, adjusting for beneficiary health and demographic characteristics. Compared with models that only included case-mix characteristics (Model 1), the addition of either denials and appeals (Model 2) or complaints and resolution (Model 3) resulted in a significant increase in predictive power, as shown by the increased R

<.001 for all partial F tests). For both plan and care ratings, denials and appeals added slightly more predictive power than complaints and resolutions.

Compared with beneficiaries who did not report denials of care, those who reported denials gave significantly lower care and plan ratings, rating their care 8.2 points and their plan 17.0 points lower than others on 100-point scales (Table 4). Similarly, those who reported filing a complaint rated their care 5.2 points and their plan 11.2 points lower than others. When denials and complaints were entered together in a model, all results remained highly significant (

<.001), with slightly less magnitude, but still indicating significantly lower plan and care ratings among those who filed an appeal or complaint. This suggests that each factor affected plan and care ratings independently.

Among those who reported being denied needed care, both care and plan ratings were somewhat worse for those who did not appeal their denials (a reduction of 7.0 points for those who appealed vs 9.4 for those who did not; 

<.001) (Table 5). Complaints that were not resolved satisfactorily were associated with much worse ratings of plan and of care than those that were resolved satisfactorily. In Model 3, including only complaint/resolution (along with beneficiary characteristics), complaints with a satisfactory resolution had an average 3.8-point lower plan rating, whereas those without a satisfactory resolution had an average 20.2-point lower plan rating (

<.001 for each, as well as for the difference between them).

Although denials/appeals and complaints/resolution were strongly associated with both care and plan ratings, the differences were much greater for plan ratings (Table 5). The model predicting care ratings with both denials/appeals and complaints/resolution (Model 4) had an R

 36% higher than the model with case-mix adjustors alone (Model 1); for plan ratings, the increase was 179%. In those models (Model 4), appeals were associated with average differences of —5.0 points for care rating and –12.4 points for plan rating. Similarly, filing a complaint that was not resolved satisfactorily was associated with differences of –6.6 and –16.3 points, respectively.

The relationship between patients and their health plans is fundamental to high-quality care. The denial and appeals process offers patients, physicians, and health plans a means to communicate and determine what care is medically necessary. The complaint resolution process allows patients to voice their concerns about care and attempt a solution. Perceived denial of needed care and unresolved complaints can adversely affect patients’ attitudes and experiences with their health plan and care.

This study of MA beneficiaries 65 years and older found that 6% reported being denied care in the past 6 months compared with the 7% prevalence in an earlier study of a single HMO’s commercially insured population

 in the prior 12 months. We found that 48% of beneficiaries who reported being denied care filed an appeal and 9% registered a complaint; 54% of complaints were resolved satisfactorily.

Our results reinforce the earlier finding

 that patients reporting a denial of coverage rated their care substantially lower than other patients. We extended previous research by comparing those who reported being denied needed care with those who were denied care and appealed their denial. In addition, beneficiaries who filed a complaint that was satisfactorily resolved had only slightly lower ratings of their plan and care, whereas those who had a complaint that was not satisfactorily resolved had much lower ratings, especially of their plan. We found that although denials/appeals and complaints/resolution were strongly associated with lower care and plan ratings, the association was much stronger for plan ratings. Our findings partially support the service recovery paradox, in that plan ratings improve when there is an appeal after a denial and a resolution after a complaint.

Given the significantly stronger negative associations of denied care and complaints with plan ratings in our study, it appears that beneficiaries attribute nonresolution of a complaint or denial of care to their health plan and much less so to the care they received. Patients may perceive denials and unresolved complaints as a breach of faith by their health plan, rather than their provider. These results support the prediction of fairness theory that patients associate accountability to the health plan (not the care provider) of authority over the complaint or the denied care.

Given that most MA plans satisfactorily resolved only about half of their complaints, there may be substantial opportunity to improve patients’ overall health plan ratings and reduce disenrollment by focusing efforts on increasing the rate of satisfactory complaint resolution and thoroughness during utilization review. Greater transparency with patients about coverage decisions, criteria, and utilization review may help.

Our findings also highlight the value of complaint and appeal data for identifying opportunities to improve customer service and patient education on criteria used in coverage decisions. This information could aid MA and other health plans in service recovery with patients.

This study had several limitations. It was not designed to link Medicare CAHPS survey responses to actual Medicare health plan coverage decisions, so it was not possible to determine whether each patient’s report of a denial of coverage reflected a health plan decision. Patients reporting a health plan denial may have misinterpreted statements by their physicians or actions by their physicians’ group practice. We knew only that the patients believed they had experienced a denial of coverage. Our study provided no data on subsequent healthcare utilization or outcomes, and it was not intended to evaluate the legitimacy of the grounds on which the MA plan based its denials of coverage.

Although just under half of patients who reported being denied care filed an appeal, filing an appeal does not seem to reflect a beneficiary perception that the denial was particularly egregious, but may instead reflect the beneficiaries’ self-efficacy with respect to filing an appeal. Patients who file a complaint that is not resolved satisfactorily may continue to view their plan experiences negatively, more so than their care experiences, which may increase their risk of disenrollment.

 The most common reason for disenrollment in 2010, applying to one-fourth of MA disenrollments, was concerns or issues related to out-of-network providers and prompted by an acute episode of illness or diagnosis (most often cancer).

Although there may be unobserved differences associated with beneficiaries whose complaints are satisfactorily resolved and those whose complaints are not, our results suggest that improved complaint resolution might substantially improve the perceptions of patients whose initial experiences were sufficiently negative to motivate a complaint. Given that many beneficiaries may have equally negative experiences but do not file complaints, active outreach to patients to elicit concerns might identify additional opportunities to improve care and patient experiences.

RAND Corporation (DDQ, DJK, MNE), Santa Monica, CA; RAND Corporation (AMH, JWD), Pittsburgh, PA; Carnegie Mellon University (AMH), Pittsburgh, PA.

 Preparation of this manuscript was supported through a cooperative agreement from CMS (contract number HHSM-500-2005-00028I).

 The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article. 

 Concept and design (DDQ, AMH, MNE); acquisition of data (MNE); analysis and interpretation of data (DDQ, AMH, JWD, DJK, MNE); drafting of the manuscript (DDQ, JWD, DJK, MNE); critical revision of the manuscript for important intellectual content (AMH, JWD, DJK, MNE); statistical analysis (AMH, DJK, MNE);  obtaining funding (MNE); administrative, technical, or logistic support (DDQ); and supervision (MNE).

Marc N. Elliott, PhD, RAND Corporation, 1776 Main St, Santa Monica, CA 90401-2138. Email: elliott@rand.org.

Articles

Patient ratings of plans and care were lower among beneficiaries filing complaints or reporting denied care. Appeals did not further predict ratings, but successful complaint resolution did.

Patient-Reported Denials, Appeals, and Complaints: Associations With Overall Plan Ratings

To review tools designed to evaluate and improve the extent of patient-centered medical home (PCMH) implementation.

Literature search and review of tools to evaluate PCMH “medical homeness” and track progress toward practice transformation.

We conducted a literature search to identify tools designed for evaluation and quality improvement during the PCMH change process. We identified and reviewed the content of 5 publicly available PCMH survey tools used by an administrator or clinical lead to collect data at the practice level for evaluation and/or quality improvement during PCMH implementation. We assessed each tool’s coverage of PCMH content, standards, and requirements. 

 We found that 3 tools (Patient-Centered Medical Home Assessment [PCMH-A], Primary Care Assessment Tool—Facility Edition, and Medical Home Care Coordination Survey–Healthcare Team [MHCCS-H]) are actionable for quality improvement. PCMH-A assesses the broadest array of practice capabilities and includes items pertaining to all National Committee for Quality Assurance PCMH standards. MHCCS-H was the only tool to contain items on comprehensiveness of care. There was variation in emphasis on main domains, with some content areas covered by only 1 tool. 

There is currently little evidence on which PCMH tools are associated with improved quality outcomes, as relatively few longitudinal studies have been conducted. Of the 5 tools we reviewed, only PCMH-A and MHCCS-H impose a light administrative burden (less than 10 minutes to complete) and can identify specific actions to improve a given practice capability. Each tool is lacking in a particular content area: PCMH-A, for example, lacks items on comprehensiveness of care, whereas MHCCS-H lacks items addressing access to care.

The American Journal of Accountable Care. 2017;5(4):e8-e18

The patient-centered medical home (PCMH) model is promoted as a way to transform US primary care practices.

 It incorporates current best practices for access to care, prevention, chronic disease management, care coordination, and responsiveness to patients.

 Monitoring the transformation of primary care practices to PCMHs requires understanding the context, change process, patient experiences of care, and outcomes of care.

 Even after obtaining recognition from one of the several organizations that grant “official” PCMH status, practices continue their transformation into a PCMH.

Practices, networks, and federally qualified health centers need an evaluative and quality improvement (QI) tool to collect data and conduct real-time analyses that help them understand their level of “medical homeness,” track progress toward transformation goals, and identify possible improvements. Practice coaches, payers, and pilot or demonstration sponsors may also use these tools. Most primary care practices cannot engage in a third-party evaluation or large-scale demonstration with medical record audits, site visits, and staff surveys, but need a practical survey tool they can use to gauge achievement of PCMH standards. 

Despite their importance, information about tools for assessing PCMH progress is limited. Malouin et al

 found that the Primary Care Assessment Tool (PCAT), available in multiple formats, was the only tool that scored high on primary care features, as it was designed to assess both structural and process features of primary care, but their evaluation was conducted early in PCMH adoption and focused on a narrower set of components than currently delineated in PCMH definitions and standards.

 included tools primarily used for the PCMH recognition process, which itself requires adequate and appropriate documentation, rather than for practice self-improvement, research, evaluation, or quality measurement. 

This article reviews PCMH tools designed for evaluative and QI purposes to assist practices and researchers wanting to select a survey tool to track outcomes of PCMH transformation. We also assess how well the identified PCMH tools align with the 2014 and 2011 standards of the National Committee for Quality Assurance (NCQA), the organization responsible for the largest share of PCMH recognition across the United States.

Using traditional bibliographic databases (PubMed, Google Scholar, and PsycInfo), we systematically reviewed the peer-reviewed literature on patient-centered healthcare to identify surveys or tools designed to evaluate or monitor the process of becoming a PCMH. We also searched the Agency for Healthcare Research and Quality’s online primary care measure directory. Search terms included “patient-centered medical home” OR “PCMH” OR “medical home” OR “care coordination” OR “team-based care” AND “questionnaire*” OR “telephone” OR “phone” OR “survey” OR “surveys” OR “tool” OR “tools.” We limited our search to English-language articles published since 1990 on adult populations. We identified additional resources through reference citations. 

Our primary inclusion criteria were that a tool: 1) was designed as a survey with questions and response options, 2) was publicly available, 3) measured half or more of the PCMH domains (ie, access, team-based care [including continuity of care], population health management, care management and support [including shared decision making and self-care/self-management support], care coordination [including care transitions], and performance measurement and QI), 4) was based on 2014 PCMH standards of care, 5) was designed for QI or evaluation (not for PCMH recognition), 6) focused on the practice (not the health system) as the unit of change, and 7) did not require an external surveyor or site visit but could be administered by practice site administrators or evaluators.  

We identified 20 potential survey tools and abstracted details on each tool’s identifying information, including whether it was designed for recognition (for a national organization like NCQA or state agencies), evaluation, or QI; whether it assessed at the practice level; form (eg, paper or web); who completes the tool; and publications documenting its reliability. We estimated time burden of a tool by counting the number of questions overall and by type (eg, closed- or open-ended questions).

 Once tools not meeting these initial criteria were excluded, we coded each question in the remaining tools by content domain and related NCQA PCMH standard and element.

 We chose the NCQA PCMH standards because they are the most widely used standards, with recent estimates suggesting 15% to 18% of primary care physicians work in a NCQA-recognized PCMH.

 We created a summary worksheet to record the percentage of items in each content domain and the percentage of items related to each of the current (2014) and previous (2011) NCQA PCMH standards and elements. The project was approved by RAND Corporation’s institutional review board (Number: FWA00003425).

 provides inclusion and exclusion information on all 20 identified tools. Of these, we excluded 3 that were not publicly available, 1 that was based on 2008 NCQA PCMH standards,

 4 that were not designed for evaluation, 1 that was not a survey, 3 that focused on the health system rather than the practice, 2 that had limited PCMH content, and 1 that required an external surveyor and site visit. 

We were left with 5 current publicly available tools designed to collect data at the practice level by an administrator or clinical lead for practice improvement, research, evaluation, or quality measurement: Medical Home Index-Long Version (MHI-LV) and -Short Version (MHI-SV), Medical Home Care Coordination Survey—Healthcare Team (MHCCS-H), Primary Care Assessment Tool–Facility Edition (PCAT-FE), and Patient-Centered Medical Home Assessment (PCMH-A) tool. 

shows details of these 5 tools. All except PCAT-FE have a light (less than 10 minutes to complete) or medium (approximately 20 minutes) administrative burden, with zero monetary cost and web-downloaded availability. Based on the Berry

 method of estimation, completion times for the tools range from 8 to 37 minutes (for 36 to 167 items). None require self-reported documentation or verification of responses. For 4 of these tools, a key clinical lead at the practice responds; for the PCAT-FE, a system administrator does so. Both the MHCCS-H and the PCMH-A also recommend that a multidisciplinary team of practice staff (including primary care physicians, physician assistants, nurses, and clinical administrative staff) collectively complete 1 survey per practice. 

Answer format varies by tool, leading to different levels of data actionability. To be actionable, a survey question must outline specific practices or elements that can be identified and changed/improved, link the specific information to a numerical value/score, and cover no more than 1 practice or element.

We judged that the PCMH-A, PCAT-FE, and MHCCS-H are generally actionable for QI, whereas the other 2 tools have actionability limitations. The PCMH-A uses a numerical 1 to 12 rating based on the extent to which recommended practices and behaviors are implemented, divided across 4 levels with specific actions or practices described for each item; it is also sensitive to practice change over a time period as short as 6 months.

 The PCAT-FE and MHCCS-H both have items that are specific to 1 action and align with the response scales, making them fully actionable. The MHCCS-H has only 1 ambiguous item (regarding primary care team roles). By contrast, the MHI-LV and MHI-SV do not help the user determine specific practices to implement. For example, the MHI-LV asks a respondent to rate whether there is “partial” or “complete” implementation of an item with several components of patient- and family-centered care; an answer of “partial” does not allow understanding of which specific component needs further work. Burton et al

 similarly documented these actionability problems with the Medical Home Index (MHI) tool. 

All 5 tools have detailed scoring instructions and include a summary-score calculation. The PCMH-A asks teams to rate their care delivery in 36 key areas associated with the 8 change concepts. Each item is scored from 1 to 12 based on the extent to which recommended practices and behaviors are implemented. The MHI-LV has groups of items that are considered “themes,” with a total of 25 items using an 8-point scale where “partial” mastery of the most basic item equals 1 point and “complete” mastery of the most advanced of the 4 items equals 8 points. The MHI-SV is a 10-item survey that scores a practice on a continuum of care across 3 levels reflecting the degree to which it has achieved components of a medical home, scored on a 1 to 5 scale. Level 1 is “good, responsive primary care”; Level 2 is “pro-active primary care (in addition to Level 1)”; and Level 3 is “primary care at the most comprehensive levels” (in addition to Levels 1 and 2). The reporter is asked to respond to each of the 10 indicators with: Level 1, Level 2—partial, Level 2–complete, Level 3–partial, or Level 3–complete. The MHCCS-H has 8 care coordination domains and a total of 35 items (34 items plus 1 global-rating item), with 5-point categorical-response scales, plus 2 responder-characteristic items. The PCAT-FE is scored by taking the average score (on a 1 to 4 scale) of each domain.

Reliability and validity have been estimated for the MHI-LV

 examined interrater reliability between 2 project staff and between practices and project staff and the internal consistency of MHI domains and themes. The intraclass correlation coefficients between the summary scores of the interviewers was .98 and between the scores of the 2 interviewers and scores of the practices, .97. Standardized coefficient alphas for the 6 domains of the MHI ranged from .81 to .91, and the overall standardized alpha was .96. In the sample of practices studied, the MHI was an internally consistent instrument with acceptable reliability and support for validity for use by pediatric primary care practices to assess PCMH implementation. For the MHCCS-H, Zlateva et al

 conducted analysis to examine internal consistency, reliability and validity of the MHCCS-H using structural equation modeling. All domains had Cronbach’s coefficient alphas above 0.80.

described a longitudinal study of 65 safety-net practices from 5 states participating in a national demonstration program for PCMH transformation that compared 15 practices that achieved PCMH recognition with 29 practices that were actively pursuing NCQA recognition practice (but had not achieved it) and 21 practices that were not pursuing NCQA recognition but that used the PCMH-A. They found that mean overall PCMH-A scores increased overtime, indicating that the PMCH-A was sensitive to change over time (7.2 in March 2010 to 9.1 in September 2012; 

 <.01). Increases were statistically significant for each change concept (

 <.05). Implementers of PCMH change (ie, facilitators) agreed with scores 82% of the time. NCQA-recognized sites (n = 15) had significantly higher PCMH-A scores than sites that were not yet recognized. Sites that had completed more transformation activities and progressed over defined tiers reported significantly higher PCMH-A scores. They concluded that the PCMH-A was sensitive to change over time and provided an accurate reflection of practice transformation.

 shows the emphases given to different content domains in the PMCH tools. We started with the content areas identified by the Burton

 review of PCMH tools and added several newly identified domains. There were 24 content areas identified across the PCMH tools. Content domains receiving the most emphasis were: 1) use of care plan in care management, 2) coordination of care, 3) provide self-care support and community resources, 4) quality improvement, and 5) enhance access and continuity using medical records. Each tool had a main emphasis: PCMH-A targeted continuity of care; MHI-LV, community resources; MHI-SV, coordination of care; and MHCCS-H, the care plan. Some content areas were only covered by 1 tool: comprehensiveness of care items (defined as the breadth of services a practice offers to address any health problem at any given stage of patient’s life [eg, the primary care practice/health center has behavior change interventions readily available for patients as part of routine care, using a 5-point agreement scale]) by MHCCS-H; empanelment, evidence-based care, quality measurement, and quality reporting by PCMH-A; and standard non—PCMH care provided (used as a base level to PCMH care for patient and family involvement) and adherence to current laws by MHI-LV. 

Our review mapped PCMH tools to both the NCQA 2011 and 2014 standards. 

 shows the relative content emphases for each of the 2011 NCQA PCMH standards and elements. The 2011 NCQA PCMH standards receiving the most emphasis across tools were: a) Standard 4: Provide Self-Care and Community Support; b) Standard 3: Plan and Manage Care; c) Standard 1: Enhance Access and Continuity; d) Standard 5: Track and Coordinate Care; e) Standard 6: Measure and Improve Performance; and f) Standard 2: Identify and Manage Patient Populations. 

Although many standards and elements remained the same from the 2011 to the 2014 NCQA PCMH standards, there are new areas of focus. The 2014 version focuses more on care management. In previous standards, practices needed to identify high-risk patients; 2014 requires also focusing on patients who would benefit from care management. NCQA 2014 includes a new standard on team-based care, giving more emphasis to providing patients with information—proactively orienting them to the practice, helping them choose a physician, and explaining the practice’s use of evidence-based guidelines. This team-based care element not only ensures that the whole care team works at the top of their licensure (included in 2011), but also improves the focus on the individual patient. 

The population health standard in 2014 incorporates many Stage 2 Core and Menu Meaningful Use requirements, use of electronic medical records, and a comprehensive health assessment. This is not a new NCQA standard, but the documentation requirements have changed. For the 10 factors, NCQA requires the practice either has a report or conducts chart audits using NCQA’s tool, the Record Review Workbook. Practices had been allowed to show examples of their comprehensive health assessments. Behavioral health has been integrated into numerous standards, where it had previously not been a focus. 

In 2011, NCQA wanted practices to develop the habit of measuring performance. As a result, the 2014 standards identify more areas for improving practice performance. Additionally, in the 2014 standards, NCQA dropped the 2011 element 4B (Provide Referrals to Community Resources); the 2011 element 6F (Report Data Externally) was incorporated into the 2014 element 6F (Report Performance); and the 2011 element 6A (Measure Performance) was divided into 2 separate elements: 2014 element 6A (Measure Clinical Quality Performance) and element 6B (Measure Resource Use and Care Coordination). NCQA has published a full crosswalk of 2011 to 2014 standards.

characterizes how well each assessment covers the 2014 PCMH standards. Those receiving the greatest emphasis across tools were: a) Standard 4: Care Management and Support; b) Standard 2: Team-Based Care; c) Standard 3: Population Health Management; d) Standard 5: Care Coordination and Care Transitions; e) Standard 6: Performance Improvement and Quality Improvement; and f) Standard 1: Patient-Centered Access. The PCMH-A had the broadest array of practice capabilities assessed (Table 3) and contained items pertaining to all 6 2014 NCQA PCMH standards and 27 elements (Table 5). The MHI-LV, MHI-SV, and MHCCS-H lacked items addressing the “measurement and improvement of performance” (PCMH Standard 6). The MHCCS-H lacked items addressing “access to care” during office hours and those addressing after-hours or electronic access within “enhance access and continuity” (PCMH Standard 1).

The differences in the 2011 and 2014 standards did not influence our content coding or coverage of content to PMCH standards, except for behavioral health, which was new to the 2014 standards. Items related to behavioral health were included in the PCMH-A and PCAT-FE, but not in the MHI or MHCCS-H. 

We searched for associations of the tools to other quality measures. We found studies using the PCMH-A to track PCMH transformation,

 but no articles specifically tying PCMH-A results to other quality measures. This may be because there has been insufficient time since its development for these studies to be performed. For the MHI-LV, we found limited and mixed results from 3 studies

; 2 showed associations with fewer hospitalizations and lower nonurgent preventable emergency department (ED) use by children and 1 showed associations with higher chronic care management subscores to better provider communication and follow-up on test results but worse access to care among adults. Another study

 found higher levels of “medical homeness” (based on the MHI) associated with lower nonurgent, preventable, or avoidable ED use by publicly insured children. To date, there is little evidence showing which PCMH recognition tool(s) may be associated with improved quality outcomes.

Many healthcare organizations and payers are investing in a PCMH tool for QI or evaluation of medical homeness and progress toward PCMH standards. With little evidence about which PCMH-recognition tool is associated with improved quality outcomes, payers and practices must decide what role quality measurement should play in their PCMH journey, how much administrative burden to place on practices to collect data, what data to use for QI related to PCMH, and how to track practice-level progress toward PCMH transformation. 

We found that the MHI-LV, PCMH-A, and MHCCS-H tools have been subjected to relatively more evaluation and validation and impose only a light or medium administrative burden to complete. The PCMH-A, PCAT-FE, and MHCCS-H (except for 1 of its 37 items) are actionable for quality improvement, whereas the 2 other tools have limitations in their ability to identify specific improvements. The PCMH-A is documented

 to be sensitive to changes in practice behavior over a time period as short as 6 months, indicating it can track QI progress and a practice’s trajectory of PCMH change. All 5 tools have detailed scoring instructions, including a summary-score calculation.

No single tool covers all relevant PCMH content, and some content is only found on 1 tool. Practice capabilities in coordination of care, care plans, and support of self-management and community resources receive the greatest emphasis. None of the reviewed tools covers all 4 main attributes of primary care: comprehensiveness of care, coordination of care, continuity, and access to care.

Because there is no evidence on what combination of practice capabilities produces the best outcomes for patients, practices and payers must decide which medical home aspects they most wish to measure and whether an available tool contains those specific practice capabilities. They must weigh these content decisions against the administrative burden they impose on physicians and clinical staff.

The decision on content includes not only practice capabilities and content areas covered in the PMCH tool but also whether the tool aligns with recognition standards and criteria a practice is pursuing. The NCQA standards are the most widely followed PCMH recognition standards. The PCMH-A had the broadest array of practice capabilities assessed and contains items pertaining to all 6 of the 2011 NCQA PCMH standards/elements and all 6 of the 2014 standards/elements. The MHI-LV and the MHCCS-H lacked items addressing the measurement and improvement of performance (PCMH Standard 6). The MHCCS-H also lacked items addressing access to care during office hours and after hours or electronic access within “enhance access and continuity (2011)/Patient-centered access (2014)” (PCMH Standard 1). Altogether, the PCMH-A could be the most widely applicable tool for most practices transforming into PCMHs, given its alignment with NCQA standards and broad content. As noted previously, the findings of this study are applicable to practices using the NCQA standards, which are the most widely used; practices using other sets of standards may see slight differences in content alignment.  

The decision on which tool to administer for tracking PCMH transformation must weigh administrative burden (ie, cost, time, need for documentation), reliability and validity of data collected, actionability of data, practice capabilities measured, and alignment with PCMH recognition standards. The PCMH-A and MHCCS-H have a light administrative burden and are actionable for quality-improvement purposes. The MHCCS-H has data available on its reliability, whereas the PCMH-A has shown sensitivity to changes over time in practices that have achieved (or have not achieved) NCQA PCMH recognition. Each of the tools we reviewed has a different constellation of practice capabilities and emphases. The PCMH-A assesses the broadest array of capabilities, whereas the MHCCS-H is the only tool with items on capability to provide comprehensiveness of care. 

RAND Corporation (DDQ), Santa Monica, CA; RAND Corporation (ZSP), Boston, MA; Division of General Internal Medicine and Health Services Research, University of California, Los Angeles (RDH), Los Angeles, CA.

This work was supported by a cooperative agreement from the Agency for Healthcare Research and Quality (AHRQ) [Contract number U18 HS016980].

 The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Concept and design (DDQ); acquisition of data (DDQ, ZSP, RDH); analysis and interpretation of data (DDQ, ZSP); drafting of the manuscript (DDQ, ZSP); critical revision of the manuscript for important intellectual content (DDQ, ZSP, RDH); obtaining funding (RDH); and administrative, technical, or logistic support (DDQ, ZSP).

Denise D. Quigley, PhD, RAND Corporation, 1776 Main St, Santa Monica, CA 90407-2138. E-mail: quigley@rand.org.

1. Bitton A, Martin C, Landon BE. A nationwide survey of patient centered medical home demonstration projects. 

. 2010;25(6):584-592. doi: 10.1007/s11606-010-1262-8.

Dentzer S. Reinventing primary care: a task that is far ‘too important to fail’. 

. 2010;29(5):757. doi: 10.1377/hlthaff.2010.0410.

Bodenheimer T, Lorig K, Holman H, Grumbach K. Patient self-management of chronic disease in primary care. 

Ferrer RL, Hambidge SJ, Maly RC. The essential role of generalists in health care systems. 

Grumbach K, Bodenheimer T. A primary care home for Americans: putting the house in order. 

We reviewed operational details and content of tools designed to evaluate patient-centered medical home (PCMH) transformation. These tools assist practice leaders in understanding specific information about the process and progress of becoming a PCMH.

Denise D. Quigley, PhD

Articles

Patient-Reported Denials, Appeals, and Complaints: Associations With Overall Plan Ratings

Tools to Gauge Progress During Patient-Centered Medical Home Transformation