Asaf Bitton, MD, MPH

 As US healthcare spending increases, insurers are focusing attention on decreasing potentially avoidable specialist care. Little recent research has assessed whether the design of modern health maintenance organization (HMO) insurance is associated with lower utilization of outpatient specialty care versus less restrictive preferred provider organization (PPO) plans.

 Observational study of Massachusetts residents aged 21 to 64 years with any HMO or PPO insurance coverage from 2010 to 2013.

 We examined rates and patterns of primary care visits, new specialist visits, and specialist spending among HMO versus PPO enrollees. We estimated multivariable regression models for each outcome, adjusting for patient and insurance characteristics.

 From 2010 to 2013, 546,397 and 295,427 individuals had continuous HMO or PPO coverage, respectively. HMO patients had fewer annual new specialist visits per member versus PPO patients (unadjusted, 0.37 vs 0.43), a difference after adjustment of 0.05 annual visits, or a 12% relative decrease among HMO members (

 <.001). These visits were more likely to be with a specialist in the same health system as the patient’s primary care physician (44.9% vs 40.7%; adjusted difference, 2.8 percentage points; 

 <.001). Mean annual spending on new specialist visits and subsequent follow-up per member was lower in HMO versus PPO patients (unadjusted, $104.10 vs $128.10), translating to 12% lower annual spending (adjusted difference, —$16.26; 

 Having HMO insurance was associated with lower rates of new specialist visits and lower spending on specialist visits, and these visits were less likely to occur across multiple health systems. The impact of this change on overall spending and clinical outcomes remains unknown.

Is specialist “gatekeeping” in modern health maintenance organization (HMO) insurance associated with differences in outpatient patterns of care?

Compared with preferred provider organization insurance, HMO insurance was associated with lower rates of new specialist visits.

These visits were less likely to occur across multiple health systems.

HMO insurance was also associated with 12% lower specialist visit spending, which was largely driven by lower use, not lower price.

HMO gatekeeping may meaningfully reduce specialist utilization, although the impact of this change on overall spending and clinical outcomes remains unknown.The majority of private health insurance options in the United States fall into 1 of 2 broad categories: preferred provider organization (PPO) plans and health maintenance organization (HMO) plans. PPO plans generally have broadly inclusive physician networks with little barrier to self-referral, whereas lower-premium HMO plans generally have less inclusive networks and rely on provider-facing managed care strategies, such as requiring primary care physicians (PCPs) to approve referrals to access specialists (ie, “gatekeeping”).

 As of 2014, across the United States, HMO insurance products accounted for 24% of enrollment in the employer-sponsored insurance market, whereas PPO products accounted for nearly 50% of the market.

 In Massachusetts, the setting for this analysis, HMO insurance was 39% of the commercial market, with an additional 19% with point-of-service (POS) coverage and 37% with PPO coverage in 2016.

 One concept underpinning the provider-facing strategies used by HMOs is that PCPs can reduce the use of low-value specialty care and ensuing downstream utilization by either treating a condition in the primary care setting or directing specialty referrals to higher-value providers within their health system. This is in contrast to the consumer-facing cost-control strategies seen in PPO plans that may impose higher cost sharing but retain open provider choice and access at the discretion of the enrollee.

In general, HMO plans require enrollees to identify a PCP to help direct their downstream utilization. Conceptually and empirically, a single PCP with overall responsibility for a patient’s care could lead to better-coordinated care while reducing potentially avoidable outpatient specialist referrals.

 Such gatekeeping arrangements, however, can also create problems for both patients and PCPs when disagreements arise over whether a specialist is needed.

The extent to which such gatekeeping affects specialist utilization or costs has mixed results in studies that are largely more than 15 years old.

 Little recent research has assessed whether the design of modern HMO insurance is associated with lower utilization of outpatient specialty care compared with modern PPO plans. Moreover, broadly understanding the effect of insurance design on specialty utilization across a large heterogeneous payer market is an important policy-relevant question recently made feasible through the availability of statewide all-payer claims databases (APCDs).

We used the Massachusetts APCD to examine the association between insurance design and outpatient care utilization patterns in multiple HMO and PPO arrangements. In Massachusetts, the primary distinction between HMO and PPO plans relates to gatekeeping requirements (ie, a referral requires PCP approval), in contrast to the existence of “closed-model” HMOs like Kaiser Permanente (ie, regardless of PCP approval, referrals outside the internal network are largely unavailable), which do not currently have market share in the state. We hypothesized that HMO enrollment would be associated with more PCP visits and fewer new specialist visits, with a larger share of specialty care received within the PCP’s health system.

The Massachusetts APCD contains detailed data on healthcare utilization, insurance eligibility, and provider credentialing across all commercial payers and public health insurance programs in Massachusetts, representing approximately 90% of the nonelderly population in Massachusetts.

Our study cohort included all Massachusetts residents aged 21 to 64 years who had 4 calendar years of continuous enrollment with any commercial HMO or PPO product from an APCD-participating commercial insurance provider from 2010 to 2013; public insurance was not included. We focused on members with this length of continuous enrollment because we were interested in the long-term association between HMO or PPO membership and specialist use among patients with stable benefit design. Enrollees may change from an HMO to a PPO plan or vice versa because of their plans for specialist use, which we wanted to reduce as a source of bias. We did not include children given the unique characteristics of the pediatric healthcare market in the state. The key exposure of interest was whether an individual was a continuous member of an HMO versus a PPO product, as defined by the insurer submitting data in the APCD member eligibility file. These designations are used by the insurers themselves to distinguish HMO from PPO plans for administering health plan benefits and requirements. We defined an HMO product as any product designated as HMO or POS (9.6% of all members, or 15.2% of HMO members in the final study sample), which have very similar benefit designs among the large insurers in Massachusetts.

 PPO products were defined as those with a designation of PPO or exclusive provider organization (EPO; 1.5% of all members, or 4.3% of PPO members in the final study sample), which have similar flexibility to PPO plans.

 We used diagnoses from all outpatient and inpatient claims in 2010 to measure baseline comorbidities for risk adjustment. We then performed cross-sectional analyses that focused on the period from 2011 to 2013. We defined continuous enrollment as 11 or more months of insurance coverage in a year, such that those with a short period of discontinuous enrollment, often due to an administrative error, were not excluded from the analyses. This project was approved by the Committee on Human Subjects at Harvard Medical School.

To attribute patients to PCPs, we assigned enrollees to the PCP with the plurality of office visits in a calendar year (“empirical” PCP assignment).

 Nonphysician primary care providers, such as nurse practitioners or physician assistants, were not included because they were not consistently identified in provider classifications provided by insurers. We identified visits to primary care based on claim lines with evaluation and management (E&M) Current Procedural Terminology (CPT) codes to physicians classified as general internal medicine, general practice, or family medicine. If there were no primary care visits, we assigned enrollees to the non—primary care office-based specialist (eg, excluding radiology, pathology) with the plurality of visits. Any ties between physicians were broken randomly, and enrollees with no visits were not assigned to a PCP or another physician. We also examined patterns of PCP assignment as recorded by insurers, which we labeled “insurance” PCP assignment.

We also assigned physicians to larger physician organizations or health systems in Massachusetts using a publicly available database of Massachusetts hospital and physician group national provider identifiers,

 grouped into physician organizations using the affiliations defined by the Massachusetts Health Care Delivery System Map, a manually curated listing of hospital and clinic affiliations within larger provider organizations (eg, which satellite clinics are part of Partners HealthCare) published by the Blue Cross Blue Shield of Massachusetts Foundation.

 We assigned 76% of physicians to organizations using this approach, and the remainder were largely unaffiliated independent physicians.

We assessed several dimensions of outpatient care delivery potentially influenced by type of insurance coverage. For primary care utilization, we calculated the average annual number of PCP visits of any kind per member from 2011 to 2013 and the proportion of patients with no PCP visits in a year. For primary care assignment, we assessed the proportion of patients ever assigned a PCP by their insurer and the percentage of all PCP visits occurring with those providers, as well as the consistency of empirically attributed and insurer-defined PCP assignment measured by the number of assigned PCPs per member over the 3-year study period.

We defined a new specialist visit as any claim containing a new visit E&M CPT code with a non-PCP, which can only be billed if a patient has not been seen by any physician within that practice in the previous 3 years.

 We then measured characteristics of new specialist visits that could be associated with higher likelihood of PCP-initiated referral, as opposed to patient self-referral. We did not attempt to identify specialist-to-specialist referrals because HMO referral policy focuses on PCPs as the key provider for approving referrals. We examined whether a new specialist visit’s claim designated a referring physician and whether that referring physician was the patient’s assigned PCP. We also measured the time in days between a new specialist visit and the most recent PCP visit for the patient in the past 365 days. When we had information on organizational affiliation for both the PCP and the specialist (“health system match” was available for 59% of new specialist visits), we assessed whether the PCP for the most recent visit or the assigned PCP was affiliated with the same provider organization as the specialist for the new visit.

We measured the percentage of patients with any new specialist visit from 2011 to 2013 and the average number of new specialist visits per patient in that period. Because HMO gatekeeping may have its most pronounced effect on the highest utilizers, we also examined the proportion of patients seeing 3 or more new specialists or seeing new specialists in 2 or more provider organizations, which we identified as approximately the top decile of utilization for the entire study population.

Lastly, we measured the average cost per new specialist visit and all subsequent visits with that specialist over the subsequent year by summing all costs (including insurer reimbursement and any out-of-pocket spending) from all E&M claims for new specialist visits for members, as well as E&M costs from any downstream established office visits with the same specialist seen in the following year. To test whether differences in spending by insurance type were driven by differences in price versus quantity of specialist visits, we standardized specialist visit prices using the average HMO cost per evaluation and management CPT code as a standard price for all encounters.

To measure comorbidities, we calculated Hierarchical Condition Category (HCC) scores from all diagnosis codes in the baseline year using software available from CMS. The HCC score represents the predicted risk of healthcare spending beyond average in the next year; a score of 1.0 corresponds to a risk profile for a Medicare enrollee with average expected annual spending.

 To derive a proxy measure of socioeconomic status, we linked individuals’ 5-digit zip code, the most granular geographic unit available, to the publicly available area deprivation index (ADI), a measure derived using a weighted combination of 17 Census-level indicators of socioeconomic disadvantage.

 We also included the median co-pays for primary care and specialist physician visits for each separate insurance product in 2010 as a measure of plan generosity. Other variables included member age as of December 31, 2010; sex; fixed effects for insurance carrier (eg, Blue Cross Blue Shield vs UnitedHealthcare); insurance plan risk type (fully insured vs self-insured); and employer size (ranging from individual to ≥500 employees; see 

We compared the characteristics of HMO and PPO patients and the unadjusted rates of the outcomes using χ

 tests. To estimate adjusted comparisons between HMO and PPO patients, we fitted a series of multivariable regression models for each outcome, with the key explanatory variable being an indicator for whether a patient was a member of an HMO versus a PPO insurance product. We fitted models with different link functions depending on the outcome: linear models for visit counts, costs, and visit intervals; Poisson models for counts of unique physicians or health systems; and logistic models for all other outcomes, which were binary or fractional in nature. All models adjusted for patient age, sex, HCC score, ADI, median primary care and specialist co-pays in that plan (with the exception of spending outcomes, which excluded co-pay covariates), and employer size. We also included a fixed effect for insurance carrier to control for unmeasured characteristics of carriers’ networks or policies that might also influence the outcomes of interest. As a sensitivity analysis, we also estimated logistic regression models predicting the propensity to have HMO versus PPO insurance and then used propensity score weighting to more robustly address imbalance on observable variables between HMO and PPO patients and replicated all of our model results (see 

Because of the mix of functional forms (eg, linear, Poisson, logistic), we used bootstrap methods within each model to present the effects of HMO membership as the adjusted difference in each individual outcome between the average HMO member and the average PPO member. For each effect, we drew 1000 simulations of the set of regression coefficients using the asymptotic multivariate normal approximation to the log-likelihood function, and we present the mean and 95% CI from these simulations.

 Analyses were performed in R version 3.1.3 and SAS version 9.4 (SAS Institute; Cary, North Carolina). The 95% CI around reported estimates reflects 0.025 in each tail, or 

The study sample included 546,397 and 295,427 individuals with 4 years of continuous coverage in an HMO or PPO insurance product, respectively, from 2010 to 2013. HMO enrollees were younger than PPO patients (45.7 vs 46.1 years, respectively; 

 <.001), had slightly lower HCC scores (0.34 vs 0.36; 

 = .01), had higher ADI scores (ie, were more socioeconomically disadvantaged; 83.8 vs 80.4; 

 <.001), and were significantly less likely to be employed by an employer with 500 or more employees (51.7% vs 81.1%; 

 <.001; Table 1). Sample characteristics after propensity score weighting were identically balanced (

HMO patients had fewer PCP visits annually than PPO patients (2.31 vs 2.45, respectively), but there was only a marginal difference after adjustment (adjusted difference, —0.02; 

). As expected, HMO patients were significantly more likely to ever have a PCP assigned by their insurer (90.4% vs 39.7%; adjusted difference, 46.2 percentage points; 

 <.001) and had a higher proportion of visits with those PCPs than PPO patients did with their designated PCPs (57.0% vs 19.0%; adjusted difference, 38.3 percentage points; 

Over the 3-year period, 56.2% of HMO patients had any new specialty visit versus 62.4% of PPO patients (adjusted difference, —5.3 percentage points; 

), with HMO patients having 14% fewer annual new specialist visits per member (0.37 vs 0.43; adjusted difference, —0.05; 

 <.001). Consistent with this lower utilization, fewer HMO patients were likely to have seen 3 or more specialists from 2011 to 2013 (13.3% vs 16.2%; adjusted difference, —2.5 percentage points; 

 <.001) or specialists in 2 or more health systems (21.8% vs 24.3%; adjusted difference, —2.2 percentage points; 

The interval between the most recent PCP visit and a new specialist visit was 7.7 fewer days for HMO versus PPO patients (

). A lower proportion of PPO patients had no PCP visits in the 60 days preceding a new specialist visit (55.0% vs 49.4%; adjusted difference, 5.3 percentage points; 

 <.001). HMO patients were more likely to have a referring physician for a new specialty visit (53.3% vs 24.5%; adjusted difference, 36.7 percentage points; 

 <.001) and had a modestly higher rate of health system match (44.9% vs 40.5%; adjusted difference, 2.8 percentage points; 

The average total cost per individual new specialty visit was $12.29 lower for HMO versus PPO patients, a relative difference of 6.1% (mean cost, $189.05 vs $201.34; 

). On average, office visit co-pays for both primary care and specialist visits were higher for HMO patients than PPO patients ($14.83 vs $13.48 for primary care; $20.78 vs $19.55 for specialists; 

 <.001 for both). Mean annual spending on new specialist visits plus subsequent downstream visits per member was also lower in HMO versus PPO patients ($104.06 vs $128.14; adjusted difference, —$16.26; 

 <.001) (Table 5), even after using a standardized price ($104.10 vs $122.98; adjusted difference, —$11.74; 

 <.001) (Table 5), suggesting that most of the difference in spending was related to quantity of new specialist visits and not to price.

There were minimal differences in our estimates for these models using propensity score weighting to balance the observable characteristics of HMO- versus PPO-enrolled individuals (

Using statewide data from Massachusetts, we found that HMO insurance was associated with modestly lower rates of new specialist visits and lower specialist visit costs compared with PPO insurance. New specialty visits within HMOs also were more likely to have closely followed a PCP visit, included a referral from a PCP in the claim, and been within the PCP’s health system. Because this study was observational, we cannot interpret our conclusions as causal, and unobserved differences between the HMO and PPO populations may explain some of the effects we observed. However, these results are consistent with the concept that modest provider-facing requirements of HMO coverage may have the effect of PCPs for HMO patients taking a more active role in managing referrals.

Given the emphasis on gatekeeping within the HMO model design and policies, the relatively modest effect size on specialty utilization is notable. There may be few consequences for PCPs not actively coordinating referrals for HMO patients, such that they might simply have administratively approved all patient-initiated referrals without trying to influence them. Consistent with this fact, we found that the referring physician matched the assigned PCP only 30.8% of the time for HMO patients, despite the fact that 90% of HMO enrollees were assigned a PCP by their insurer. This suggests that current HMO policies may not be implemented consistently to maximize referral coordination. Supporting this observation, we find contrasting patterns of use between patients’ insurance-assigned versus empirically assigned PCPs, with 57% of HMO patients’ PCP visits occurring with insurance-assigned PCPs. It is not clear whether this proportion is desirable from the insurer’s perspective or too low. However, the match between patients’ assigned PCP and the PCP actually seen could be an important determinant of the effectiveness of HMO design.

The modest effect size we find is consistent with the “norms hypothesis,” which postulates that physicians have a relatively uniform approach to care consistent with overall financial incentives as opposed to the payment arrangement for an individual patient they are treating.

 As accountable care organization (ACO) and other risk-based contracts become more prevalent, and as physician organizations take a more active role in managing new specialty use, we will likely see more pressure on PCPs to actively coordinate and approve specialty care (especially within ACO networks to which they belong) in the future.

We also found that reduced spending on new specialist visits was driven largely by differences in utilization, rather than price.

 Although HMOs frequently negotiate lower prices in return for a more restricted network, our results show that the bulk of spending differences was driven by lower utilization of specialty care in HMOs. This strategy, likely influenced by PCP decision making, is a strong contrast to consumer-facing approaches, such as high deductibles, reference pricing, or price transparency, that aim to achieve reduced spending by increasing consumer sensitivity to prices, rather than reducing utilization overall. It is possible that a combination of consumer- and provider-facing approaches could be more effective at reducing medical spending than either alone.

Our findings are consistent with those of some prior research on the association of gatekeeping with specialist utilization. In 1979, a randomized controlled trial of a gatekeeping benefit design with more than 2000 participants found decreased rates of all specialist visits among patients with a gatekeeping arrangement versus those without.

 In contrast, a 1998 natural experiment in a large multispecialty physician group where gatekeeping arrangements were eliminated found no change in total specialist visits, although rates of new specialist visits increased modestly after gatekeeping elimination.

 In that study, however, the group also had constrained the availability of specialists within its system, which might have served to limit new specialist visits. Both of these studies are now decades old, making them less applicable to the current healthcare landscape.

This study has several important limitations. First, this is a cross-sectional and observational analysis, so our results cannot be interpreted as causal. We addressed this limitation in part by using propensity score weighting to address the impact of observable patient characteristics on the outcomes of interest. Second, our sample is restricted to Massachusetts residents, which may not generalize to other states, although HMO and PPO insurance are the most commonly used types of plans throughout the country. In addition, we relied on insurers’ designations of HMO versus PPO plan types to identify exposure to PCP gatekeeping patterns. We also combined similar plan types, like POS and EPOs, into these categorizations. This coarse categorization may misclassify the level of PCP control over referrals in a plan because we can only observe the HMO/PPO designation. Third, we are unable to capture potential mechanisms that providers or patients might use to mitigate the influence of gatekeeping tactics, such as automated approval processes or delegation of referral approvals to nonclinical administrative staff, which could bias our findings toward the null. Fourth, our ability to capture socioeconomic status, an important confounder, is limited by relatively coarse data on 5-digit zip code alone. Fifth, our cross-sectional analysis assumes that any changes in benefit design over time were similar between HMO and PPO plans. Due to missing data in the APCD, we were unable to confirm this assumption, which may bias our results. Sixth, we did not have access to accurate and complete data on some aspects of cost sharing, such as deductibles, which could have influenced our findings. Lastly, we were limited in our ability to capture the appropriateness of a referral using claims data alone, which lack the detailed clinical information necessary to assess a referral’s clinical necessity.

We found that HMO coverage was associated with similar PCP utilization but reduced specialist utilization compared with PPO coverage. To our knowledge, this analysis is among the first examinations of HMO benefit design in the current era across a large multipayer population. Not only was specialist utilization reduced, but the referrals that did happen among HMO patients appeared to be more physician-directed and coordinated within health systems. It is not clear whether this pattern is associated with improved outcomes, patient satisfaction, or population health, a set of questions that deserve further research. However, in light of the growing use of benefit designs such as high-deductible health plans that shift decision making to patients who might be poorly equipped to make treatment choices, modern implementation of HMO methods may present an alternative provider-facing strategy for cost control worth revisiting. However, care must be taken to avoid previous errors in overly onerous gatekeeping designs that were not sensitive to patient needs and appropriate clinical decision making.

 Department of Health Policy and Management, Harvard T.H. Chan School of Public Health (MLB), Boston, MA; Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women’s Hospital (MLB, AB), Boston, MA; Department of Medicine, Massachusetts General Hospital (ZS), Boston, MA; Department of Health Care Policy, Harvard Medical School (ZS, AB, SR, BEL), Boston, MA; Ariadne Labs (AB), Boston, MA; Division of General Medicine and Primary Care, Beth Israel Deaconess Medical Center (BEL), Boston, MA.

 Supported by a grant from the National Institute for Health Care Management.

Articles

Is specialist “gatekeeping” in modern health maintenance organization (HMO) insurance associated with differences in outpatient care? The study finds that HMO gatekeeping may meaningfully reduce specialist utilization.

Gatekeeping and Patterns of Outpatient Care Post Healthcare Reform

Using Behavioral Risk Factor Surveillance System (BRFSS) telephone survey data, we evaluated whether individuals who reported access to enhanced primary care features experienced improved quality of care.

We assessed a sample of 16,731 Massachusetts residents contacted by telephone using random-digit dialing, to complete the BRFSS in 2008. A randomized subset of 5693 respondents received an additional 5-question enhanced primary care assessment module. We defined an “enhanced” primary care group as those who reported having a regular, personal provider and responded that their provider “always” or “almost always” 1) had knowledge of their medical history, 2) gave them an appointment right away when necessary, 3) was up-to-date in their knowledge of the patient’s specialist care, and 4) asked them about all of their medications. Multivariable logistic regression was used to compare the “enhanced” versus “usual care” groups and assess several quality process measures.

Nearly one-third of participants endorsed having indicators of enhanced care, and this group reported significantly higher rates of diabetes process measures (56% vs 38%), cholesterol screening (89% vs 81%), influenza vaccination (57% vs 49%), pneumonia vaccination (51% vs 43%), and lower cost and/or access barriers to care (22% vs 33%).

Enhanced primary care was associated with improved self-reported quality outcomes in a statewide telephone survey. A brief, 5-question module provided a novel population measure of access to enhanced primary care. This is a scalable option for other states hoping to characterize their own primary care improvement efforts through the patient-centered medical home model.

	We evaluated Behavioral Risk Factor Surveillance System telephone survey data from Massachusetts residents that included a patient-centered medical home (PCMH) assessment module. 

		Patient-reported indicators of the PCMH correlated with improved quality process measures in diabetes, cholesterol screening, vaccination, and access to care.   

		A brief, 5-question module provided a novel population measure of access to enhanced primary care. 

		This may be a scalable option for other states hoping to characterize their own primary care usage and utility as health insurance expansion continues nationally.

Massachusetts was one of the leaders in insurance expansion with the 2006 Massachusetts Mandated Health Insurance Law.

 Recent reports have cited positive outcomes from the Massachusetts reform effort, such as increases in the total number of Massachusetts residents insured, growing rates of employer-based insurance coverage, strong public and physician support, reductions in disparities in access based on race or ethnicity, increasing amounts of preventive services, fewer low-income or chronically ill adults reporting unmet medical needs,

 and even lower mortality rate among those getting insurance.

 However, insurance reform only addresses a small part of the deficiencies in the larger healthcare delivery system. Patient-centered medical homes (PCMHs) are generating significant state and national attention as one solution in what is generally a costly and ineffective system plagued with disparities in health outcomes and significant barriers to high-quality, integrated medical care.

Most PCMH assessment surveys are based on individual practice or demonstration reports, and such surveys have been at the forefront of reports on the success of the PCMH model.

 Although such demonstrations are a critical part of the evaluation process of new care models, they are context-specific and take time to generate evaluable data. Another approach to evaluate the effect of the PCMH-associated components is to use publically available survey data and parsimonious definitions of what factors compose a PCMH. One such evaluation was performed on the 2007 to 2008 National Survey of Children’s Health, and it identified improvements in 6 of 10 quality measures among children in a PCMH, as defined by a 5-component measure.

 Such patient care experience surveys are often unavailable, however, and to date, there is no state-level evidence on the availability of PCMH components in primary care practices or their association with quality of care.

The Behavioral Risk Factor Surveillance System (BRFSS) was developed by the CDC as a random-digit-dial landline telephone survey.

 The CDC collects data monthly, using this survey in all states and territories of the United States. In order to evaluate the availability of primary care providers, PCMH components in primary care, and sociodemographic predictors of the preceding variables in Massachusetts, a 5-question module was added to the BRFSS for a subset of responders. Using the data provided by this module, we then evaluated the association of PCMH components with quality of care.

In preparation for statewide initiatives in medical homes, one of the authors, who was a member of the Massachusetts Executive Office of Health and Human Services, worked with experts in PCMHs to develop a 5-question short form survey module (

). The questions were adapted from work on the Primary Care Assessment Survey (PCAS) by Safran et al.

 Officials from the Massachusetts Department of Public Health, which administers the survey in the state, gave permission for the module to be piloted in a subset of participants in the 2009 Massachusetts BRFSS (MA-BRFSS) survey. 

The MA-BRFSS survey was completed in 3 waves, and the experimental module was used in the second wave. We used descriptive statistics to characterize the subset of responders and a 

 test to look for statistical variance between the experimental module—wave 2&mdash;and the other survey waves. We then evaluated whether the responder had a primary care physician according to their survey results and used a multivariable logistic regression to test for associated subject characteristics between the subset of responders with a primary care physician and the subset without.

To determine whether having an enhanced primary care experience was related to completion of healthcare process measures, we divided responders into 2 groups. One group, termed the “enhanced primary care” group, endorsed having a primary care physician and responded that their provider “always” or “almost always” 1) had a knowledge of their medical history, 2) gave them an appointment right away when necessary, 3) was up-to-date in their knowledge of the patient’s specialist care, and 4) asked them about all of their medications. A multivariable logistic regression was used to test for associated subject characteristics between the “enhanced primary care” group and the “usual care” group without these PCMH attributes. A 

test was used to characterize the 2 groups.

We performed a multivariable logistic regression analysis to assess differences between the enhanced and usual care groups in several quality measures. We selected these indicators because they are common measures of clinical quality in the management of chronic disease and preventive care. Quality measures included diabetes annual care processes (which includes an annual visit, eye exam, foot exam, cholesterol screening, and hemoglobin A1C testing); lipid screenings for patients with cardiovascular disease, diabetes, and hyperlipidemia; annual influenza vaccination in eligible patients (all patients aged ≥6 months)

; and pneumonia vaccinations in eligible patients (all adults ≥65 years and adults aged 19-64 years who smoke, have chronic pulmonary disease, or are immunodeficient).

An additional binary quality measure assessing cost barriers and access to care was created based on the responses to the following: “Was there a time during the last 12 months when you needed to see a doctor but could not due to the cost?” and “About how long has it been since you last visited a doctor for a routine checkup?” Subjects who responded that they did forgo a visit to the doctor due to cost in the past year or that they had not had a checkup within 1 year were given a positive value on the “cost/access” variable; subjects who did not meet one or both criteria were not given a positive value. The regressions were controlled for age, sex, income, education, and insurance status as covariates.

This study was reviewed by the Institutional Review Board at both the Massachusetts Department of Public Health and the CDC, and was considered exempt due to being a low-risk study without any patient identifiers.

Of the 16,731 respondents to the MA-BRFSS in 2009, 5693 respondents were randomized to receive the short form survey module. 

 summarizes subject characteristics for the respondents to the short form survey module (wave 2) compared with other respondents (waves 1 and 3). There were no significant demographic differences between the 2 groups, confirming that the random sampling for the survey module was successful.

Among participants, 29.8% answered “always” or “almost always” to all 5 PCMH module questions, indicating that they had an enhanced primary care experience. Subjects who were aged 50 to 74 years (

 <.001) were more likely to report having an enhanced primary care experience; subjects who were either uninsured (

 <.001) or were Asian, Native Hawaiian or Pacific Islander, American-Indian, multiracial, or selected “other” for their racial affiliation (

A multivariate logistic regression adjusted for age, sex, income, education, and insurance status found significant differences between the enhanced and usual primary care groups in all quality measures tested. Subjects receiving enhanced care were more likely to receive diabetes care

 = .002). Subjects in this enhanced group were also less likely to report experiencing barriers to access to care, measured by whether they reported an instance in the past 12 months where they needed to see a doctor but could not due to cost and by whether more than 1 year had elapsed since their last checkup (

Almost one-third of residents reported having access to a primary care practice with enhanced primary care attributes. Importantly, having enhanced primary care was associated with better performance on preventive, access, and chronic disease measures. Our findings have favorable implications for the wider adoption of the PCMH model—the main mode of primary care improvement and transformation nationwide in the United States. Our findings also contribute to a growing body of research showing that the PCMH-enabled primary care transformation may result in modest increases in quality measures early on by improving care processes, particularly in the area of prevention.

For example, a recent prospective multi-payer study in Hudson Valley, New York, found that PCMH practices improved significantly more (from 1% to 9%) on 4 of 10 quality measures than practices with electronic health records and paper records.

 In a single private payer demonstration project in New Jersey in 2011 with 8 medical home practices, mammography and nephropathy screening increased, although changes in 7 other HEDIS measures were not significant.

 A separate randomized controlled trial in New York evaluating PCMH transformation over 2008 to 2010 among 32 practices (18 intervention practices) found improvements in 2 of 11 quality indicators: hypertensive blood pressure control and breast cancer screening.

Notably, other PCMH studies have not shown significant impact on quality measures,

 enforcing the need for additional research in this area to understand why some initiatives are associated with quality improvement while others are not. Understanding how the PCMH model can impact quality of care is critical for optimal primary care design, especially given the enormous and rapid uptake of the PCMH model. From 2009 to 2014, the number of patients covered by PCMH initiatives increased from nearly 5 million to almost 21 million across more than 100 state and federal initiatives.

In addition, we need more efficient and effective methods to evaluate primary care and access to care, with a particular focus on the patient experience as medicine becomes increasingly patient-centered. A PCMH transformation study evaluating patient experiences in a pilot practice in eastern Massachusetts with Lean enhancement and payment reform found that patient experience was sustained or improved across PCMH domains including the personal physician and communication domain.

 Our study builds on this finding, utilizing a simple tool for evaluating access to enhanced primary care directly from patients in a large, state-based population survey of patient respondents. This 5-question survey module was a valuable addition to the BRFSS and provided a new population measure of access to enhanced primary care. This survey of critical primary care functions may be a replicable and scalable strategy that other states and healthcare systems can use to characterize their own primary care transformation efforts and reach. 

While it may be that the enhanced primary care components evaluated are inseparable from what might be simply termed high-quality traditional primary care, such a distinction may be irrelevant. The PCMH model provides structure and script for the delivery of better primary care. This is invaluable, as it allows struggling practices to implement a comprehensive plan for improvement rather than disparate quality improvement efforts that may fail due to their lack of integration into the central practice philosophy and administration. However, there is still great variability in how PCMHs are implemented from site to site,

 and evaluations at a population level are important. In addition, evaluations within a single system may reflect characteristics of that system only and may not be generalizable to PCMHs implemented in a broad array of practice types.

In spite of the demonstrable increases in quality and access to healthcare brought about by the Massachusetts healthcare reform law of 2006, healthcare costs continue to increase steadily. Reform efforts have not contributed to any additional rise in healthcare costs, but neither have they led to a significant decrease in overall costs.

 One of the largest drivers of healthcare expenditure is preventable hospitalizations and emergency department visits.

 There is evidence to suggest that these expenditures can be mitigated by high-quality, long-term, prevention-based primary care focused on managing the chronic conditions that often result in hospitalization when mis- or undermanaged.

 Delivering such care is the goal of the PCMH, although it remains to be seen whether the continued expansion of this model results in the expected and sustained cost reductions.

This study is a cross-sectional pilot study that includes only 1 year of data, although the sample is large and representative of the population of Massachusetts. A majority of the differences in quality outcomes in the enhanced primary care group were modest; however, the differences in diabetes care were quite large and may have been even larger had we not used a bundled care process measure, which can result in lower scores.

 Furthermore, the 5-question module used in this study was validated as a measure of primary care quality,

 but not as an indicator of whether a patient was treated by a PCMH. Thus, it is an indirect measurement of medical home attributes. As such, we cannot say with certainty that these differences in quality are a direct result of receiving enhanced care or are indicative of practices that are themselves medical homes, and we had no direct measure of which patients were receiving their care in a PCMH. The study is also limited by its reliance on self-report for all measures; however, this direct feedback from patients provides an important window into the patient perspective and experience. 

Evaluation of the PCMH and how it affects patient experience and quality measures on a population level is critical to monitor the progress of reform measures, such as the ongoing large-scale adoption of medical homes. Other major health reforms, such as increased insurance coverage, are being deployed on a state level, and we need new population-level tools to evaluate these initiatives. The pilot implementation of the 5-question enhanced primary care module in the Massachusetts BRFSS provides a simple, low-cost model, which other states may use to gather this crucial experience and quality data. We found that higher levels of enhanced primary care attributes at practices were associated with substantially improved quality for an array of measures, suggesting that this benefit may be realized even at the population level.

The authors would like to thank Bruce Cohen, PhD, Director of Research and Epidemiology and Bureau of Health Statistics, Research and Evaluation at the Massachusetts Department of Public Health; Liane Tinsley, MPH, for assistance with implementing the PCMH module into the BRFSS survey; and also Jennifer Kincheloe, PhD, MPH.

 Brigham and Women’s Hospital (AB, AWB, JSW, DWB), Boston, MA; Department of Health Care Policy, Harvard Medical School (AB, JSW), Boston, MA; Ariadne Labs (AB), Boston, MA; Veterans Affairs Boston Healthcare System (AWB), Boston, MA; Pennsylvania State University College of Medicine (SC), Hershey, PA.

 Dr Bitton is a board member of Health Leads and is a senior advisor for the Center for Medicare & Medicaid Innovation. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (AB, JSW, DWB); acquisition of data (AB, JSW); analysis and interpretation of data (AB, AWB. JSW, SC, DWB); drafting of the manuscript (AB, AWB, JSW, SC, DWB); critical revision of the manuscript for important intellectual content (AB, AWB, SC, JSW, DWB); statistical analysis (AB, SC); obtaining funding (AB, DWB); administrative, technical, or logistic support (AB, DWB, SC, AWB); and supervision (AB, JSW, DWB).

 Asaf Bitton, MD, MPH, Assistant Professor of Medicine and Health Care Policy, Division of General Medicine and Ariadne Labs, Brigham and Women’s Hospital, 1620 Tremont St, 3rd Fl, Rm 3-002P, Boston, MA 02120. E-mail: abitton@partners.org.

In a statewide telephone survey, patient-reported indicators of the patient-centered medical home correlated with improved process measures in diabetes, cholesterol screening, vaccination, and care access.  

Enhanced Primary Care and Impact on Quality of Care in Massachusetts

To determine whether clinical decision support (CDS) is associated with improved quality indicators and whether disabling CDS negatively affects these.

Using the 2006-2009 National Ambulatory and National Hospital Ambulatory Medical Care Surveys, we performed logistic regression to analyze adult primary care visits for the association between the use of CDS (problem lists, preventive care reminders, lab results, lab range notifications, and drug-drug interaction warnings) and quality measures (blood pressure control, cancer screening, health education, influenza vaccination, and visits related to adverse drug events).

There were an estimated 900 million outpatient primary care visits to clinics with EHRs from 2006-2009; 97% involved CDS, 77% were missing at least 1 CDS, and 15% had at least 1 CDS disabled. The presence of CDS was associated with improved blood pressure control (86% vs 82%; OR 1.3; 95% CI, 1.1-1.5) and more visits not related to adverse drug events (99.9% vs 99.8%; OR 3.0; 95% CI, 1.3-7.3); these associations were also present when comparing practices with CDS against practices that had disabled CDS. Electronic problem lists were associated with increased odds of having a visit with controlled blood pressure (86% vs 80%; OR 1.4; 95% CI, 1.3-1.6). Lab result notification was associated with increased odds of ordering cancer screening (15% vs 10%; OR 1.5; 95% CI, 1.03-2.2).

The use of CDS was associated with improvement in some quality indicators. Not having at least 1 CDS was common; disabling CDS was infrequent. This suggests that meaningful use standards may improve national quality indicators and health outcomes, once fully implemented.

		The use of some electronic health record clinical decision support (CDS) was associated with improved quality of care among national primary care outpatient visits from 2006 to 2009.

		Most practices were missing at least 1 CDS module considered essential by meaningful use standards.

		Meaningful use standards may have a significant impact on national quality of care and health outcomes once fully implemented.

The CMS Electronic Health Record Incentive Program, better known as “meaningful use,” provides initial monetary incentives for adoption of electronic health records (EHRs) and subsequently imposes monetary penalties if physician practices do not meet requirements. The meaningful use program has spurred a significant rise in the number of office-based physicians reporting EHR use, from 24% in 2005 to 48% in 2009 to 72% in 2012.

The meaningful use goals and objectives are divided into 3 stages. Stage 1 requirements include using a certified EHR to capture and share patient data by 2015; stage 2, beginning in 2014, includes use of advanced care processes with clinical decision support (CDS); and stage 3, to begin in 2016, will require providers to demonstrate outcome improvements. The time frame for stages 2 and 3 was recently extended.

 As of March 2013, 73% of eligible providers had registered to participate in the meaningful use incentive program and 44% of eligible providers had received incentive payment for meeting meaningful use objectives.

Although the meaningful use requirements have already been established, the evidence is inconsistent regarding improvement in healthcare processes or patient outcomes as a result of the implementation of general and individual EHR components. Prior studies have shown that EHR-based CDS is associated with improved prescribing safety,

 CDS has also been associated with some improvements in quality indicators,

 Despite having EHRs, many physicians report being unable to complete basic panel management activities,

which affects their ability to deliver high-quality care for patients with chronic conditions.

Notwithstanding the mandate to use certified EHRs, use of an EHR in the outpatient setting has not yet been shown unequivocally to improve quality indicators across a broad range of practices and EHRs, especially because there is substantial variability among the 1600 certified EHR platforms.

In addition to the large number of EHRs, they also can be customized to each practice’s needs and resource availability, so results will vary between practices in terms of impact on care even with the use of a single vendor’s practice. Certain types of practices, such as smaller community-based practices, may be less able to fully implement a complete EHR with clinical decision support, which is the kind postulated to have the greatest potential to improve quality.

 Decisions about choices around the CDS functions implemented may affect the quality of care delivered by that practice. To further inform the potential effect of meaningful use stages 2 and 3, we examined the association between having CDS, not having CDS, and disabling CDS, and quality of care indicators using nationally representative data just prior to the start of the meaningful use incentive program.

We performed a retrospective, cross-sectional analysis of adult primary care ambulatory clinic visits in the National Ambulatory Medical Care Survey (NAMCS) and National Hospital Ambulatory Medical Care Survey (NHAMCS) outpatient department records from 2006-2009 in order to analyze data with little impact from current meaningful use incentive payments. We examined a number of patient and practice characteristics to determine any differences in access to practices with EHRs based on prior work showing associations between these characteristics and EHR availability.

On the premise that most preventive care takes place through a primary care practice, we limited our sample to just those practices.

The NAMCS and NHAMCS are conducted by the Ambulatory Care Statistics Branch of the National Center for Health Statistics (NCHS) of the CDC. These surveys focus on outpatient visits to non—federally funded US medical practices.

The NCHS uses a complex multistage probability sampling design based first on geography, then physician specialty, and finally patient visits to individual practices. Information is collected at ambulatory visits by participating practices or US Census Bureau representatives. The NCHS assigned weights to each visit record to allow for estimation of national statistics. The NCHS institutional review board approves the protocols for the NAMCS and NHAMCS, including a waiver of informed consent for patients.

The NAMCS collects information at a visit level from community-based physician practices; from 2006 to 2009, approximately 30,000 community-based office visits were recorded annually. The NHAMCS collects information at a visit level from hospital-based physician practices; from 2006 to 2009, approximately 34,000 hospital-based office visits were recorded annually.

In all years included in the analysis, both the NAMCS and NHAMCS included questions in their intake surveys that asked whether the practice had an electronic medical record (for other than billing purposes), a computerized patient problem list, computerized warnings of drug interactions or contraindications, a computerized system to view lab results, a computerized system that highlights out-of-range lab values, and a computerized system for reminders for guideline-based interventions and/or screening tests. Response variables indicated that a practice had a function (“yes”), did not have a function (“no”), had the function but disabled it (“turned off”), or was unsure if the practice had the function (“don’t know” or “blank”).

The NAMCS and NHAMCS record health information for each visit sampled. The respondent indicates the top 3 reasons for the visit, which are then coded by NCHS; the patient’s vital signs, including blood pressure; whether any health education was ordered or provided during the visit; and whether an influenza vaccination was ordered or administered during the visit. The respondent also indicates any screening services ordered or performed during the visit. The respondent can record mammography, Pap smears, and scope procedures (NCHS records and codifiers, free text scope procedures, 1 of which is sigmoidoscopy/ colonoscopy), among others.

The NAMCS and NHAMCS collect patient and practice demographic information: patient age, patient race, patient income category (imputed based on patient zip code by NCHS), primary reason for the visit, region of the country, practice ownership, and number of physicians in the practice (solo or non-solo). Race data are collected in the manner already in use by the practice. Due to high nonresponse rates for race (24% to 33% in NAMCS, and 12% to 15% in NHAMCS for the years included), the NCHS also provides a race variable imputed based first on the patient’s location, then on physician specialty and 

International Classification of Diseases, Ninth Revision, Clinical Modification

 code for primary diagnosis, and finally on a random basis.

Our predictors of interest were EHR CDS modules mandated by stage 1 of meaningful use: electronic problem lists, preventive care reminders, lab result reporting, out-of-range lab notification, and drug-drug interaction warnings. We performed analyses with each predictor separately, and performed analyses combining them into a composite group of key EHR features. Based on prior work that examined associations between certain EHR features and clinical outcomes,

 we developed a matrix between CDS functions and the clinical outcomes of interest and performed analyses accordingly (

Our main outcomes of interest were based on national quality metrics related to adult primary care: blood pressure control (defined as a systolic blood pressure less than 140 mm Hg), age- and gender-appropriate cancer screening (mammography, Pap smear, and sigmoidoscopy or colonoscopy), health education for particular conditions, influenza vaccination during the months of October through March, and adverse drug events (as measured by visits coded to be related to an adverse drug event). We also combined the receipt of cancer screening, health education, and influenza vaccination into a composite “preventive care” outcome measure.

We were interested in the difference in outcomes based on whether a practice had a CDS function or not. Specifically, we examined whether there were significant differences in the rates of the clinical outcomes depending on whether a visit was to a practice with the CDS or without the CDS.

We categorized practices into 3 groups: all CDS tools active; without 1 or more CDS function; or any disabled CDS. We considered the practice to have all CDS active if they answered “yes” to having all 5 of the individual CDS. We considered the practice to be without 1 or more CDS functions if they answered “no,” “turned off,” don’t know,” or “blank” to any of the CDS functions. We considered the practice to have “disabled CDS” if they answered “turned off” for any of the CDS functions.

We collapsed practice ownership into 5 categories: physician or physician group, community health center, health maintenance organization, medical or academic health center, and other. All other covariates were used as provided by NCHS.

The unit of analysis was the patient visit. We followed NCHS guidelines based on the hierarchical sampling design. We performed all statistical tests on data with less than a 30% relative standard error (the standard error divided by the estimate itself) and more than 30 sample records.

We used multivariable logistic regression modeling to control for a number of patient and practice level covariates. We modeled age as a continuous variable, and all other covariates as categorical variables. We evaluated categorical variables with the χ2 test and continuous variables with the t test. We performed multivariable logistic regression modeling for each quality measure individually. The main predictors of interest were the status of the EHR CDS function. We considered P values of less than .05 to be significant.

We used SAS statistical software (version 9.3; SAS Institute, Cary, North Carolina) for all analyses.

From 2006-2009, the NAMCS/NHAMCS databases included 104,102 adult primary care visits, representing 2 billion adult primary care visits in the United States. Of these, 45% of visits were to practices with EHRs. Of those visits, 97% also had at least 1 of the 5 CDS functions of interest. Of visits using EHRs with at least 1 CDS function, 77% did not have (for any reason), and 15% had disabled (actively turned off), at least 1 of the 5 CDS functions of interest.

We compared patient and practice demographics for visits to clinics with all the CDS functions versus visits to clinics with at least 1 CDS function missing (

). Patients were of similar age (50.9 years vs 51.1 years,

 = 0.84). Rates of visits for flares of chronic problems (7%) and pre- or post surgery (2%) were similar between the 2 groups; rates of visits for new problems (39% vs 41%) and follow-up of chronic problems (32% vs 33%) were less common and visits for preventive care (17% vs 16%) were more common at clinics without at least 1 of the CDS functions versus clinics with all the CDS functions (

 <.0001). Of visits to practices with at least 1 CDS function missing versus those to practices with all the CDS functions, rates of visits to health maintenance organizations (3% vs 7%) were lower, and to physician groups (60% vs 50%), community health centers (5% vs 3%), or medical/academic health centers (18% vs 12%) were higher (

 <.0001). There were no differences in the distribution of visits to clinics without at least 1 of the CDS functions versus visits to clinics with all the CDS functions by gender, race, or region of the country.

Most patients who made visits (81%) had controlled blood pressure. Of eligible visits, there were low documented rates of age-appropriate cancer screening (13%) and influenza vaccination (4%). Approximately 0.2% of visits were related to an adverse drug event. Patients who made visits to clinics with all the CDS functions present versus those who visited clinics that were missing at least 1 of the CDS functions were more likely to have controlled blood pressure (86% vs 82%; OR 1.3; 95% CI, 1.1-1.5) and more likely to not have adverse drug event visits (99.9% vs 99.8%; OR 3.0; 95% CI, 1.3-7.3) (

We found similar results when specifically analyzing visits to clinics with all the CDS functions versus visits to clinics with any of the CDS functions disabled. Patients were more likely to have controlled blood pressure (86% vs 83%; OR 1.5; 95% CI, 1.02-2.3) and more likely to not have adverse drug event visits (99.9% vs 99.8%; OR 5.2; 95% CI, 1.4-19.0) (

We next examined only those CDS functions directly related to each clinical outcome. There were higher odds of receiving cancer screening if the visit was to a clinic with electronic lab results (15% vs 10%; OR 1.5; 95% CI, 1.03-2.2) or notification of acceptable lab ranges (16% vs 11%; OR 1.4; 95% CI, 1.03-1.8) than if the visit was to a clinic without these CDS functions. There were higher odds of having controlled blood pressure (86% vs 80%; OR 1.4; 95% CI, 1.3-1.6) if the visit was to a clinic with electronic problem lists rather than to a clinic without this EHR function (

Because of extremely small numbers and/or absence of outcomes in certain subgroups, we were unable to analyze the association of disabling particular CDS functions with specific clinical outcomes.

Using national surveys of ambulatory visits to primary care clinics, we found significant associations between the use of CDS and some, but not all, clinical quality measures before the enactment of meaningful use. Physicians reporting use of CDS had patients with better blood pressure control and fewer visits related to adverse drug events; these associations were present regardless of whether the physicians with CDS were compared with physicians without CDS or with physicians with disabled CDS. Physicians who reported having electronic problem lists had better rates of blood pressure control of their patients. Physicians with electronic lab results or notification of lab ranges were more likely to provide ageappropriate cancer screening. As we enter stages 2 and 3 of meaningful use in which clinicians must use CDS and then provide data about their performance with respect to patient outcomes, hopefully performance will continue to improve, though this remains to be demonstrated.

The incidence of not having at least 1 of the CDS functions was common, which is likely because the meaningful use mandate was not yet in place at the time of these national surveys. However, despite not having this mandate, the incidence of disabling any CDS was infrequent. Keeping decision support active is a “best practice” in clinical decision support,

but prior experience has demonstrated that many practices elected to disable specific types of decision support before meaningful use. These decisions can sometimes be indicated if the EHR is generating too many false positive warnings, which can create alert fatigue, although there typically are better solutions. The low rate of disabling CDS is reassuring and may indicate that the performance characteristics of CDS are improving. Notably, the odds of having controlled blood pressure and avoiding adverse drug events were higher at clinics with active CDS compared with those who had disabled it.

Prior studies of EHR use and clinical quality outcomes have been inconsistent in their findings. One study using earlier data from the same surveys used here found almost no associations between CDS use and clinical quality measures.

A more recent study found a significant, although small, improvement in blood pressure control of patients visiting physicians with CDS functions in their EHR versus patients visiting physicians without EHRs or CDS.

 In our study, we found an association between general CDS use and blood pressure control, and more consistent associations between quality indicators and CDS functions that were clinically related to these measures. These associations may be present because EHRs and CDS are evolving in a positive direction, spurred in part by the requirements of certification and also those of meaningful use.

Despite the positive findings, we have also found a lack of association between general CDS use and cancer screening, influenza vaccination, or health education, just as many previously have shown a lack of association between general EHR use and certain health outcomes.

 These findings may be because it is harder to change outcomes than quality processes. Some diseases, such as diabetes and hypertension, may be more amenable to clinical decision support than others, in part because there are clear guidelines and metrics.

This study has several limitations. We could not directly measure actual use of CDS during a visit, as the survey consisted of self-reported metrics. Even when physician practices have EHR features at their disposal, clinicians find ways to bypass or ignore them.20,21 Of the small number of practices that chose to disable CDS functionality, we do not know the reasoning behind their decisions, which may, in some cases, have been justified. In addition, the distinction between not having CDS functions and having turned off CDS may be sufficiently nuanced as to lead survey respondents to misclassify their EHR, although this would not affect our analysis of the difference between visits to clinics with all 5 of the CDS functions versus those without CDS. In addition, there were low rates of cancer screening and influenza vaccination captured in this visit-based incident data. There were also low rates of adverse drug event outcomes.

In contrast with some other studies, we found that the use of CDS functions was associated with improvements in measures of health quality at a national level. In addition, decisions to disable these CDS functions, although infrequent, negated these increases in care quality. With the continued implementation of meaningful use, there should be a further decline in practices without EHRs or that choose not to implement or to disable CDS. Overall, meaningful use standards that include CDS appear to have a significant positive effect on some national quality-of-care indicators and health outcomes. It will be important to evaluate the evolving impact of meaningful use as the stages continue to be more widely implemented and better integrated with care processes; we anticipate further gains in healthcare quality indicators and outcomes as a result.

 Division of General Medicine and Primary Care, Brigham Women's Hospital (RGM, JAL, DWB, AB); Brigham and Women’s Hospital (RGM, JAL, AB); Harvard Medical School (RGM, JAL, DWB, AB); Harvard Medical School Center for Primary Care (AB); Harvard School of Public Health (DWB), Boston, MA.

 Dr Mishuris is supported by an Institutional National Research Service Award to the Harvard Medical School Fellowship in General Medicine and Primary Care (T32 HP10251). No sponsor had a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review or approval of the manuscript; or decision to submit the manuscript for publication.

Dr Bates holds a minority equity position in the privately held company Medicalis, which develops Web-based decision support for radiology test ordering, and he has served as a consultant to Medicalis. Dr Bates serves as an advisor to Calgary Scientific, which makes technologies that enable mobility within electronic health records. He is on the clinical advisory board for Patient Safety Systems, which provides a set of approaches to help hospitals improve safety. He has received funding support from the Massachusetts Technology Consortium. Dr Bitton serves part-time as a senior advisor to the Center for Medicare and Medicaid Innovation (CMMI). The research presented in this paper was conducted solely within the scope of his faculty appointments and research at Brigham and Women’s Hospital and Harvard Medical School, and in no way reflects any official position of CMMI. The authors report no other conflicts of interest.

 Concept and design (RGM, JAL, DWB, AB); acquisition of data (RGM, AB); analysis and interpretation of data (RGM, JAL, DWB, AB); drafting of the manuscript (RGM, JAL, AB); critical revision of the manuscript for important intellectual content (RGM, JAL, DWB, AB); statistical analysis (RGM, JAL, AB); administrative, technical, or logistic support (DWB); and supervision (JAL).

Using EHR clinical decision support is associated with improved quality of care. Most primary care practices are missing at least 1 "meaningful use" clinical decision support module.

Asaf Bitton, MD, MPH

Articles

Gatekeeping and Patterns of Outpatient Care Post Healthcare Reform

Enhanced Primary Care and Impact on Quality of Care in Massachusetts

Using Electronic Health Record Clinical Decision Support Is Associated With Improved Quality of Care

Toward a Better Understanding of Patient-Reported Outcomes in Clinical Practice