John L. Adams, PhD

To estimate the relationship between quality improvement and electronic health record (EHR) adoption in US hospitals.

 National cohort study based on primary survey data about hospital EHR capability collected in 2003 and 2006 and on publicly reported hospital quality data for 2004 and 2007.

 Difference-in-differences regression analysis to assess the relationship between EHR adoption and quality improvement for acute myocardial infarction, heart failure, and pneumonia care.

Availability of a basic EHR was associated with a significant increase in quality improvement for heart failure (additional improvement, 2.6%; 95% confidence interval [CI], 1.0%-4.1%). However, adoption of advanced EHR capabilities was associated with significant decreases in quality improvement for acute myocardial infarction and heart failure. We observed 0.9% (95% CI, -1.7% to -0.1%) less improvement for acute myocardial infarction quality scores and 3.0% (95% CI, -5.2% to -0.8%) less improvement for heart failure quality scores among hospitals that newly adopted an advanced EHR, and 1.2% (95% CI, -2.0% to -0.3%) less improvement for acute myocardial infarction quality scores and 2.8% (95% CI, -5.4% to -0.3%) less improvement for heart failure quality scores among hospitals that upgraded their basic EHR.

 Mixed results suggest that current practices for implementation and use of EHRs have had a limited effect on quality improvement in US hospitals. However, potential "ceiling effects" limit the ability of existing measures to assess the effect that EHRs have had on hospital quality. In addition to the development of standard criteria for EHR functionality and use, standard measures of the effect of EHRs on quality are needed.

(Am J Manag Care. 2010;16(12 Spec No.):SP64-SP71)

	Consistent availability of an electronic health record (EHR) over the study period was associated with a significant increase in quality improvement for heart failure; however, adoption of advanced EHR capabilities was associated with significant decreases in the improvement of acute myocardial infarction and heart failure quality scores.

		Prolonged availability of an EHR was associated with some significant gains in quality improvement. n Recent adoption of or upgrade to an advanced EHR was associated with smaller quality gains.

		"Ceiling effects" may limit the usefulness of standard hospital quality measures for assessing the effects of EHR adoption on quality.

	There is general consensus that widespread adoption of health information technology (IT), in particular electronic health records (EHRs), will result in increased efficiency and improved patient care.

 This belief is reflected in the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009, which includes funds to stimulate adoption of EHRs.

 The Congressional Budget Office estimates net payments of approximately $30 billion for Medicare and Medicaid incentives over the life of the program.

 A sizable portion of these payments will be made to hospitals that are able to demonstrate “meaningful use” of a “certified” EHR. Using EHRs to improve quality is an example of  meaningful use.

 Smaller (eg, 75 bed) hospitals can receive up to $3.5 million in incentive payments, whereas larger (eg, 500 bed) hospitals could receive up to $6.1 million over the life of the program.

 The Congressional Budget Office projects that these incentives will induce 25% more US hospitals to adopt an EHR that would not have done so otherwise.

	An expected benefit of EHR adoption is improved quality of care.

 However, much of the current knowledge about the relationship between health IT and hospital quality comes from a few hospitals that may not be representative of the broader set of hospitals being targeted by the HITECH incentives.

 that have examined the  relationship between hospital quality and EHR use in larger samples of hospitals have to a varying degree reported positive associations between EHR use and hospital quality. With adoption of hospital IT, some studies (eg, those by Amarasingham et al

) report substantial decreases in hospital mortality rates and   complications, while more recent studies (eg, those by DesRoches et al

) report only modest improvements in hospital quality associated with the availability of an EHR. Aside from analyses by McCullough et al

 studies have  been limited to cross-sectional data. In addition to their limited capacity to control for confounding factors, cross-sectional studies may not fully address the important question facing policy makers: Will installing a new EHR (or increasing the functionality of an existing EHR) lead to increased improvements in quality over time? To examine this question, we evaluated longitudinal data on EHR adoption and hospital quality from a large sample of US hospitals.

	We used primary survey data from the Health Information and Management Systems Society (HIMSS) to measure hospital EHR adoption. The HIMSS Analytics Database includes approximately 90% of hospitals in the United States.

 The database contains information on the implementation status of a wide range of clinical IT applications and has been frequently used for research purposes.

 To resolve previously observed inconsistencies among the HIMSS data,

 we dropped hospitals that failed to disclose their  software vendor or reported a vendor that was inconsistent with the clinical IT application reported. Hospitals with self-developed systems were included in the analysis. 

) lists software vendor and clinical application combinations included in the analysis.

	Data on hospital characteristics were obtained from the American Hospital Association Annual Survey Database.

 This database includes 800 variables on more than 3900 US

	Data on hospital quality were obtained from the Hospital Compare

 database for 2004 and 2007. The Hospital Compare database includes process-of-care measures that indicate how often hospitals provide elements of clinical care that are well-established interventions for 3 common clinical conditions (acute myocardial infarction [AMI], heart failure, and pneumonia) for more than 4200 hospitals. These measures are calculated based on all hospital patients, not just Medicare beneficiaries.

	We selected all nonfederal general acute care hospitals located in the United States from the American Hospital Association Annual Survey Database (3971 hospitals). We linked these eligible hospitals to the HIMSS Analytics Database using Medicare provider numbers, restricting the analytic file to 2086 hospitals that reported their EHR capability in 2003 and 2006. We linked the combined data set to the Hospital Compare database using Medicare provider number and year. Because the HIMSS database captures the most  up-to-date information on a given hospital’s EHR implementation status at year end, we linked quality measures from the ensuing year (eg, 2006 data from the HIMSS database  were linked to 2007 data from the Hospital Compare database). This linkage process resulted in a final database that contained 2021 hospitals with observations in both years.

	We selected 17 measures of hospital process quality across 3 clinical conditions that were common in both years of the Hospital Compare database (

). Eight of these measures were for processes related to the treatment of AMI, 4 for processes related to the treatment of heart failure, and 5 for processes related to the treatment of pneumonia.  These process-of-care measures were chosen because they apply to conditions that are common causes of hospitalization,

 because they are generally regarded as being valid  indicators of quality,

 and because EHRs are more likely to facilitate adherence to recommended processes of care than to affect patient outcomes (eg, in-hospital mortality).

 The dependent variables for this analysis were 3 composite measures of hospital process quality for AMI, heart failure, and pneumonia. The composite measures were constructed  using the approach prescribed by the Joint Commission

 of grouping 17 individual process measures by condition (AMI, heart failure, or pneumonia), summing the  numerators of the individual measures (ie, recommended care delivered), and then dividing by the sum of the denominators (ie, total eligible population). This produced an overall  quality performance rate for each clinical condition. Only hospital-year composite measure combinations with at least 30 denominator observations were included in the analysis.

	There is no standard measurement of EHR capability.

 However, the current state of the art is to delineate between EHR systems that offer more advanced functionalities and those that do not.

 have advocated a 3-tiered framework (no EHR, basic EHR, and comprehensive EHR) for classifying EHR capability. Their   framework is based on the presence or absence of 24 EHR functionalities. McCullough et al16 observed that, while the classification framework by Jha and DesRoches and their colleagues establishes an important standard for leading EHR adopters, the restrictive inclusion criteria for full adoption do not facilitate the analysis of the typical EHR adopter.

	We sought to adapt the 3-tiered framework proposed by Jha et al

 and to create a less restrictive classification that would allow for the analysis of typical EHR adopters. To determine the level of EHR capability for the hospitals in our study, we evaluated the self-reported implementation status of the following 4 clinical IT applications: clinical data repository, electronic patient record, clinical decision support systems, and computerized provider order entry (see eAppendix B for definitions of clinical IT applications). Hospitals that did not report having the full complement of technology necessary to constitute a basic EHR were included in the first tier (no EHR). Hospitals that reported having an operational electronic patient record, clinical data repository, and clinical decision support systems were included in the second tier (basic EHR). Hospitals that reported having EHRs with all the functionality of the second tier plus an operation- al computerized provider order entry system were classified in the third tier (advanced EHR).  Computerized provider order entry was chosen as the marker of an advanced EHR because its adoption is low compared with other clinical IT applications, indicating that it is often implemented after other elements of the clinical information system are already in place. Furthermore, well-documented functional enhancements accompany its implementation in conjunction with other clinical IT applications. In particular, computerized provider order entry is regarded as a more effective means of delivering clinical decision support because it facilitates decision support at the point of care.

	The goal of the analysis was to assess whether acquisition or upgrade of an EHR was associated with increased improvement in hospital quality over time, controlling for baseline characteristics that might influence changes in quality. We first assessed whether baseline EHR capability varied by hospital characteristics using Fisher exact test. To adjust for baseline differences between hospitals with different EHR capabilities, we estimated a propensity score for baseline EHR capability using an ordinal logistic regression analysis. We regressed baseline EHR capability on the hospital covariates listed in 

. Hospitals were assigned an indicator variable (range, 1-5) based on the quintiles of the propensity score distribution. Covariate balance between the 5 levels of the propensity score indicator was assessed via ordinal logistic regression analysis.39

	We used a difference-in-differences analytic approach to estimate the relationship between EHR transitions and improvement in each of 3 composite measures of hospital quality. To evaluate the association between quality improvement over time and the availability of an EHR, we compared hospitals that maintained a basic or an advanced EHR with hospitals that reported having no EHR in 2003 and 2006. To evaluate the association between quality improvement over time and new adoption (and upgrades) of an EHR, we stratified hospitals by their baseline EHR capability in 2003 and then compared hospitals that newly adopted or upgraded an EHR with hospitals that did not change their EHR capability. This approach allowed us to specifically examine EHR transitions that the HITECH legislation is designed to induce.

	Unadjusted difference-in-differences estimates were calculated using standard methods.

 Adjusted differences in differences were calculated via linear mixed-effects regression analysis. We estimated separate regression models for each composite measure. Each regression model included a random-effects term to adjust for clustering within individual hospitals and fixed effects for the hospital’s EHR transition (no transition, new adoption, or EHR upgrade), baseline propensity score (range, 1-5), period (baseline vs  follow-up), and characteristics listed in Table 2. Each regression model also included an interaction term between time and the EHR transition variable, which estimated the association between the composite measure and a given EHR transition relative to no EHR transition. The unadjusted and adjusted estimates can be interpreted as the additional quality gains associated with adoption (or upgrade) of an EHR compared with a referent set of hospitals that did not adopt or upgrade their EHR. For the most part, interpretation of the adjusted and unadjusted results was consistent. However, adjusted estimates were generally more precise; therefore, we report them only. All statistical analyses were performed using R version 2.10.0 (R Foundation for Statistical Computing, Vienna, Austria).

 The study qualified for exemption by the local institutional review board.

	As listed in column 1 of Table 2, the hospitals in our sample varied in size; the largest hospital group (31.0%) had 100 to 199 beds. Most hospitals in our sample were nonteaching (71.1%), not for profit (68.4%), or affiliated with a health system (62.1%). More than half (53.8%) of hospitals were located in metropolitan areas, and 41.9% had a dedicated coronary care unit. Consistent with other evidence,

 we found that baseline levels of EHR capability were associated with hospital size, teaching status, healthcare system affiliation,  urbanization, and the presence of a dedicated coronary care unit. 

 summarizes how the study sample of hospitals compares with the full American Hospital Association  Annual Survey Database sample of general acute care hospitals. Generally, the hospitals in our sample were larger and nongovernment owned. They were also more  likely to be affiliated with a healthcare system, be located in an urban area, have a dedicated coronary care unit, and be a teaching facility.

, the number of hospitals in our sample with a basic or an advanced EHR increased from 24.0% in 2003 to 37.7% in 2006. We also found a sizable increase in the percentage of hospitals with an advanced EHR, from 2.0% in 2003 to 12.2% in 2006. 

 lists levels of improvement in hospitals that did not experience a transition in EHR capability during the study period. Relative to hospitals with no EHR, hospitals that maintained a basic EHR realized significantly greater improvement in their heart failure quality scores (increased improvement, 2.6%). Hospitals that maintained a basic EHR experienced similar increases in AMI and pneumonia quality scores compared with hospitals that did not, and quality scores in hospitals with an advanced EHR did not improve significantly more or less than quality scores in hospitals without an EHR.

 gives changes in AMI, heart failure, and pneumonia quality scores among hospitals that changed their EHR capability. Quality scores did not improve significantly more or   less in hospitals that adopted a basic EHR than in hospitals that did not adopt an EHR. However, in hospitals that newly adopted an advanced EHR, AMI and heart failure quality  scores improved significantly less than in hospitals that did not adopt an EHR (−0.9% for AMI and −3.0% for heart failure). Acute myocardial infarction and heart failure quality scores improved significantly less in hospitals that upgraded their basic EHR compared with hospitals that maintained their basic EHR (−1.2% for AMI and −2.8% for heart failure). We found no significant relationship between new EHR  adoption or upgrade and quality improvement for pneumonia. Full regression tables for the propensity score model and forthe   mixed-effects regression models are available in 

	During the study period, the quality of care for AMI, heart failure, and pneumonia was broadly improving. Heart failure quality scores improved significantly more among hospitals that maintained a basic EHR than among hospitals with no EHR. We did not observe a similar effect on AMI or pneumonia quality scores, nor did we find that adopting or upgradingan EHR accelerated quality improvement. Instead, our results indicate that new adoption or upgrade to an advanced EHR was associated with smaller gains in AMI and heart failure

	Our findings overall were mixed; on the one hand, the increased improvement in heart failure quality scores over time associated with maintenance of a basic EHR is encouraging. On the other hand, the smaller quality gains associated with new adoption or upgrade to an advanced EHR are somewhat counterintuitive. Although unexpected, our results are consistent with findings by Greenhalgh et al,

 who report that less complex EHRs may have greater positive effects than more sophisticated ones. This phenomenon may be  attributable to the complex nature of healthcare work environments. Hospitals have been described as “the most complex human organization[s] ever devised,”

 and the introduction of increasingly complex technology into already complex work environments may trigger various unintended interactions that undermine or outweigh the potential  benefits of the new technology.

	Our study has some limitations. First, it is possible that EHR adoption might have different effects on quality improvement for conditions other than the 3 we studied. Second, our approach to measuring EHR capability did not account for the extent or adequacy of EHR implementan tion within a given hospital or the frequency and manner in which the EHR was used, nor were we able to account for the substantial variation in functionality that exists between different EHRs. Moving forward, metrics of meaningful use of an EHR should make it possible to better assess and identify which elements of EHR use have the greatest effect on clinical quality. Third, although it was our intent to analyze the effect of EHR adoption at typical hospitals, the subset of hospitals for which we had data may not be entirely representative of all US hospitals. It is also possible that uncaptured baseline differences between hospitals that already had or subsequently adopted an EHR could bias our results.

	Fourth, the potential for “ceiling effects” may limit the interpretation of our results. The composite measures for AMI and pneumonia may be particularly affected by this phenomenon. Our results suggest that improving quality scores beyond 91.0% to 92.0% for pneumonia and 93.0% to 94.0% for AMI may be considerably more challenging than improving quality

	below those levels. These ceiling effects may explain why we observed a significant decrease in the rate of quality improvement for AMI and no significant change in the rate of quality improvement for pneumonia among hospitals that adopted new EHR capabilities. The heart failure composite quality scores seem less likely to be subject to ceiling effects, as the mean 2007 scores are generally lower (83.0% to 87.0%) than the scores for the other clinical conditions. In fact, Table 5 illustrates that hospitals newly adopting an advanced EHR had lower heart failure quality scores than hospitals not adopting an EHR in 2007, despite having slightly higher baseline quality scores in 2004. This result suggests that new adoption of an advanced EHR indeed slowed quality improvement in these hospitals.

Articles

A national study of electronic health record (EHR) adoption and hospital quality finds that existing measures may be inappropriate for assessing the effect of EHR adoption on quality.

Electronic Health Record Adoption and Quality Improvement in US Hospitals

To quantify rates of stand-alone e-prescribing (SEP) adoption and use among primary care physicians (PCPs) participating in a SEP initiative and to determine which physician and patient characteristics were associated with higher rates of each.

Using records from an insurer-led SEP initiative, we compared the characteristics of 297 PCPs who adopted SEP through the initiative with the characteristics of 1892 eligible PCPs who did not. Among 297 adopters, we studied the extent of SEP use.

Dependent variables included each physician’s adoption of SEP and his or her e-prescribing use ratio (the ratio of electronic prescriptions to pharmacy claims in the same period). Independent variables included characteristics of PCPs (specialty, practice size, and prescribing volume) and their patients (patient age, sex, race/ethnicity, and household income).

Solo practitioners, pediatricians, and physicians with more patients from predominantly African American zip codes were underrepresented among SEP adopters. The mean (SD) e-prescribing use ratio among adopters was 0.23 (0.28). Twenty percent of physicians maintained e-prescribing use ratios above 0.50. Available physician characteristics explained little of the variance in use, but physicians in smaller practices had greater use (P = .02).

Certain categories of physicians may need more tailored incentives to adopt SEP. On average, adopters used the SEP system for only about one-quarter of their prescriptions. Some adopters achieved high levels of SEP use, and further research is needed to elucidate the factors that enabled this.

e-Prescribing is seen as a critical technology for improving medication use.

In a health plan—sponsored e-prescribing initiative, the mean e-prescribing rate of participating primary care physicians (PCPs) was 1 prescription per 4 pharmacy claims, but some PCPs achieved high use.

Given this low use, future initiatives may need to consider more resources to increase e-prescribing use.

Efforts should be made to ensure that all patient demographics benefit from e-prescribing.

Higher e-prescribing use among physicians in smaller practices suggests that e-prescribing may be an appropriate manner of extending health information technology to these physicians, who traditionally are reluctant users of such technology. 

There is evidence that in some settings health information technology (HIT) can improve patient outcomes and reduce healthcare costs.

 Most of this evidence comes from 4 healthcare organizations at which academic physicians and employees are usually required to use homegrown electronic medical records (EMRs).

 However, few physicians practice in these types of environments. More than 75% of physicians practice in groups of 5 or fewer.

 Unfortunately, the structure of these small community private practices is not conducive to providing the financial and time investment necessary for EMR adoption.

 As a result, only 9% to 14% of these practices have adopted EMRs compared with 23% to 50% of larger practices.

Stand-alone e-prescribing (SEP) has been proposed as a possible method of transitioning community physicians toward EMR functionality without the initial investments required for a full EMR system.

 Indeed, recent legislation promises to increase Medicare reimbursement for e-prescribing physicians in the short term and to decrease Medicare reimbursement for paper prescribers in the long term.

We are aware of only 1 prior study that evaluates adoption and use of commercial SEP systems by community physicians. Fischer et al

 examined use and adoption of the PocketScript system, which was offered without cost to high-volume outpatient prescribers in Massachusetts. A striking finding of their analysis was the low use of e-prescribing, which (although increasing throughout the period studied) was less than 30% of all eligible prescriptions 1 year after adoption. The authors cite anecdotal evidence of increased e-prescribing since that period but present data only through early 2005. 

To further characterize experiences with SEP from other states using another e-prescribing system and in a more recent period, we quantified the rates of e-prescribing adoption and use that occurred when Horizon Blue Cross Blue Shield of New Jersey (Horizon BCBSNJ) offered SEP to community physicians participating in their health maintenance organization (HMO) and preferred provider organization network. Our primary study objectives were to quantify rates of SEP adoption and use and to determine which physician and patient characteristics were associated with higher rates of each.

Horizon BCBSNJ, New Jersey’s largest health insurer, provides coverage for 3.2 million members. In late 2004, Horizon BCBSNJ launched an initiative offering its physicians Caremark’s iScribe SEP software (Caremark is Horizon BCBSNJ’s pharmacy benefits manager). The program installed SEP systems for individual physicians rather than for practices as a whole. All features of the program, including the target population, recruitment, and incentives provided, were determined by Horizon BCBSNJ for purposes of improving care delivery. Our analysis of physicians’ SEP adoption and use was subsequently designed to use secondary data from the program and from other sources.

Of approximately 14,250 physicians in the Horizon BCBSNJ provider network, about 5890 were eligible for the program based on prescribing activity that resulted in at least 500 Horizon BCBSNJ pharmacy claims annually (this cutoff was determined by Horizon BCBSNJ for purposes of program feasibility). Eighty-seven percent of these eligible physicians were in practices containing 5 or fewer physicians. An initial wave of recruitment focused on the highest-volume prescribers (>2500 filled prescriptions per year), and subsequent phases targeted incrementally lower-volume prescribers. By the time the allocated resources were expended, 4457 physicians had received the e-prescribing offer (

Study Population and Dependent Variable for Physician Adoption Analysis

We retrospectively constructed 2 cohorts, one of physicians who adopted the offered SEP system and another of physicians who did not. Physician adoption was then used as the dependent variable in our adoption analysis. Physicians were also characterized based on their specialty, practice size, prior Horizon BCBSNJ pharmacy claims volume, and assigned primary care patient panel. Horizon BCBSNJ ensured that all patients in their HMO and point-of-service insurance plans had an assigned primary care physician (PCP), whereas all patients with other Horizon BCBSNJ insurance plans did not. To focus our analysis on PCPs, physicians without any assigned Horizon BCBSNJ primary care patients were excluded from the analysis. However, physicians with few Horizon BCBSNJ—assigned primary care patients were included in the analysis, with the expectation that they probably also provided primary care to many non–managed care patients insured by Horizon BCBSNJ.

Dependent Variable for e-Prescribing Use Analysis

The second major goal of our project was to study physician use of SEP among adopters. We calculated an “e-prescribing use ratio” (range, 0-1) by dividing the number of SEP prescriptions the physician wrote for Horizon BCBSNJ patients by the number of Horizon BCBSNJ pharmacy claims attributable to the physician during the same quarter. Each physician’s ratio was calculated for each quarter of the e-prescribing use evaluation period (January 1, 2006, to June 30, 2006). Because pharmacy claims may be generated for prescriptions written before a given quarter, it was possible for a PCP’s ratio to exceed 1; this occurred in particular when total prescription denominators were low. Therefore, all eprescribing use ratios were capped at 1 for the analysis. The ratio numerator included all prescriptions generated through the SEP system whether printed or electronically transmitted. Because we wanted to understand physician behavior rather than patient behavior, electronic renewals (physician behavior) were counted in the ratio numerator, and renewal claims were counted in the ratio denominator. Refills (patient behavior) of existing prescriptions were excluded from the ratio numerator and denominator.

After all PCPs were assigned e-prescribing use ratios, some physicians were also classified as “never having used” the system if records did not show any electronic prescriptions after the day of activation (when test prescriptions were often transmitted). Other PCPs were classified as having “quit e-prescribing” if they had initially used the system but later stopped synchronizing their personal digital assistant and sending any electronic prescriptions by the last quarter of the use evaluation period.

Independent Variables for PCP Characteristics and Patient Panel Data

Caremark provided physician specialty and practice size information, which was available only in previously determined groupings (1, 2-5, 6-10, and >10 physicians). Each physician’s total pharmacy claims volume was provided by Horizon BCBSNJ for 2003 (the calendar year before the start of program recruitment) in categories (0-250, 251-500, etc), which we aggregated into approximate “high,” “medium,” and “low” tercile categories. Physicians were categorized as low-volume prescribers (<1750 Horizon BCBSNJ pharmacy claims in 2003, which represented the 35th percentile), mid-volume prescribers, and high-volume prescribers (>3500 Horizon BCBSNJ pharmacy claims in 2003, which represented the 71st percentile).

We used zip codes to estimate the racial/ethnic makeup of the neighborhoods from which the PCPs’ patients were drawn. First, patients living in zip codes with more than 50% African American residents (per 2000 US Census data) were categorized as living in majority African American neighborhoods; those from zip codes with more than 40% Hispanic residents were categorized as living in Hispanic plurality neighborhoods. An analysis of the studied zip codes showed that these predominantly African American and Hispanic neighborhood categorizations were mutually exclusive more than 99% of the time. The PCPs were then categorized based on having at least 10% of their patients living in majority African American and Hispanic plurality neighborhoods (which represented the 80th and 87th percentiles, respectively).

Our analysis consisted of 2 components. In the first component (SEP adoption analysis), we compared the physician characteristics of adopting PCPs versus control PCPs using t test, χ2 test, and multivariate logistic regression. The second component measured use among adopting PCPs via a 2-part model. The first part was a logistic regression model in which the dependent variable was “never having used” or “quit e-prescribing” (as already described) versus having some evidence of e-prescribing use into the last quarter of the observation period. The objective was to identify factors that predispose physicians to stop e-prescribing. The second part was a multivariate linear regression model that examined the association between physician characteristics and extent of SEP use among the subsample of physicians who had started e-prescribing and did not quit. We excluded “never having used” and “quit eprescribing” physicians to examine ongoing use barriers among those physicians who continued to try to use SEP.

Both regression models were constructed by beginning with all available predictor variables included, and then model fit was manually reassessed after elimination of each nonassociated (P >.05) predictor variable. Model fit was assessed using R2 for the linear regression and C statistic for the logistic regression. Colinearity was assessed using correlation coefficients between all variables remaining in each of the final models. Only complete cases were analyzed in the regression models. All analyses were performed using statistical software (SAS, release 9.1; SAS Institute, Inc, Cary, NC).

PCP Characteristics Associated With SEP Adoption

Of 4457 physicians invited to join the SEP program, 427 adopted the SEP system during 2005 (Figure 1). An additional 64 physicians who adopted the SEP system in 2006 (part way through the use evaluation period) were excluded from our analyses. Of 427 adopting physicians, 297 (69.6%) were classified as PCPs based on their having any primary care patients assigned from a Horizon BCBSNJ managed care plan as of January 1, 2006. There were 3966 control physicians; 1892 (47.7%) had primary care patients assigned to them. Therefore, PCPs participated at a higher rate than non-PCPs. The control and adoption cohorts had medians of 68 and 69 assigned primary care patients, respectively.

 compares the adoption and control cohorts of PCPs and their patients. Practice size distribution differed between the cohorts (

 <.001). Pharmacy claims volume and specialty of the cohorts were not significantly different, nor were sex, age, and household income of the PCPs’ patients. The PCPs in the adoption cohort were less likely to have more than 10% of their patient panel live in majority African American neighborhoods (

Multivariate logistic regression analysis showed that the association of SEP adoption with practice size and neighborhood racial/ethnic makeup persists after controlling for other PCP characteristics (

). No association was found between SEP adoption and PCP pharmacy claims volume or patient panel age, sex, or household income. χ2 Test for physician specialty showed no statistically significant difference between the 2 groups of physicians. When adjustments were made for practice size and race/ethnicity, the regression model indicated that pediatricians were significantly less likely to adopt e-prescribing (odds ratio, 0.61; 95% confidence interval, 0.41-0.92; P = .02). None of the distinct variables in the final model had a correlation coefficient exceeding 0.15.

PCP Characteristics Associated With Extent of e-Prescribing Use

Among 297 SEP-adopting PCPs, the mean (SD) e-prescribing use ratio was 0.23 (0.28) (interquartile range, 0.00-0.39), which represents 23 electronic prescriptions per 100 pharmacy claims (

). Thirty-four PCPs (11.4%) never used the system after it was installed, and another 22 PCPs (7.4%) quit using it after at least some initial use. Among the remaining 241 PCPs, e-prescribing use ratios were less than 0.25 in 141 (58.5%), 0.25 to 0.50 in 51 (21.2%), and greater than 0.50 in 49 (20.3%).

In the first part of our 2-part model, we used logistic regression to estimate the association of physician and patient characteristics with being in the “never having used” or “quit e-prescribing” categories. However, these classifications were not significantly associated with any of our independent variables. In the second part of our 2-part model (

), physician practice size was significantly associated with e-prescribing use ratio (omnibus 

 = .02), although this factor accounted for little of the variance (R2 = 0.01). Among those using e-prescribing at all, PCPs in larger practices (6-10 and >10 physicians) tended to use their e-prescribing systems for fewer of their prescriptions. None of the distinct variables in the final model had a correlation coefficient exceeding 0.15.

SEP has been proposed as an entry-level HIT accessible to most US physicians who work in small practices. Because small practices have been slow to adopt other forms of HIT,

 several initiatives are promoting SEP to this group of physicians.

 Despite the interest, there are limited data about which physicians actually adopt SEP and which patient populations might be affected.

 Furthermore, levels of use are particularly important to understand for SEP because physicians can easily revert to paper prescribing.

Our adoption analysis found that PCPs who adopted SEP were more likely to be in practices with 2 to 10 physicians and were less likely to be pediatricians or have patients living in majority African American neighborhoods. A prior study

 of SEP adoption reported the practice size and specialty of adopting physicians, but these physicians were not compared with nonadopters. Other studies

 of EMR adoption found that adoption increases steadily with increasing practice size.  In contrast, we found that physicians in practices of 2 to 5 physicians and 6 to 10 physicians were equally likely to adopt SEP and that both groups had 1.8 times the odds of adopting SEP compared with solo practice physicians. Because our sample underrepresented large group practices, our confidence interval for this group is large, and we cannot draw any conclusions about the relative likelihood of these physicians to adopt SEP. Nonetheless, our results from small and midsize physician groups suggest that SEP is being adopted by physicians who might otherwise have been unlikely to use HIT.

In a health plan–sponsored e-prescribing initiative, participating PCPs' mean e-prescribing rate was 1 prescription per 4 pharmacy claims, but some PCPs achieved high use.

Adoption and Use of Stand-Alone Electronic Prescribing in a Health Plan-Sponsored Initiative

: To evaluate measurement of physician quality performance, which is increasingly used by health plans as the basis of quality improvement, network design, and financial incentives, despite concerns about data and methodological challenges.

: Evaluation of health plan administrative claims and enrollment data.

: Using administrative data from 9 health plans, we analyzed results for 27 well-accepted quality measures and evaluated how many quality events (patients eligible for a measure) were available per primary care physician and how different approaches for attributing patients to physicians affect the number of quality events per physician.

: Fifty-seven percent of primary care physicians had at least 1 patient who was eligible for at least 1 of the selected quality measures. Most physicians had few quality events for any single measure. As an example, for a measure evaluating appropriate treatment for children with upper respiratory tract infections, physicians on average had 14 quality events when care was attributed to physicians if they saw the patient at least once in the measurement year. The mean number of quality events dropped to 9 when attribution required that the physician provide care in at least 50% of a patientâ€™s visits. Few physicians had more than 30 quality events for any given measure.

: Available administrative data for a single health plan may provide insufficient information for benchmarking performance for individual physicians. Efforts are needed to develop consensus on assigning measure accountability and to expand information available for each physician, including accessing electronic clinical data, exploring composite measures of performance, and aggregating data across public and private health plans.

Available administrative data for a typical health plan may provide insufficient information for benchmarking performance among individual physicians.

For any single quality measure, most physicians had few quality events attributed to them even using a less stringent rule for attributing patient measures to physicians. 

Efforts are needed to encourage the aggregation of databases across public and private health plans and to maximize the data available for characterizing physician perfor

Measurement of physician quality performance is increasingly used by health plans as the basis for quality improvement, network design, and financial incentives.

 Still, efforts to measure physician performance face a number of challenges, in particular the need for sufficient sample size to support reliable measurement and the lack of consensus on methods for attributing patient measures to clinicians.

Researchers have noted that measurement and comparison of physician quality can be hampered by sample size.

 A minimum threshold of 30 patients is a common guideline for supporting comparisons for an individual measure,

 and evidence suggests that at least 35 to 45 observations are needed to make valid comparisons.

 One challenge in obtaining sufficient sample size relates to the measure itself. Many quality measures describe a select group of patients and, by definition, will yield a small number of patients for any physician. Other measures apply to larger proportions of patients, but the ability to capture information on a physician&#8217;s entire panel of patients is limited (as when performance measurement relies on data from a single health plan).

A related issue in quality measurement is attribution. Which physicians should be responsible for a quality measure? Given the current focus on team-based chronic disease care and the reality that most patients receive care from multiple clinicians,

 some authors argue that the most appropriate level of accountability is not the individual physician but rather a formal or informal group of physicians.

 Healthcare organizations often attribute patient quality measures based on utilization or a specific set of services, despite the challenges in identifying which physician should be held responsible for the fulfillment (or lack of fulfillment) of a quality measure.

Efforts are needed to understand how these issues may affect the meaningfulness and soundness of physician profiling efforts. In this study, we used a data set that is typical of the information used by health plans to characterize physician performance. Using 27 well-accepted measures that can be obtained from administrative data, we evaluated (1) how many quality events were available per physician and (2) how different attribution rules affect the number of quality events. 

We focused on 27 measures describing acute, chronic, and preventive care activities performed by PCPs. Only measures that could be obtained through administrative claims data were included. 

) lists all quality measures used in this study, as well as the period used to attribute patients and quality events to physicians.

We identified physicians by the unique identifiers used by each health plan. Primary care physicians, including family physicians, general internists, and general pediatricians, were identified based on their specialty designated in health plan credentialing records.

In selecting an attribution approach, we considered the interactions between clinicians and patients in the course of delivering care, the kinds of services involved, the evidence of a physician&#8217;s involvement in the patient&#8217;s care, and the data sources available. For this study, we applied a measure-specific attribution logic based on administrative data. Measures were attributed to PCPs based on the outpatient visits they provided to patients during a prescribed time frame specific to each measure. Visits were defined using Healthcare Effectiveness Data and Information Set codes for preventive and ambulatory health services.

 To test a less stringent approach to attribution, a patient measure was attributed to a physician if the patient had 1 or more visits during the prescribed time frame. In addition to this &#8220;1-visit&#8221; rule, 2 more stringent rules were assessed: a PCP was attributed responsibility for a patient&#8217;s measure (1) if the patient completed at least 30% of his or her ambulatory visits with that physician (30% rule) and (2) if the patient completed at least 50% of his or her ambulatory visits with that physician (50% rule).

A quality event occurred each time a patient was eligible for a quality measure. Therefore, a single patient could contribute multiple quality events if he or she was eligible for multiple measures (eg, preventive screening and another measure).

Overall, 57% of 170,168 PCPs represented in the study claims data could be attributed responsibility for at least 1 quality event (ie, &#8805;1 of their patients was eligible for &#8805;1 of our selected quality measures). 

 summarizes findings based on the 1-visit rule and describes the percentage of PCPs with more than 30 quality events for a measure. Except for preventive measures, few PCPs had more than 30 observations for any given measure. However, these high-volume providers account for a larger share of quality events overall, particularly for preventive care measures. For example, only 17% of physicians had more than 30 quality events for colorectal cancer screening, but these physicians accounted for 78% of the quality events for this indicator. Only 1% of physicians had more than 30 quality events for annual glycosylated hemoglobin testing among patients with diabetes mellitus, but they accounted for 16% of the quality events for this measure.

 summarizes how moving from a less stringent rule to a more stringent rule for attribution affects the number of patients available for characterizing physician performance. For example, using the 1-visit rule for the measure assessing appropriate care for upper respiratory tract infections in children, physicians on average had 14 eligible patients for that measure in the measurement year. The mean number of quality events dropped to 11 when care was attributed using the 30% rule and to 9 when care was attributed using the 50% rule. Relative to the 1-visit rule, the 50% rule reduced by about half the number of quality events per physician for a measure. Adopting a more stringent rule for a measure also reduced the number of PCPs with at least 1 quality event for that measure (data not shown).

Limitations of this study included the number of measures studied, the reliance on administrative data only, and the lack of direct information about the physician&#8217;s relationship with the patient. These findings are based on only 27 quality measures from administrative data. However, all are well tested and nationally endorsed, and most are included in health plans&#8217; and employers&#8217; physician performance measurement programs for PCPs. Administrative data were used because most physician-level measurement efforts around the country rely on these data. However, administrative data limit the type of clinical actions that can be profiled.

 These limitations demonstrate the issues that health plans often face in developing meaningful provider profiles. Finally, the study describes findings for PCPs only. Within the context of this study, we found similar challenges in achieving sample sizes that provided more than 30 quality events for specialist physicians.

As our results demonstrate, several practical steps are needed to ensure that physician profiles based on administrative data have sufficient information for reliable estimates. First, pooling administrative data within communities across all health plans, government purchasers, and other entities is critical to construct a more complete database representing most or all care rendered by that community&#8217;s physicians. Regional and national quality initiatives promoted by the Centers for Medicaid & Medicare Services14 and by the Robert Wood Johnson Foundation

Second, composite measures should be considered. Care should be taken in selecting and weighting the individual measures for inclusion in a composite. Furthermore, the use of composites creates additional challenges in interpreting quality results and specific actions for improving care. However, composites constructed around a particular condition or patient care activity may provide insights into quality performance and increase the number of quality events available for comparing providers.

Third, efforts are needed to encourage physician practices, health plans, and other entities to make readily available more clinically detailed data for quality measurement construction. Initially, the more widespread availability of electronic data that is already present in some settings (eg, laboratory results) will allow for a larger number of potentially more meaningful quality measures to be constructed without the expense of medical record review, and efforts are critically needed to improve the capabilities of electronic medical records to report quality measures.

 Efforts to augment routinely available administrative data by including additional codes (eg, 

) that capture critical information about outcomes or results of patient care may be promising but have not yet been tested for accuracy or reliability in widespread applications, to our knowledge.

2. Landon BE, Normand SL, Blumenthal D, Daley J

. Physician clinical performance assessment: prospects and barriers. 

4. Hofer TP, Hayward RA, Greenfield S, Wagner EH, Kaplan SH, Manning WG

. The unreliability of individual physician â€œreport cardsâ€ for assessing the costs and quality of care of a chronic disease. 

. Measuring patientsâ€™ experiences with individual primary care physicians: results of a statewide demonstration project. 

8. Pham HH, Schrag D, Oâ€™Malley AS, Wu B, Bach PB

. Care patterns in Medicare and their implications for pay for performance. 

http://www.ingenix.com/Products/Employers/HealthandProductivity/EvidenceBased

. Reliability of individual and composite measures for profiling physician performance. 

. A comparison of administrative-only versus administrative plus chart review data. 

. The Robert Wood Johnson Foundation: health and health care improvement. 

17. Casalino LP, Elster A, Eisenberg A, Lewis E, Montgomery J, Ramos D

. Will pay-for-performance and quality reporting affect health care disparities? 

. Pay for performance at the tipping point. 

Typical health plan data provide limited information for benchmarking physician performance using even a less stringent rule for attributing patient measures to physicians.

Availability of Data for Measuring Physician Quality Performance

Objective: To examine the reliability of quality measures to assess physician performance, which are increasingly used as the basis for quality improvement efforts, contracting decisions, and financial incentives, despite concerns about the methodological challenges.

Study Design: Evaluation of health plan administrative claims and enrollment data.

Methods: The study used administrative data from 9 health plans representing more than 11 million patients. The number of quality events (patients eligible for a quality measure), mean performance, and reliability estimates were calculated for 27 quality measures. Composite scores for preventive, chronic, acute, and overall care were calculated as the weighted mean of the standardized scores. Reliability was estimated by calculating the physician-to-physician variance divided by the sum of the physician-to-physician variance plus the measurement variance, and 0.70 was considered adequate.

Results: Ten quality measures had reliability estimates above 0.70 at a minimum of 50 quality events. For other quality measures, reliability was low even when physicians had 50 quality events. The largest proportion of physicians who could be reliably evaluated on a single quality measure was 8% for colorectal cancer screening and 2% for nephropathy screening among patients with diabetes mellitus. More physicians could be reliably evaluated using composite scores =17% for preventive care, >7% for chronic care, and 15%-20% for an overall composite).

Conclusions: In typical health plan administrative data, most physicians do not have adequate numbers of quality events to support reliable quality measurement. The reliability of quality measures should be taken into account when quality information is used for public reporting and accountability. Efforts to improve data available for physician profiling are also needed.

)When health plan administrative data are used to evaluate physician performance, most quality measures require at least 50 quality events per physician to gain a reliable estimate of physician performance (ie, to ensure that a quality measure is able to distinguish a physician’s performance from average performance).

Composite measures allow more physicians to be evaluated reliably but are less actionable for quality improvement.

The physician-level reliability of quality measures should be considered when quality information is used for public reporting and accountability.

Efforts to improve the quality and quantity of data available for physician profiling are also needed.

Measuring physician performance is becoming commonplace as health plans and purchasers look for ways to drive quality improvement and to increase physicians’ accountability and rewards for achieving quality goals. A recent study

 reported that, among 89% of health maintenance organization plans using physicianoriented pay-for-performance programs, more than one-third measured and rewarded quality at the individual physician level. In addition, public and private purchasers are demanding more information about America’s physicians and hospitals to aid in value-based purchasing and selection of health plans and providers.

However, concerns remain regarding the validity and reliability of such physician performance profiles. Several factors are needed to support fair and accurate comparisons among physicians. These include evidence-based quality measures, complete and accurate data sources, and standardized methods of data collection. Physician-level reliability of a quality measure is another key consideration in this measurement. Physician-level reliability refers to the ability of a quality measure to distinguish an individual physician’s performance from the performance of physicians overall. Good physician-level reliability requires the following 2 factors: (1) a sufficient number of patients eligible for a given quality measure and (2) performance variation across physicians on that quality measure.

 The greater the number of a physician’s patients who are eligible for a quality measure, the more precise the estimate of the physician’s performance. When performance variation for a given quality measure across physicians is limited, the likelihood that a physician’s performance is statistically significantly different from that of his or her peers is also decreased. Hofer and colleagues6 showed that not controlling for a quality measure’s physician-level reliability significantly misrepresented performance differences across physicians. However, adjusting performance profiles in such a manner is not commonplace across the healthcare industry.

Ensuring that measurement results are valid and reliable is important when purchasers and plans (and potentially consumers) use the data to make decisions about which physicians get financial rewards or other benefits. The stakes are particularly high when profiling results are used for public reporting or eligibility for participation in a health plan network. Paying attention to the validity and reliability of data will help to ensure that these decisions are based on real differences in performance among physicians rather than any shortcomings of the measurement.

Although performance results based on limited sample sizes could be adjusted for the reliability of individual measures, 

 the creation of composite scores may also be a useful way to increase the reliability of physicians’ performance scores.

 Little is known about the extent to which constructing composite scores mitigates the limitations of sample size and reliability, while continuing to provide useful and understandable information.

To date, there have been few reports regarding the reliability of physician-level performance scores associated with commonly used practices and methods in the healthcare industry. To begin to address this deficiency, this study relied on a large data set that combined patient-level administrative data from 9 large health plans to compute performance for primary care physicians (PCPs) using 27 commonly measured quality indicators. This data set is typical of data sources often used by individual health plans to profile physician performance. Specifically, we examined for each quality measure and composite score the proportion of PCPs who could be evaluated given different minimum sample size criteria and the physician-level reliability under those minimum sample size criteria. Our primary research questions were the following: (1) What is the physician-level reliability of commonly used performance measures calculated exclusively based on administrative data? (2) Can more physicians be reliably evaluated using a composite score?

This study used administrative data from the Ingenix Impact Pro database.

 Deidentified claims and enrollment data for individuals enrolled in 9 health plans from 9 separate geographic regions for 2003 and 2004 were available for this study. Each of these plans had at least 250,000 members and accounted for 15% to 50% of managed care enrollees in their markets (Table 1). In all, these plans covered more than 11 million unique members and many physicians and employer groups. The members included in these organizations were primarily enrolled in commercial health maintenance organization, preferred provider organization, and point-of-service health plan product designs, with fewer individuals enrolled in Medicare risk products. Pharmacy benefit status, an indicator of the general availability of pharmacy data to support measurement, ranged from 51% to 80% of the enrolled populations for each plan. Although the study population was drawn from multiple geographic census regions, most individuals were located in the northeast United States. The data were deidentified to protect patient, physician, and organization confidentiality. This study was reviewed and determined to be exempt by Chesapeake Research Review, Inc (Columbia, MD).

Because the Impact Pro database may not include complete data on all services (eg, pharmacy, laboratory, or mental health services) needed for calculating some performance measures, we conducted specific analyses to assess the completeness of the data available for the study. Using only administrative data sources, we compared performance rates based on Impact Pro data with performance data reported to the National Committee for Quality Assurance (NCQA) through the Healthcare Effectiveness Data and Information Set (HEDIS) reporting. If we found more than a 5—percentage point difference between the plan’s reported rate to the NCQA and the rate in the Impact Pro database, the data were excluded for that quality measure.

Twenty-seven quality measures often used to assess care effectiveness were calculated using study data following HEDIS specifications.

 The quality measures were identified from an environmental scan of existing physician quality measures and prioritization by an NCQA expert panel on physician profiling. The quality measure set primarily includes quality measures that have been endorsed by the AQA Alliance and the National Quality Forum. Only quality measures that could be obtained through administrative claims data were included because we were emulating efforts to profile physicians based on data commonly available and used by health plans. Quality measures for diabetes care, cervical cancer screening, and colorectal cancer screening are specified for hybrid data collection for HEDIS (ie, using medical records data to supplement claims). Relying exclusively on administrative data for performance calculations for these quality measures may not accurately reflect performance.

 The selected quality measures describe preventive, chronic, and acute care activities and were considered appropriate for supporting comparisons of PCPs. (See eAppendix Table 1, available at 

Identification and Attribution of Quality Events to Physicians We identified individual physicians using the unique physician identifiers used by health plans. Because we did not have a way to link a physician’s claims in one data set to that physician’s claims in another health plan’s data set, we did not pool patients for the same physician across health plans. Most of the 9 health plans in our study did not operate in the same healthcare markets, so pooling would have a limited effect. Primary care physicians, including family physicians, general internists, and general pediatricians, were identified based on the specialty designated in the credentialing records of the participating health plans.

The 9 health plans included in this study generally did not require patients to designate a PCP. Therefore, we developed algorithms based on patient care patterns to attribute a patient’s care to 1 or more physicians. We required at least 1 claim for an outpatient visit during the measurement period to attribute a patient to a PCP for inclusion in the study. This means that some patients who were eligible for quality measures may not have been attributed (eg, a woman eligible for mammography would not be attributed if she did not have a qualifying visit during the year). Outpatient visits were defined based on coding conventions established through HEDIS to identify preventive and ambulatory health services.

For a physician to be considered responsible for a quality event (defined as an event for which a patient is eligible for a quality measure), the patient had to have a visit with the physician during a period when the physician would have an opportunity to meet the quality indicator. We chose this less stringent approach to maximize the number of quality events assigned to each physician. Any PCP rendering 1 or more visits for a patient during the eligibility period was considered responsible for the quality measure. A specific patient may be eligible for multiple quality measures and contribute multiple quality events for the responsible physicians. Likewise, more than 1 physician could be responsible for a specific quality event.

Individual Measures. For each quality measure, a physician’s rate of performance was computed as the number of patients meeting indicator requirements divided by the number of eligible patients attributed to that physician. In preliminary analyses, we observed high variation in performance rates among physicians with fewer than 10 attributed quality events for a quality measure, so we excluded these physicians from the analysis. We determined the number of physicians meeting 4 different volume thresholds (≥10, ≥20, ≥30, and ≥50 quality events) for each quality measure. We also report the mean number of quality events per physician within each volume category and the mean performance rates for each category. The volume categories are not mutually exclusive; a physician with 25 quality events for a quality measure will be included in the volume categories for 10 or more and for 20 or more.

To determine how well quality measures capture differences in physician performance, we calculated a reliability estimate for each quality measure.

 The methods are summarized herein (see eAppendix. Reliability Formula, available at 

). Reliability estimates were calculated for physicians within each plan because we wanted the variability between physicians to reflect the natural variation within a plan and not to be unduly affected by patient or other factors that might differ across plans. We selected 2 plans (plans C and I in Table 1) for inclusion in our presented findings to illustrate potential differences across plans in reliability results for different performance measures. The estimated reliability for a quality measure was computed as the ratio of the physician-to-physician variance divided by the sum of the physician-to-physician variance plus the measurement variance. This reliability estimate varies between 0 and 1, and a reliability of 0.70 or higher is typically considered acceptable for psychometric purposes.

 Reliability estimates were computed separately for each of the volume categories based on the mean number of quality events for physicians within that category and on the minimum number of quality events. For example, in the category for 30 or more quality events, all physicians have at least 30 quality events, but some have many more than 30, which tends to elevate the mean reliability. The reliability at the minimum for that volume category reflects the reliability for physicians who had no more than 30 quality events. The reliability estimate at the mean number of quality events for the quality measure and volume category reflects typical experience of physicians in this population. The reliability at the minimum number of quality events per physician shows the “worst case” reliability for the volume category. The reliability estimate at the minimum level of quality events also facilitates comparison of reliability estimates across quality measures because the mean number of quality events varies by quality measures. To promote comparability across volume categories for a quality measure, the physician-to-physician variance for the group of physicians with 10 or more quality events was used for computing reliability for all groups. The Spearman-Brown prophecy formula was used to produce the reliability estimate at each volume level.

 We also present the distribution of reliability estimates by the number of quality events per physician.

At least 50 quality events per physician are needed to reach a minimum level of reliability for most quality measures calculated from administrative data.

John L. Adams, PhD

Articles

Electronic Health Record Adoption and Quality Improvement in US Hospitals

Adoption and Use of Stand-Alone Electronic Prescribing in a Health Plan-Sponsored Initiative

Availability of Data for Measuring Physician Quality Performance

Benchmarking Physician Performance: Reliability of Individual and Composite Measures