To evaluate the accuracy and precision of random sampling in identifying healthcare system outliers in diabetes performance measures.

Cross-sectional analysis of 79 Veterans Health Administration facilities serving 250 317 patients with diabetes mellitus between October 1, 1999, and September 30, 2000.

Primary outcome measures were poor glycosylated hemoglobin (A1C) control and good low-density lipoprotein cholesterol (LDL-C) and blood pressure (BP) control. Facility performance for each measure was calculated using 150 separate random samples and was compared with results using the bootstrap method as the criterion standard for determining outlier status (defined as a ≥ 5% difference from the mean, within the 10th or 90th percentile, or ≥ 2 SDs from the mean).

The study population was largely male (97.4%), with 54.0% of subjects being 65 years or older. The facility-level mean performances were 22.8% for poor A1C control, 53.1% for good LDL-C control, and 55.3% for good BP control. Comparing the random sampling method with the bootstrap method, the sensitivity ranged between 0.64 and 0.83 for the 3 outcome measures, positive predictive values ranged between 0.55 and 0.88, and specificity and negative predictive values ranged between 0.88 and 0.99.

The specificity and negative predictive value of the random sampling method in identifying nonoutliers in performance were generally high, while its sensitivity and positive predictive value were moderate. The use of random sampling to determine performance for individual outcome measures may be most appropriate for internal quality improvement rather than for public reporting.

suggested that a key feature of information dissemination to stakeholders is the use of industry benchmarks based on the "evaluability principle."

In this conceptual model, information is most likely to be used by all stakeholders if it can distinguish between best and worst performers. Furthermore, it was recommended that findings would be most useful if their robustness could be evaluated by using a wide range of assumptions or methods. Despite these recommendations, hospitals, managed care health plans, states, and federal agencies are increasingly labeled as best or worst performing units based on a variety of publicly reported performance measures, despite concerns about the accuracy of "report cards."

Random sampling strategies are used to generate data for report cards and are dependent on the comparisons that are to be made.

For example, if the objective of a report card is to provide a broad overview of quality, simple random sampling might suffice, whereas stratified random sampling might be necessary to compare quality of care among multiple patient subgroups.

The National Committee for Quality Assurance (NCQA) is a leader in measuring the quality of ambulatory care in the United States.

Managed care health plans are evaluated on sets of composite measures to determine comparative scoring on a variety of general prevention and disease-specific indicators as part of a rigorous accreditation process, with results made available on a public basis as part of the NCQA Quality Compass report. The NCQA recommends a simple random sampling strategy for data collection within separate product lines of healthcare plans. However, in contrast to physician report cards,

the statistical certainty with which managed care health plans can be identified as outliers on performance of individual ambulatory care measures is not as well studied. We reasoned that the single random sampling method used to determine the outlier status of a plan might not necessarily reflect the overall composition of the population.

The Veterans Health Administration (VA) is a nationwide integrated healthcare system with semiautonomous facilities that function in many ways like managed care health plans.

We used the VA's automated clinical, pharmacy, and laboratory data

to evaluate the statistical certainty of the current practice of using a single random sample to evaluate managed care health plan outliers in performance, defined as the percentage of the eligible population meeting the measure specifications. We constructed facility-level performance measures for glycemic, lipid, and blood pressure (BP) control for persons with diabetes mellitus using different dichotomous thresholds. To examine the accuracy of the random sampling approach (the ability of random samples to correctly identify the outlier status), we compared performance measures derived from a single random sample with those that were derived among all facility-level patients using a more rigorous statistical technique (the bootstrap method) as the criterion standard. We evaluated 3 different approaches to random sampling (unadjusted, age stratified, and age standardized). To examine the precision of the random sampling approach (the ability of random samples to repeatedly identify the outlier status), we redrew 150 separate random samples and examined the effect of age adjustment and measure thresholds on facility outlier classification.

Inpatient and outpatient utilization data and 

International Classification of Diseases, Ninth Revision,

codes were obtained from the National Patient Clinical Dataset (Austin, Tex), and medication data were provided by the VA's Healthcare Analysis Information Group.

We identified veterans as having diabetes mellitus in fiscal year 1999 if they had more than 1 diabetes diagnosis code (codes 250.xx, 357.2, 362.0, and 366.41) associated with separate calendar days or any diabetes-specific medication prescription (insulin, sulfonyureas, biguanides, alpha glucosidase inhibitors, and thiazolidinediones). We obtained utilization, pharmacy, laboratory, and death data on these veterans in fiscal year 2000 (October 1, 1999, through September 30, 2000).

To approximate the NCQA criteria for indemnity plan member inclusion in the Health Plan Employer Data and Information Set reporting, we included veterans who had at least 1 faceto- face visit with a clinician during the reporting period and were alive on the last day of the reporting period.

 The VA New Jersey Health Care System Institutional Review Board approved this study.

To minimize confounding of outlier status by variation in glycosylated hemoglobin (A1C) test results due to laboratory assay method variation, we restricted our analyses to those facilities using assays certified by the National Glycohemoglobin Standardization Program.

 We identified 125 facilities, with 397 147 veterans who met our criteria for evaluation of diabetes care during the observation period. We eliminated 11 facilities with less than 1000 patients. We then excluded facilities that were more than 4 SDs from the means with respect to the proportion of A1C tests (25 facilities), low-density lipoprotein cholesterol (LDL-C) level (5 facilities), or BP level (5 facilities) to avoid identification of outliers that were likely due to incomplete data collection attributable to the semiautomated extraction programs that were used for administrative data collection, as previously reported.

These exclusion criteria reduced the number of facilities from 125 to 79. The final study sample comprised 250 317 patients, with a facility-level range of 1015 to 13 598 patients per facility.

We used the accountability performance measures for diabetes care, reflecting glycemic, lipid, and BP control based on those developed by the Diabetes Quality Improvement Project

and reported by the NCQA in the 2000 Diabetes Comprehensive Care measures.

 Poor A1C control was defined as the proportion of the population with an A1C test not performed or with an A1C level of 9.5% or higher. Good LDL-C control was defined as the proportion of the population with an LDL-C test performed and with an LDL-C level less than 130 mg/dL (< 3.37 mmol/L). Good BP control was defined as the proportion of the population with BP less than 140/90 mm Hg. We also constructed performance measures using thresholds of 8.5% or higher for A1C level, less than 100 mg/dL (< 2.59 mmol/L) for LDL-C level, and less than 130/80 mm Hg for BP to more closely approximate professional society guideline recommendations during the study period.

All 6 performance measures were based on the last recorded values.

We followed the Health Plan Employer Data and Information Set guidelines for calculation and sampling to draw the random samples for this study.

Based on the binomial theorem, a sample size of 411 subjects provides 90% power for detecting differences when the mean percentage of patients meeting a measure is 50%, which is justified as a tradeoff between statistical certainty and the cost of obtaining data from medical record review. Therefore, we drew 150 separate single random samples of 411 subjects from each facility among the entire facility population to be used for our primary analyses.

The NCQA requires commercial managed care health plans to collect and report results separately by populations covered by different product lines, including Medicare. Consequently, we used 2 other random sampling strategies that involved age adjustment (age-stratified random sampling and age-standardized random sampling) by randomly sampling within 2 age strata (≥ 65 vs < 65 years old). In the first strategy (age stratified), we sampled from the 2 age strata according to the proportions at the facility level to ensure that the proportions of patients in the age categories were the same as the proportions in that facility.

In the second strategy (age standardized), we sampled from the 2 age strata to ensure that the proportions of patients in the age categories were the same as the proportions in the entire study population.

To create a criterion standard that most likely represented a facility's true performance measure, we used the bootstrap method. Bootstrapping is a computer-intensive method that resamples from the original data to create a large number of bootstrap samples to cover various events of random chance. It has been proven to be an effective tool in statistics to estimate parameters and to assess variability.

In our context, 1000 bootstrap samples, each consisting of 411 randomly drawn subjects with replacement, were obtained for a given facility. The bootstrap mean was calculated for each of the 6 performance measures. Because bootstrap means are highly accurate assessments of the corresponding true performance measures, they were treated as if they were the true values. We repeated the same bootstrap method for the age-stratified and age-standardized sampling approaches.

The outlier status for each facility on each of the performance measures was determined using 3 separate strategies for each of the 150 individual random samples with and without age adjustment, as well as for the bootstrap method. In the first strategy, an individual facility was considered an outlier if its performance measure percentage was different by at least 5% from the performance mean across facilities, which is a strategy recommended by the NCQA.

In the second strategy, an individual facility was considered an outlier if its performance measure was within the 10th or 90th percentile of rank order (the best performers were in the 10th decile), which is used by the NCQA Quality Compass and by others.

In the third strategy, based on determination of the 95% confidence interval, an individual facility was considered an outlier if its performance measure was 2 or more SDs from the mean.

Within each random sampling strategy, the facility outlier status calculated from each of the 150 individual random samples was compared with the outlier status determined using the bootstrap method for each performance measure. We defined accuracy as the ability of a single random sample to correctly identify the bootstrap outlier status. For each random sampling strategy, we calculated 4 evaluation measures (sensitivity, specificity, positive predictive value [PPV], and negative predictive value [NPV]) for each performance measure for each of the 3 outlier definitions.

Sensitivity was defined as the probability of a facility being identified as an outlier by a random sampling method, given that it is an outlier facility (according to the bootstrap method). Specificity was defined as the probability of a facility being identified as not being an outlier, given that it is not an outlier facility. Positive predictive value was defined as the likelihood that a facility is an outlier using the bootstrapping method given that it was identified by random sampling. Negative predictive value was defined as the likelihood that a facility is not an outlier using the bootstrapping method given that it was not identified by random sampling. We determined the ranges of the sensitivity, specificity, PPV, and NPV of the random sample outlier status compared with the bootstrap outlier status using each of the 3 random sampling approaches and the 3 outlier approaches for each of the 6 performance measures. We defined precision as the ability of the single random samples to repeatedly identify the outlier status as determined by the bootstrap method. Therefore, we reported the ranges of sensitivity, specificity, PPV, and NPV from the 150 separate samples for each random sampling approach for each measure.

gives the distribution of patient characteristics and diabetes care measures by facility inclusion status. The study population was largely male (97.4%), with 54.0% of patients being 65 years or older. The mean ± SD A1C level was 7.6% ± 1.7%, the LDL-C level was 107.4 ± 32.6 mg/dL (2.78 ± 0.84 mmol/L), the systolic BP was 139.0 ± 21.0 mm Hg, and the diastolic BP was 73.7 ± 12.1 mm Hg. The patients from the excluded facilities had similar demographic characteristics and mean values for A1C level, LDL-C level, and BP, but as expected they had lower percentages of A1C and LDL-C tests performed.

Facility-level performance was determined for individual diabetes care measures, presented as the overall facility mean and by percentile of rankings (

). The mean facility rates of poor A1C control were 22.8% (using a ≥ 9.5% threshold) and 33.0% (using a ≥ 8.5% threshold). The mean facility rates of good LDL-C control were 53.1% (using a < 130 mg/dL [< 3.37 mmol/L] threshold) and 29.8% (using a <100 mg/dL [<2.59 mmol/L] threshold). The mean facility rates of good BP control were 55.3% (using a <140/90 mm Hg threshold) and 32.7% (using a <130/80 mm Hg threshold). There was substantial variation among the facilities, with 2-fold to almost 3-fold differences between the best and worst performers. For A1C level, the worst performing facility had 34.7% of its patients with diabetes mellitus with A1C levels of 9.5% or higher, and the best performing facility had just 13.2%. For LDL-C level, the worst performing facility had 32.3% of its patients with diabetes mellitus with LDL-C levels less than 130 mg/dL (< 3.37 mmol/L), and the best performing facility had 68.2%. For BP, the worst performing facility had 43.2% of its patients with diabetes mellitus with BPs less than 140/90 mm Hg, and the best performing facility had 67.8%. Variations were similarly large across the lower threshold performance measures. These results demonstrate a difference of 20 and 30 percentage points between the best and worst performing VA facilities for each measure and a difference of 15 to 20 percentage points between the facilities in the lowest and highest quartiles. These differences are indicative of clinically meaningful variation in care provided among outliers.

Based on the bootstrap criterion and using the decile method (whether a facility was within the 10th or the 90th percentile of rank order), there were 16 outlier facilities (20%).Using the outlier identification strategies of 5% or greater difference from the mean and 2 SDs from the mean, respectively, the numbers of outlier facilities were 4 (5%) and 6 (8%) for A1C level of 9.5% or higher, 9 (11%) and 7 (9%) for LDL-C level less than 130 mg/dL (< 3.37 mmol/L), and 29 (37%) and 3 (4%) for BP less than 140/90 mm Hg (Table 2).

shows the means and ranges of the sensitivity, specificity, PPV, and NPV for the random sampling method without age adjustment compared with the bootstrap method for A1C level of 9.5% or higher, LDLC level less than 130 mg/dL (< 3.37 mmol/L), and BP less than 140/90 mm Hg for each of the outlier status methods (≥ 5% difference from the mean, ≥ 2 SDs from the mean, and the decile method). Overall, random sampling had fairly acceptable accuracy and precision for detecting nonoutlier status. Specificity was fairly high (range, 0.88-0.99 across measures), with a fairly narrow range on 150 repeated samples (usual range, 0.9-1.0 for most performance measures but as low as 0.63 for BP outlier status using ≥ 5% difference from the mean). Similarly, NPVs had uniformly high means (range, 0.88-0.99) across performance measures, thresholds, and outlier determination strategies, with fairly narrow ranges on repeated sampling (usual range, 0.9-1.0).

However, the mean sensitivity and the PPV for the 150 random samples were significantly lower, with broader ranges for individual samples. The means for sensitivity varied between 0.73 and 0.80 (range, 0.20-1.00) for A1C level, between 0.64 and 0.83 (range, 0.30-1.00) for LDL-C level, and between 0.64 and 0.75 (range, 0.30-1.00) for BP. Similarly, the mean PPVs varied between 0.55 and 0.67 (range, 0.27-1.00) for A1C level, between 0.68 and 0.88 (range, 0.33-1.00) for LDLC level, and between 0.77 and 0.86 (range, 0.33-1.00) for BP.

The low sensitivity and PPV were not a result of the method used to determine outlier status, although results varied slightly depending on the strategy used. For example, for A1C level of 9.5% or higher, the 5% or greater difference from the mean approach had the largest difference between sensitivity and PPV, and the best and worst decile method had similar sensitivity, specificity, PPV, and NPV. In contrast, for LDL-C level less than 130 mg/dL (<3.37 mmol/L), the best and worst decile method had different sensitivity, specificity, PPV, and NPV. For BP less than 140/90 mm Hg, the 5% or greater difference from the mean approach to outlier detection was the least favorable of the 3 approaches, and the decile method had similar results. Our results did not differ when age-stratified or age-standardized random sampling strategies were used or when lower performance measurement thresholds (≥ 8.5% for A1C level, <100 mg/dL [<2.59 mmol/L] for LDL-C level, and <130/80 mm Hg for BP) were used (data available from the author).

The specificity and NPV of the random sampling method in identifying nonoutliers in performance were generally higher, with narrow ranges, than its sensitivity and PPV. Therefore, the random sampling method was more accurate and precise in identifying facilities that were not outliers than in identifying facilities that were outliers with respect to performance on glycemic, cholesterol, and BP control. Neither different random sampling approaches that took age into account, nor lower thresholds for glycemic, cholesterol, and BP control, consistently improved the sensitivity or the PPV of the random sampling method in the identification of outliers. These findings were consistent across 3 different methods of outlier detection—identification of at least a 5% difference from the mean, at least 2 SDs from the mean, and ranking in the 10th or 90th percentile of facilities. The inconsistency of the random sampling approach in identifying outlier status compared with a criterion standard derived from the entire population contributes to a growing body of literature raising concerns about the increasingly widespread practice of ranking hospitals, health plans, and physician groups.

Our findings can perhaps be attributed to the possibility that several factors beyond the control of the healthcare plan are not randomly distributed among facilities and could result in biases among random samples using administrative data. For example, veteran clinical users may have enrollment in multiple systems of care,

accounting for discrepancies between medical record abstraction and administrative data.

However, care in multiple systems would not bias the findings that we observed with BP measurements, because all BP readings were based solely on VA data. Although age and race biases in the availability of administrative data have been reported in the private sector,

the automated laboratory databases used by the VA at the facility level render this an unlikely explanation for our findings. Furthermore, our findings were similar when we adjusted for age effects by using stratified sampling and age standardization. Patient-level factors can also contribute to facility variation in performance. For example, factors under provider management, such as shared decision making, can affect glycemic, lipid, and BP control.

However, factors extrinsic to a healthcare system, including patient age, duration of diabetes mellitus, and medical comorbidities account for as much as 10% of the variation in A1C levels

Therefore, it is possible that risk adjustment could improve the accuracy and precision of the random sampling method. However, risk-mix adjustment to facilitate comparison among healthcare plans has not yet been implemented for use in public reporting,

 although it is recommended in the health services literature

 and is used in a voluntary physician recognition program in which qualifying physicians may receive a bonus for each patient with diabetes mellitus covered by a participating employer or health plan.

The use of composite measures may also improve the PPV of random samples, but if there is a small difference among outliers, variations in patient mix among the random samples may still result in large changes in outlier status based on an "all or none" threshold.

The effect of the accuracy and precision in outlier identification will relate to the context in which the information is to be used and in the consequences that could result from its use.

For example, if the objective is to evaluate nonoutliers with reasonable certainty for the purpose of focusing internal quality improvement strategies by identifying possible best practices,

then our results suggest that the use of single random samples is an appropriate statistical method based on its high specificity and NPV.

However, if the objective of report cards is to identify best and worst performing healthcare plans or large physician groups that could affect direct marketing to the public and influence consumer choice, then accrediting and payer systems may consider evaluating the accuracy and precision for the extreme outliers separately, providing confidence intervals for the results. Although we recognize the need for a balance between public reporting

and statistical uncertainty, recent public sector

Web site reports do not present sufficient information to evaluate the statistical confidence with which outliers are ascertained.

Strengths of this study include the VA's standardized national data repositories for laboratory and clinical results, which permitted the construction of performance measures that identified "actual" outliers. Furthermore, the data sets and measures used herein have been studied for reliability

and had already been used at the time of the study.

We also used only National Glycohemoglobin Standardization Program-certified A1C test results to minimize variation attributable to A1C laboratory assays.

We also recognize several limitations, including the facts that the VA population is homogeneous and that variations among VA facilities may be less than those among private sector plans.

However, greater variation in outlier determination may well be seen in settings with more heterogeneity.

In conclusion, the specificity and NPV of the random sampling method in identifying nonoutliers in performance on individual diabetes care measures were generally high and precise, while its sensitivity and PPV in identifying outliers were moderate, with less precision. We suggest that random sampling may not meet the level of certainty necessary for the objective of external reporting (to consumers and payers) as opposed to internal feedback (to plans and physicians) for individual diabetes care measures. Our findings add to the increasing debate about the effect of public reporting on consumer, physician, and health plan behavior.

Therefore, we recommend that issuers of report cards provide greater transparency in the certainty of identifying best and worst performing plans (or providers), consistent with recommendations from the Institute of Medicine.

We thank Christina Croft, BA, for assistance in the preparation of the manuscript.

From the Center for Healthcare Knowledge Management, VA New Jersey Health Care System, East Orange (LP, CLT, MM, MR, AT, DH), University of Medicine and Dentistry of New Jersey-New Jersey Medical School, Newark (LP, CLT, DH), and Rutgers University, New Brunswick (MX, YS, JK, SC); and the Deep South Center on Effectiveness at the Birmingham VA Medical Center and University of Alabama at Birmingham, Birmingham (MS).

This study was supported by grant IIR 00-072-1 from the Department of Veterans Affairs Health Services Research and Development Service, Washington, DC (LP); and by grant NSF SES-0241859 from the National Science Foundation, Arlington, Va (MX). The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs and NSF.

Address correspondence to: Leonard Pogach, MD, MBA, Center for Healthcare Knowledge Management, VA New Jersey Health Care System, 385 Tremont Avenue (111-Medicine), East Orange, NJ 07018. E-mail: leonard.pogach@med.va.gov.

1. Hurtado MP, Swift EK, Corrigan JM, eds. 

Designing the National Healthcare Quality Report. In: 

Envisioning the National Healthcare Quality Report.

 Washington, DC: Institute of Medicine, National Academy Press; 2001.

Articles

MANAGERIAL

Diabetes Healthcare Quality Report Cards: How Accurate Are the Grades?

To document the presence and functioning of differentpractice systems in a small sample of medical groups inMinnesota and to examine the relationship between the presenceof practice systems and prior adoption of an electronic medicalrecord (EMR).

Descriptive study of the frequency of practicesystems in 11 medical groups.

We recruited 11 medical groups for the study. Fourgroups had an EMR; the other groups used paper medical records,often supplemented by electronic ordering or data systems. Usingan on-site audit team, we validated the presence of practice systemsorganized under 8 categories.

All of the medical groups had implemented a substantialnumber of practice systems for care management of patientswith chronic conditions. Although the medical groups with an EMRtended to have more comprehensive practice systems in place,the medical groups without an EMR also had most of the practicesystems.

Although required in some functions, an EMR maynot be necessary in facilitating practice systems that support consistentmanagement of patients with chronic illness. Approachesare needed that will encourage the implementation of practice systemsin medical groups with and without an EMR.

The need for major improvements in medical carein the United States has become clear since therelease of important reports from the Institute ofMedicine

and the publication of a study of the qualityof national healthcare by McGlynn et al.

This need isparticularly important in the care of patients withchronic disease, an area that will become even morecritical as the population of older persons with theseconditions increases.

As illustrated by the chronic caremodel (the dominant conceptual framework for effectivecare of chronic conditions), care delivery organizationsof all sizes must implement practice systems toprovide consistent and comprehensive care.

refers to organized processes designedto assure that certain information or services are collectedor provided routinely to patients or to healthcarepersonnel (eg, reminders, test results, and education).

The problem is that few of these practice systems arein place, even in larger medical groups. Casalino et al

studied the presence of practice systems (termed 

[CMPs] by them) in 1040 physicianorganizations with 20 or more physicians, medicalgroups that should have the resources to implementsuch support for delivery of quality healthcare. Of a possible16 CMPs, they found a mean of only 5 CMPs permedical group. Because the presence of external incentivesand clinical information technology systems wasstrongly associated with CMP use, they suggested thatproviding these might increase the use of CMPs.

and other studies by Casalinoet al, there is little information in the medical literaturedocumenting the extent to which practice systems arepresent in medical groups. Moreover, the sparse informationthat is available about practice systems is neitherdetailed nor verified by on-site audits. As part of astudy testing the validity of a new questionnairemethod to measure the presence of practice systems forthe care of patients with chronic disease, we documenteddetailed information about practice systems among11 medical groups in Minnesota through self-assessmentand on-site audits. In addition, we wanted to learnwhether medical groups with an electronic medicalrecord (EMR) were more likely to have such practicesystems than medical groups without an EMR.

This study was conducted in Minnesota in collaborationwith the Institute for Clinical Systems Improvement,a quality improvement collaborative that includes mostof the medical groups and hospitals in the state amongits members. At the time of this study, those memberorganizations included about 75% of the physicians inMinnesota.

We obtained contact information from theInstitute for Clinical Systems Improvement for 19 oftheir 38 medical group members who provide primarycare to adults, specifying only that we wanted to recruitmedical groups with a diversity of locations, sizes, andsophistication about quality improvement methods.

Recruitment was conducted by first sending a letterdescribing the study to the medical director (or equivalent)of each medical group, followed by telephone callsfrom one of us (LIS) until each medical group had decidedwhether to participate. Three medical groupsdeclined participation (each on the grounds of havingtoo much activity or turmoil at the time), and 2 medicalgroups agreed too late to be included. Three medicalgroups participated in pretesting the survey and the onsiteaudit, leaving 11 medical groups with completeinformation for this report.

This article is based on information gathered in onsiteaudits conducted by 2 trained and experiencednurse auditors. The auditors met with each participatingmedical group's quality improvement director andother staff for assessment of information about whichthey had particular knowledge. The on-site audit coveredthe following 8 practice systems and their componentprocesses: (1) continuity of care (a system tomaintain an ongoing and effective relationship betweenan individual clinician and a group of healthcare practitionersinvolved in providing care for a given patient),(2) registry (an organized system that allows the officeor clinic to group patients by diagnoses and otherparameters and uses the groupings to assist in the provisionof care), (3) clinical information (systems andprocesses associated with a database of key patient andpatient population information that can help managepatient care), (4) systematic monitoring (the use of adatabase to monitor key indicators of chronically illpatients' medical conditions for information that mayrequire immediate attention), (5) clinician reminders(special communications intended to help the office orclinic team adhere to best practices related to the careof the individual patient), (6) performance tracking andfeedback (the process of using clinical information systemsto aggregate key indicators culled from a patientregistry or other data source for the purposes of benchmarkingperformance and directing improvement activities),(7) clinical quality evaluation and improvement(a formal process to assess care, develop interventions,and use data to monitor the effects), and (8) care management(a set of specifically defined services for managingpatients with chronic illness involving multiplepractitioners and care between office visits).

The 8 practice systems and their components hadbeen previously identified by an expert advisorypanel convened by the National Committee for QualityAssurance to create the Practice Systems AssessmentSurvey (http://www.ncqa.org/Programs/RADD/researchreports.htm). This advisory panel was formedto identify the practice systems and components importantfor implementing the chronic care model framework.For each of the 8 practice systems selected by theNational Committee for Quality Assurance panel, theon-site auditors in the present study reviewed evidencethat the practice system and its components were presentand usable. At the end of the on-site audit, the auditorscompleted an assessment of how well andconsistently each practice system that was present wasbeing used. An investigator or data collection supervisor(SCS) accompanied the auditors to most of the sitevisits to monitor them, and several debriefing sessionswere conducted with the entire investigator group toclarify and verify the information and its collectionprocess. After the on-site audit information was enteredinto an electronic database and the data werecleaned, item frequencies were organized by medicalgroup for the analysis herein. This study was approvedand monitored by the HealthPartners InstitutionalReview Board.

Descriptive information about the participatingmedical groups is given in 

In this and subsequenttables, data from the 7 medical groups without anEMR are contrasted with data from the 4 medicalgroups with an EMR. One of the 4 medical groups hadan EMR that comprised all of the functions tested in theon-site audit; the other 3 supplemented their EMR withseparate ordering or data systems. Six of the 7 medicalgroups with paper medical records managed someinformation with separate electronic systems.Examples of such systems are registries created fromelectronic billing systems, electronic reporting systemsin in-house laboratories, and electronic appointmentsystems with the ability to include specific reminders.

gives information about the presence of practicesystems as demonstrated in the on-site audit, withthe auditors' subjective assessment of how consistentlythe practice systems were being used. Although almost allof the 11 medical groups had at least some component ofeach practice system present, the auditors found thatsome practice systems were not consistently used, withthe medical groups with an EMR beingmore likely to consistently use existingpractice systems. Overall, the practice systemthat was least likely to be present wasregistry of patients with chronic conditions,and this was most lacking in themedical groups without an EMR. Based onthe auditors' assessments, some of the 7medical groups without an EMR did notuse systematic monitoring or clinicianreminders consistently even though theyhad a non-EMR method to do so.

Three items related to the practicesystem of continuity of care wereassessed. Because all 11 medical groupshad evidence of each of those components,findings for this practice systemare not given in a table. Each medicalgroup demonstrated that it had identifieda personal clinician for each patient, hada process to assure that most patient visitswere with that clinician, and had formalprimary care teams to facilitateaccess and follow-up, with expandedroles for nurses or other team members.

summarize the extent towhich the other 6 practice systems were present in the11 medical groups. These tables give the number ofmedical groups having the components of the practicesystem summarized in each table. Most components ofeach practice system were present in all of the 11 medicalgroups. Although the 4 medical groups with anEMR were somewhat more likely to have all of thepractice system components, the 7 medical groupswithout an EMR had other ways to perform the functionsof the practice systems. In particular, there is littledifference among the 11 medical groups in thepresence of most components of the performancetracking and feedback practice system and no differenceamong the 11 medical groups in the presence ofall components of the clinical quality evaluation andimprovement practice system. To clarify this, we italicizedeach table component in which the medicalgroups with an EMR were more than twice as likely asthe medical groups without an EMR to have that component.Most italicized components are found in Table3, which addresses the clinical information practicesystem.

Our findings demonstrate that the 11 medical groupsstudied had a high number of the practice systems thatare believed to be important for providing effective carefor patients with chronic conditions. They also had mostof the detailed components of those practice systems.Although the medical groups with an EMR had morepractice systems and components present, the medicalgroups relying on paper medical records had other waysto implement most of the components of the practicesystems. Only 13 of 60 total components in the practicesystems were more than twice as likely to be present inthe medical groups with an EMR, suggesting that anEMR is necessary for or enhances these capabilities. Asexpected, those components were largely related toinformation technology, such as reminders, registries,and data about individual clinicians.

Few studies have documented any aspect of the presenceof practice systems among medical groups. Afterpostulating the chronic care model,

studied72 leading chronic disease programs. They foundthat only 1 program included all 6 elements of thechronic care model framework, and only 5 other programsincluded 5 of 6 elements. Solberg et al

assessedthe importance of practice systems in delivering preventiveservices and reported in the mid 1990s that practicesystems were infrequent findings among 44 medicalpractices in Minnesota.

reported that, among 1040physician organizations (with &#8805; 20 physicians) respondingto a survey about the extent to which physicianorganizations use CMPs, 50% had 4 or fewer of apossible 16 CMPs for chronic disease care (similar to thepractice systems described in this study), and only 22%had more than 8 CMPs. A subsequent investigation byLi et al

studied the presence of 4 CMPs (registry,guidelines, case management, and physician feedback)among 987 medical groups that provided care forpatients with diabetes mellitus. They found that 48%had 0 to 1, 20% had 2, and 32% had 3 to 4 of these CMPs.The characteristics associated with the presence ofthese CMPs were external incentives, a computerizedinformation system, and ownership by a hospital or ahealth maintenance organization. Results of otherinterview studies

among leading healthcare deliverysystems suggest that the main barriers to successfulimplementation of CMPs are inadequate resourcesor information systems, physician busyness or resistance,and lack of an effective means for reimbursement.The main facilitators were strong leadership, an organizationalculture valuing quality of care, the presence ofelectronic information systems, and supportive healthplans.

There seems to be widespread perception on the partof policy makers that an EMR is the principal or eventhe only change required for closing the quality of caregaps identified by the Institute of Medicine.

There islittle clear evidence on this important issue, but clearlyan EMR is not the sine qua non of efficacy. The VeteransHealth Administration has made remarkable strides inimproving its quality of care, with Asch et al

documenting10% to 20% better performance by the VeteransHealth Administration in chronic disease care and preventivecare (but not acute care) compared with caredelivered by a national sample of providers. Although itsintegrated electronic information system was creditedfor some of this improvement, the Veterans Health Administrationhas implemented other quality improvementand comparative performance reporting activitiesas well. In a randomized trial of electronic informationsystem implementation of cardiac care guidelines targetingprimary care physicians and pharmacists inIndiana, Tierney et al

found no effect of cardiac careguidelines generated by an EMR on physicians' adherenceto evidence-based guidelines and suggested thatmethods of affecting clinician behavior other than anEMR were needed.

Before the value of an EMR in improving healthcaredelivery can be thoroughly tested, the types of practicesystems that are primarily or exclusively driven by EMRadoption need to be characterized in detail. The presentstudy takes a first step toward that goal. Because the 11medical groups described herein appeared to be able toimplement almost as many practice system componentswithout an EMR as with an EMR, the stage is set forcomparison trials of the effectiveness of differentapproaches to information systems and of various typesof practice systems.

Although the present study is valuable in documentingin detail for the first time, to our knowledge, thepresence of a variety of practice systems in a sample ofmedical groups, it has significant limitations for generalizationof the results. The number of medical groupsstudied was small, and although we recruited a diversesample, the medical groups were all large, with moremidlevel practitioners and registered nurses (Table 1)than most primary care practices in the United States,where only 18% of physicians work in groups of 10 ormore.

Also, the medical groups described herein wereall members of a sophisticated quality improvementcollaborative (the Institute for Clinical SystemsImprovement) that focuses on encouraging the developmentand effective use of practice systems, althoughhalf the medical groups were new to that membership.Whether increased use of practice systems is drivingquality improvement or, conversely, participation inquality improvement is driving the adoption of practicesystems (or some other factor is driving both) is a fertilearea for further inquiry. Despite these limitations,this study demonstrates that private medical groups,including those without an EMR, can organize theirpractices for systematic care of persons with chronicconditions.

This study would not have been possible without the close cooperationof the staff and leadership of the Institute for Clinical Systems Improvement.We are especially indebted to the leaders, physicians, and staff of the followingmedical groups that contributed to the implementation of this study:Affiliated Community Medical Center, CentraCare Clinic, CommunityUniversity Health Care Center, Fairview Lakes Regional Health Care,Fairview Red Wing Regional Health Care, Family Practice Medical Center,Grand Itasca Clinic and Hospital, Hutchinson Medical Center, MeritCare,North Clinic, Northwest Family Physicians, Olmsted Medical Center, ParkNicollet Health Services, and Stillwater Medical Group. We also appreciatethe careful work of the on-site audit team, including Laurie Van Arman, LPN,Betty Lindstrom, RN, and Colleen King.

From HealthPartners Research Foundation (LIS, SEA, MJT) and Institute for ClinicalSystems Improvement (ALRM), Minneapolis, Minn; and National Committee for QualityAssurance, Washington, DC (SHS, SCS, GP).

This study was supported by grant 048908 from The Robert Wood Johnson Foundation,Princeton, NJ.

Address correspondence to: Leif I. Solberg, MD, HealthPartners Research Foundation,PO Box 1524, MS 21111R, Minneapolis MN 55440-1524. E-mail: leif.i.solberg@healthpartners.com.

To Err Is Human: Building a Safer Health System.

Washington, DC: National Academy Press; 1999.

Crossing the Quality Chasm: A New Health System for

Washington, DC: National Academy Press; 2001.

The quality of health care delivered toadults in the United States. 

Priority Areas for National Action: Transforming Health Care Quality.

4. Institute of Medicine Committee on Identifying Priority Areas for Quality

Washington, DC: National Academy Press; 2003.

5. Wagner EH, Austin BT, Davis C, Hindmarsh M, Schaefer J, Bonomi A. 

Improvingchronic illness care: translating evidence into action. 

Chronic disease management: what will it take to improve care forchronic illness? 

7. Casalino L, Gillies RR, Shortell SM, et al. 

External incentives, information technology,and organized processes to improve health care quality for patients withchronic diseases. 

8. Farley DO, Haims MC, Keyser DJ, Olmsted SS, Curry SV, Sorbero M. 

Regionalhealth quality improvement coalitions: lessons across the life cycle. Santa Monica,Calif: RAND Health; 2003:70.

9. Wagner EH, Davis C, Schaefer J, Von Korff M, Austin BT. 

A survey of leadingchronic disease management programs: are they consistent with the literature?

Practice Systems for Chronic Care: Frequency and Dependence on an Electronic Medical Record

To determine if the quality of medical imaging reportsdiffers significantly between radiologists and nonradiologists.

A retrospective nonblinded review of randomlyselected chest and long bone x-ray reports by orthopedists and primarycare physicians compared with randomly selected imagingreports generated by radiologists.

Guideline for Communication: Diagnostic Radiology. 

We randomly selected 1 report from each of 50 highself-referring physicians privileged by 2 metropolitan New Yorkarea health plans for both bone and joint studies and chest x-raysfor a total of 200 reports (50 bone and joint x-rays from each planand 50 chest x-rays from each plan). We compared them with 50randomly selected radiologist-generated reports. The reports wereevaluated for quality based on the American College of Radiology's

-test between proportions at the95% confidence interval.

Radiologists consistently provided higher-quality medicalimaging reports than nonradiologists.

To improve imaging service quality, all providersshould be held to the same standards for reporting and communicationof results.

Guideline for Communication: Diagnostic Radiology

Twenty-first century radiology requires rapid andaccurate communication of imaging results. TheAmerican College of Radiology's 

states that every imaging study should have an "officialinterpretation."

The College further defines "officialinterpretation" as the written report that becomes partof a patient's permanent medical record. The AmericanMedical Association (AMA) supports this position, statingin the 

"A written report, signed by the interpretingphysician, should be an integral part of a radiologicprocedure or interpretation."

According to KarenZupko and Associates Inc

(a practice management consultingcompany), the Center for Medicare and MedicaidServices (CMS) applies section 15023 of the Medicare

to outpatient radiology services. Thisregulation requires a written report for payment of professionalservices. It also distinguishes between an"interpretation and report" and a "review," notation, orcomment about the study.

Finally, the NationalCommittee for Quality Assurance (NCQA) has generalguidelines for keeping good medical records.

In 2002, the Consensus Workgroup on Health InformationCapture and Report Generation publishedtheir findings and recommendations regarding improvementin healthcare documentation and communication.They identified inadequate, illegible documentation andlimited access to the medical record as contributing tomedical errors.

The group also found that despite thefact that "accurate, accessible, and shareable healthinformation is a well-accepted prerequisite of goodhealth care," inadequacies in documentation practices,accessibility, and shareability are accepted in the UnitedStates. They added that these substandard practicescompromised "patient safety, public safety, continuity ofpatient care, healthcare economics, clinical research,and outcomes analysis." They also noted that lack of uniformityof medical records and the use of free text madefinding important information difficult and time consuming.The inconsistent terminology used made it difficultto understand the information once located.

During our work in radiology benefits management,we found that the availability of plain film reports wasinconsistent when performed at the treating physician'soffice (self-referral) but consistent when performed at aradiology site. We decided to determine the extent ofthis problem.

We evaluated the imaging reports of internists andorthopedists privileged by 2 metropolitan New York areahealth plans to perform chest and long bone and joint x-raysin their offices. We searched our database and identifiedthe 50 physicians who self-referred the most chestand bone x-rays for each plan. We then randomly selected1 case for each physician and requested a copy of theexamination report for both bone and joint studies andchest x-rays for a total of 200 reports (50 bone and jointx-rays from each plan and 50 chest x-rays from eachplan). After reviewing the responses we noted that radiologistsactually generated a few reports, even though thestudy had been performed in the treating physician'soffice. We eliminated these reports from our study.

Additionally, we randomly selected 50 radiologist-generatedreports from those on file in our QualityManagement department. These included 22 chestx-rays, 21 bone and joint films, and 7 abdominalfilms, upper gastrointestinal series, or barium enemaexaminations.

We determined the required elements for a radiologyreport (

making a fewappropriate modifications for this study. The ACRguideline encourages inclusion of the following 4 demographicelements:

Of these factors, we included only the latter 2 elements.Other elements included in the ACR guideline,but not evaluated in our study, are:

Contrast material administered (including concentrationand volume)

Conclusion or impression if the report is short

A board-certified radiologist with more than 25 years'experience interpreting plain films then compared thesubmitted materials to required elements. The reviewerwas not blinded to the specialty of the interpretingphysicians. We analyzed the data using a 2-sample 

testbetween proportions at the 95% confidence level(

An imaging report accompanied all 50 cases of theradiologists. Of 200 requests for reports from the nonradiologistgroup we received 163 responses, for aresponse rate of 81.5%. Of the 163 responses, a radiologistactually read 16, either at a radiology facility or in anonradiologist's office. We excluded these 16 cases fromthe study, resulting in 147 cases for review. Of these, 56(38.1%) had no report as described by the AmericanCollege of Radiology, although several did include anotation or comment about the study in office notes orin a letter to the referring physician (these were includedin the analysis). Another 17 (11.6%) sent a reportthat was dated after our request for the report or wasaddressed in a manner indicating that the report mighthave been generated in response to our request. These17 cases were retained in the data analysis because wecould not accurately determine whether the reportexisted prior to our request. Therefore, up to 49.7% ofthe cases may not have had an appropriate report.Seventy four (50.3%) sent a real report (

).Nonradiologists failed to use a unique identifier 70% ofthe time. Handwritten notes or reports were often illegible,resulting in useless documentation. No handwrittennotes were found in the radiologists reports.

Radiologists' reports were consistently better thanthose of nonradiologists. A statistically significant difference was noted between the radiologists' reportsand those of the nonradiologists for the followingcategories of demographic data: use of uniqueidentifier number, patient's date of birth, patientsex, and interpreting physician's name and signature.For clinical information the data differedsignificantly between the radiologists' reports andthose of the nonradiologists for the description ofthe examination, views taken, description of findings,and presence of an impression or conclusion(

Radiologists did not include the indications forthe examination as often as did nonradiologists(46% vs 58%). This information may have beenomitted because the requesting physician mayhave provided the radiologist with inadequateinformation. When we stratified the nonradiologists' results according to internists or familypractitioners and orthopedists, only 14 of 68(21%) chest x-ray reports from internists or familypractitioners included a reason for the examinationcompared with 67 of 79 (86%) bone andjoint x-ray reports from orthopedists, mostlybecause the orthopedist reports were frequentlypart of a letter to the referring physician.

We designed this study to evaluate the adequacy ofmedical record documentation, not the films' technicalquality or interpretation accuracy. This distinction isimportant because self-referring physicians performsignificantly more x-rays than radiologist-referringphysicians and the absence of a good medical recordcan negatively impact patient care. In 1998, Spettelland colleagues

reported that in nonhospital settingsnonradiologists performed approximately 67% of chestand spine films and between 78% and 86% of bone andjoint films.

Healthcare documentation has come under scrutinyin the last few years. In 2002, a report from theConsensus Workgroup on Health Information Captureand Report Generation indicated that the accuracyand accessibility to healthcare information was compromisedin the United States.

The report stated thatpatient safety, public safety, continuity of patient care,healthcare economics, and clinical research and outcomesanalyses were all adversely affected by poordocumentation and record keeping. This group stronglyadvocates adoption of electronic medical recordsthat are uniformly structured and designed to besearchable.

Inadequate Imaging Reports Compromise Care

Documentation of an imaging examination is consideredto be an important part of a patient's medicalrecord.

The American College of Radiology (ACR) hasdescribed guidelines for structuring good imagingreports, while the NCQA has general guidelines formedical records maintenance. Both organizations agreethat each page of the medical record should includethe patient's name, date of service, interpreting physician'sname, and the physician's signature (thiscould be a unique electronic identifier). The ConsensusWorkgroup on Healthcare Information Capture andReport Generation also recommends the use of aunique patient identifier. A unique identifier allowshealthcare providers to differentiate patients with thesame name and date of birth.

In our study we found myriad documentation errorsamong records from nonradiologists. The absence of areport or a brief comment about an x-ray in a chart noteor in a letter to a referring physician (which happenedin up to 49.7% of the cases in the current study in thenonradiologist group) can lead to inadequate communicationbetween healthcare providers. This communicationbreakdown may result in repeat examinations,which increase costs, expose patients to unnecessaryradiation, and may potentially delaypatient care. Failure to describe the findings orto include an appropriate impression or differentialdiagnosis limits the examination's value toother providers. The use of excessively shortnotations in a record or letter often makes locatingor interpreting the results difficult whenanother provider, healthcare facility, or healthplan requests information about a patient.

reportedsimilar observations in their review of nonradiologists' imaging reports. In their study 62% ofthe offices evaluated did not issue a formal radiologyreport. At many of these sites "a note wasmade in the chart cryptically stating that a radiographwas either positive or negative."Another study of chest x-rays performed byPennsylvania's Blue Shield found 21 of 98reports to be incomplete.

The current study also demonstrated thatalthough radiologists' reports more consistentlycomplied with national recommendations, thereis room for improvement, especially in documentingthe reason for the examination.

In addition to these issues concerning goodpatient care, legal issues exist regarding medicalrecord documentation and the matter of reimbursement.As stated previously, the CMS indicatesthat all imaging studiesshould include a writtenreport, and the AMA furthersuggests that the report besigned by the interpretingphysician. Radiology billingis unique in that the reimbursementis divided intoprofessional (identified bymodifier -26) and technical(identified by modifier -tc)components or is billed asglobal (no modifiers andcomprises the professionalplus technical components).Examinations performed inan office setting are usuallybilled globally. The professionalrelative-value units, orwork that determines theprofessional reimbursement,includes not only imageinterpretation but also reportpreparation. Failure to provide a report is not only poor patient care and insufficientrecord keeping in the event of a lawsuit, but canalso lead to claims of fraudulent billing for services,which are incompletely provided. Because reimbursementis the same for chest or bone x-rays regardless ofthe specialty of the interpreting physician, reportingstandards should also be the same.

One limitation of this study is that we obtained theradiologists' reports from our Quality Management division,rather than randomly requesting them. However,in the course of our normal procedures, radiologistshave consistently provided us with a written report.

In the interest of better patient care and improvedcommunication between healthcare providers, imagingservice providers should be required to produce legiblex-ray reports that are easily identified. This requirementshould be part of a health plan's privileging program.Additionally, we recommend that health plans definethe report's essential elements, including patient identificationrequirements. Furthermore, we suggest that theAmerican College of Radiology's guideline for communicationbe adopted and that all imaging service providersbe held to this national standard.

From CareCore National, LLC, Wappingers Falls, NY.

CLINICAL

Radiology by Nonradiologists: Is Report Documentation Adequate?

To determine whether providing clinicians with the results of a patient-reported mental health assessment would have a significant impact on patients' mental health outcomes.

The study used a portion of the SCL-90 (Symptom Checklist-90) to track the perceived mental health of 1374 patients in a managed behavioral healthcare system over 6 weeks.

Participants were randomized into a feedback group whose clinicians received clinical feedback reports at intake and at 6 weeks, and a control group whose clinicians received no report.

Patients in the feedback group achieved statistically significant improvement in clinical status relative to controls.

Overall, the study suggests that patient-reported mental health assessments have the potential both to become acceptable to clinicians and to improve the effectiveness of clinical care.

Since Howard and colleagues introduced the paradigm of patient-focused research,

investigations of patient-reported assessments (like those solicited by the 2003 National Institutes of Health Request for Applications, "Dynamic Assessment of Patient-Reported Chronic Disease Outcomes") have focused primarily on their usefulness as a research tool, tracking large groups to assess the effectiveness of a particular treatment or to streamline processes for cost control. However, a growing body of research suggests that a reliable patient-reported assessment tracking system may offer direct clinical benefits as well.

This may be particularly true in psychotherapy, where studies suggest that a lack of reliable feedback mechanisms may contribute to therapists' inability to make accurate assessments of patient progress.

 Similarly, in a meta-analysis of available studies, Sapyta et al

found that when patients provide feedback to psychologists regarding the patients' general health and progress in treatment, significant benefits may result, especially for patients who are not doing well. Brown and Jones

suggest that early identification of patients who are not progressing in treatment can help clinicians keep patients engaged in the process long enough to improve.

have conducted a series of studies to investigate whether providing periodic feedback to therapists about patient progress leads to improvement in treatment outcomes. Their results showed that feedback to therapists resulted in fewer treatment sessions with no worsening of outcome for patients making progress, while patients who were deteriorating or not progressing received more sessions and had better outcomes than predicted. Patient-reported feedback, then, allows therapists to focus resources where needed, ensuring that those patients most at risk of failing to respond to treatment receive the attention and encouragement they need to continue in treatment.

Even a minimal feedback program can have real benefits for patients and therapists, increasing the effectiveness of therapy. In a 2002 study, Percevic showed that 1-time feedback provided to therapists early in treatment can yield significant effects for all patients, presumably by allowing therapists to more closely tailor treatment programs to patient needs.

December 2005

Diabetes Healthcare Quality Report Cards: How Accurate Are the Grades?

Practice Systems for Chronic Care: Frequency and Dependence on an Electronic Medical Record

Radiology by Nonradiologists: Is Report Documentation Adequate?

The Acceptability and Effectiveness of Patient-reported Assessments and Feedback in a Managed Behavioral Healthcare Setting

Geographic Variation of Asthma Quality Measures Within and Between Health Plans

Efforts to Improve Subsequent Treatment of Cardiovascular Risk Factors in Older Patients With Diabetes Hospitalized for a Cardiac Event