Background: The most widely used performance measure forasthma, the Health Plan Employer Data and Information Set(HEDIS), has been criticized because the delay between classification(year 1) and assessment of medication dispensing (year 2) mayproduce a "misalignment"and weaken the validity of the measure.
Objective: To examine whether a previously observed associationbetween the HEDIS performance measure and asthma-relatedemergency department visits is robust when the period betweenthe classification and outcome assessment is evaluated during a2-year period as defined.
Methods: Children (N = 2766) aged 3 to 15 years enrolled in 1of 3 managed care organizations with at least 1 asthma diagnosislisted for a hospitalization, an emergency department visit, or anambulatory encounter and at least 2 consecutive years of data foranalysis from July 1996 through June 1999 were identified.
Results: Children did not consistently meet the HEDIS criteriafor persistent asthma, and 24% to 28% of children did not requalifyin year 2 of observation. Multivariate regression models showedthat a protective relationship between controller medication dispensingand asthma-related emergency department visits was nolonger seen among children meeting the HEDIS criteria for persistentasthma when the total period of observation is extended to 2years (odds ratio, 0.7; 95% confidence interval, 0.4-1.2).
Conclusions: Our results suggest that the variable nature ofasthma may affect how the HEDIS performance measure should beused for assessing quality of care. The period between identificationof the target population and performance assessment shouldbe closely related in time.
(Am J Manag Care. 2005;11:325-
Guidelines for the Diagnosis and Treatment
The developed by the National AsthmaEducation and Prevention Program1 have beenwidely disseminated to help achieve more uniform managementand improve disease outcomes. However, variationsin asthma management practices persist, andasthma continues to impose a burden on society,2 witha high proportion of asthma-related morbidity concentratedin identifiable subpopulations.3 Targeting disease-specific intervention to the individuals at highestrisk allows efficient use of healthcare resources and hasbeen shown to be effective.4 Managed care organizations(MCOs) have attempted to use performance measuresas a way to reduce variation and to monitor and identifysuboptimal management practices through the use ofperformance measures.
The most widely used performance measure for asthmais the "use of appropriate medications for peoplewith asthma,"which is one of the many measuresincluded in the Health Plan Employer Data andInformation Set (HEDIS), developed by the NationalCommittee for Quality Assurance to evaluate the performanceof health plans. This measure is computed asthe proportion of people who filled at least 1 prescriptionfor a controller medication among those identifiedas likely to have persistent asthma. Persons are classifiedas having persistent asthma based on data fromyear 1, while the assessment of controller dispensing isassessed based on data from year 2. However, performancemeasures, in general, have been criticized as havingunintended negative effects on care by "overlookingimportant dimensions of quality not captured by theseperformance measures."5,p575 In addition, specificaspects of the HEDIS asthma measure have been criticized.5 Because the HEDIS asthma measure relies solelyon automated data (ie, no reports of symptomfrequency or pulmonary function data), the denominator(the subpopulation of individuals with persistentasthma) can only be estimated. Second, the HEDISasthma measure requires data from 2 consecutive yearsfor evaluation; subjects are classified as having persistentasthma using data from year 1, while the HEDIS performance measure, the proportion with controller medicationdispensing among the individuals identified ashaving persistent asthma, is evaluated using data fromyear 2. The potential misclassification of subjects withpersistent asthma and the delay between classification(year 1) and performance assessment (year 2), producinga "misalignment,"may weaken the validity of themeasure.5 A third criticism is that the HEDIS measuredoes not emphasize the regular use of controller medications.For persons who truly have persistent asthma,the single dispensing of a controller medication during a12-month period is inadequate.
The Pediatric Asthma Care Patient OutcomesResearch Team, in a clinical trial assessing the effectivenessand cost-effectiveness of strategies to implementasthma guidelines,6 previously evaluated theability of the HEDIS performance measure to identifyindividuals at increased risk of future adverse asthma-relatedevents. The prior analysis demonstrated that,among children defined by the HEDIS criteria as havingpersistent asthma, identification of children whohave been dispensed a controller medication can helpstratify children based on their risk of future adverseevents.7 In this analysis, we sought to extend theseresults to examine the criticisms of the HEDIS measure.Does the misalignment between the period of identifyingchildren with persistent asthma and the periodof performance assessment in the HEDIS performancemeasure affect its ability to identify children with persistentasthma and sustained high utilization?
Provider profiling is now widely practiced in manymanaged healthcare systems.8 We based our analysiswithin 3 large geographically diverse managed caresystems.
Group Health Cooperative of Puget Sound.
GroupHealth Cooperative of Puget Sound is a centralizedgroup-model health maintenance organization. At thetime of the study, it was the sixth largest nonprofithealth maintenance organization in the nation, servingan estimated 383 000 members in the Puget Sound area.
Harvard Pilgrim Health Care.
Harvard PilgrimHealth Care (HPHC) is a mixed-model managed careorganization (MCO) concentrated in eastern Massachusetts.At the time of the study, HPHC had a staff-modelcomponent and a network practice component.The Pediatric Asthma Care Patient Outcomes ResearchTeam trial was conducted in HPHC network practices.Each of these network practices was affiliated with severalhealth insurers or MCOs in addition to HPHC.Therefore, a second MCO, Blue Cross Blue Shield ofMassachusetts, was enlisted to increase the number ofchildren per practice under analysis.
Prudential Health Plans, Chicago.
Rush-Prudential Health Plans enrolled an estimated 350 000members at the time of the study. Rush-PrudentialHealth Plans had a staff-model component and an independentpractice association model component.
Automated claims and pharmacy data were collectedas part of the Pediatric Asthma Care Patient OutcomesResearch Team trial from each of the 3 study sites. All3 MCOs maintain computerized information systemsthat capture basic demographic data and claims files forall hospitalizations and emergency department (ED)visits. Automated pharmacy records contain detailedinformation on prescriptions dispensed at outpatientpharmacies. Approval for this study was obtained frominstitutional review boards at each of the participatinginstitutions.
Classification of Diseases, Ninth Revision, Clinical
The source population for this study consisted ofchildren aged 3 to 15 years enrolled in 1 of the 3 studyMCOs with at least 1 asthma diagnosis (codes 493.00-493.99) listed for a hospitalization,an ED visit, or an ambulatory encounterduring a 3-year period from July 1996 through June1999. In addition, study subjects had to have continuousenrollment for the 12-month study period, prepaidprescription drug coverage, and at least 1dispensing of an asthma medication. Approximately90% of all members have prepaid drug coverage thatprovides up to a month's supply of medicine for anominal payment. For this analysis, the study populationwas further limited to children with 2 consecutiveyears of data available.
The HEDIS DenominatorThe HEDIS denominator is the subpopulation ofchildren meeting the HEDIS criteria for persistentasthma. Children are included if they meet 1 of 4 criteriabased on medication dispensing or healthcareutilization; each criteria is assessed during a 12-monthperiod:
In addition to the 4 criteria, the absolute level ofβ-agonist dispensing has been independently associatedwith the risk of asthma-related hospitalizations and EDvisits.9 Therefore, we refined the HEDIS denominatorby stratifying the level of β-agonist dispensing. TheHEDIS high reliever subpopulation included childrenwho met the HEDIS criteria for persistent asthma andhad at least 4 β-agonist dispensings per person-year,while the HEDIS low reliever subpopulation includedchildren who met the HEDIS criteria for persistentasthma and had fewer than 4 β-agonist dispensings perperson-year.
Classification Using Pharmacy Data
Automated medical records were inspected for typeand quantity of asthma pharmacotherapy providedduring each 1-year period. Asthma controller therapywas defined to include inhaled corticosteroids, inhaledcromolyn sodium and nedocromil sodium (referred toas "cromolyn"), and oral antileukotriene and theophyllinepreparations. Reliever agents included inhaledor pediatric oral reliever preparations such as short-actingβ-agonists and anticholinergics but excluded long-actingβ-agonists such as salmeterol xinafoate.10Pharmacy data included initial dispensing and refills ofall prescription medications. Salmeterol and oral corticosteroidswere not included in this analysis.
For each type of drug, the frequency of dispensing wascalculated for each study subject as the sum of the numberof canisters or containers of drug dispensed duringeach 1-year period. Canister equivalents for nebulizedreliever agents and inhaled and nebulized anti-inflammatorymedications were created on the basis of expectedduration of supply with typical dosages. This method correctsfor differences in days of medication supplied byvarious anti-inflammatory preparations at standard dosesand by reliever agents formulated as metered-doseinhalers versus a nebulized solution.11 We defined 1 canisterequivalent of a reliever agent as 1 canister ofalbuterol sulfate and considered 2.5 mg of nebulizedalbuterol as equivalent to 4 puffs of the metered-doseinhaler.11 One dispensing of an oral formulation wastreated as 1 canister equivalent regardless of the quantitydispensed. Controller medications were not weightedfor potency, as fluticasone propionate accounted for only5% of controller dispensing, budesonide was not beingused during this period, and recommended doses of otherinhaled corticosteroids are similar in potency.
The HEDIS Performance Measure
The HEDIS performance measure (version 3.0) identifiesthe proportion of individuals who have been dispenseda controller medication (yes or no) amongsubjects meeting the HEDIS criteria for persistent asthma.The performance measure specifies that 2 years ofdata are necessary for review; the information in year 1is used to target individuals, and the information in year2 is then used to assess prescribing practices. Threeyears of data were available for this project. Therefore,2 separate cohorts of children with 2 consecutive yearsof data (cohort 1 with data from years 1-2 and cohort 2with data from years 2-3) were included in the analysis,and children with all 3 years of data contributed observationsto both cohorts.
Emergency Department Visit Outcome Variables
Classification of Diseases, Ninth Revision,
Each MCO maintains computerized files of all officevisits, hospitalizations, and ED visits of its members;coded information includes visit dates and diagnosis codes. All patient datafiles were linked by scrambled and untraceable patientidentification numbers.
We included all events and pharmacy dispensingsthat were billed to the MCOs. This would not includeitems completely paid for out of pocket or by anotherinsurer. Previous work from within one of the healthplans indicates that the number of such occurrences issmall.12 Emergency department visits were included inthe analysis only if they were related to asthma.
The distributions were tabulated according to organizationaland demographic characteristics. Differencesin the proportion of children in each stratum wereassessed for significance by χ2 tests and Mantel-Haenszel methods for analysis of 2 × κtables. TheHEDIS performance measure was tested in a predictivemodel examining the relationship of controller medicationdispensing and ED events. Multiple logistic regressionwas then used to assess independent effects of theHEDIS performance measure in models for ED visits.
Because published literature has demonstrated thatage, sex, and MCO are associated with varying rates ofmedication use,9 multivariable models included thesevariables. In examining the association of age with EDvisits, models incorporating a quadratic term in additionto the linear term had improved fit. General estimatingequations were used to account for multipleobservations among some subjects and for clusteredobservations at the level of the clinical practice.
We identified 2 overlapping cohorts of children withpersistent asthma who had at least 2 consecutive yearsof data: cohort 1 (n = 1532), children identified in year1 with information available in year 2, and cohort 2(n = 1234), children identified in year 2 with informationavailable in year 3. A subpopulation of children(n = 615) had data available in all 3 years and contributedobservations to both cohorts. Demographiccharacteristics of the 2 cohorts were similar withregard to age, sex, and proportion of children coveredby Medicaid (Table 1).
Overall, about half of the children in both cohortsmet the HEDIS criteria for persistent asthma duringyear 1. A slightly lower proportion of children (51.2% vs54.9%) met the HEDIS criteria for persistent asthma inthe baseline period of cohort 1 (year 1) compared withthe baseline period of cohort 2 (year 2). However, whenwe examined the breakdown of the 4 inclusion criteriathat make up the HEDIS definition for persistent asthma,the distribution was found to be similar for all 3years of data. Most children were identified as havingpersistent asthma by medication dispensing criteriaalone (≥4 asthma medication dispensing events duringa 12-month period). Lower proportions of children metthe HEDIS criteria for persistent asthma for 2 consecutiveyears, 36.7% and 41.9% (cohort 1 and cohort 2,respectively) (Table 2).
We then examined the consistency of the HEDIS criteriato identify subjects with persistent asthma. Amongchildren who met the HEDIS criteria for persistent asthmain year 1 for each cohort, one quarter did notrequalify in year 2 of available data. Within cohort 1, ofthe 785 children who met the criteria in year 1, therewere 223 (28.4%) who did not requalify in year 2; similarly,within cohort 2, of the 677 children who met thecriteria in year 1, there were 160 (23.6%) who did notrequalify in year 2 (Table 2).
We then examined whether the association betweenthe HEDIS performance measure and asthmamorbidity was robust when the period of observationis extended for 2 years. This analysis included all childrenwith at least 2 consecutive years of automateddata (N = 2766).
Similar to prior analysis,7 the dispensing of a controllermedication was associated with the risk of asubsequent asthma-related ED visit, and the baselinereliever agent dispensing modified the strength of therelationship. However, in contrast to prior observations,now the association was significant only in theHEDIS low reliever population (Table 3). A significantprotective relationship between controller medicationdispensing and asthma-related ED visits was no longerseen among the entire population meeting the HEDIScriteria for persistent asthma (odds ratio [OR], 0.7;95% confidence interval [CI], 0.4-1.2; and OR, 0.3;95% CI, 0.2-0.4; for the present and prior analyses,respectively).
We then examined the relationship between theHEDIS performance measure and asthma-related EDvisits when the measure was modified to include onlyindividuals with consistent controller medication dispensing(≥4 controller medications per 12-month period).Again, the association between controllermedication dispensing and asthma-related ED visits wassignificant only in the HEDIS lowreliever subpopulation (OR, 0.9; 95%CI, 0.6-1.3; OR, 0.5; 95% CI, 0.3-1.0;and OR, 1.2; 95% CI, 0.8-2.0; for thetotal, low reliever, and high relieverpopulations, respectively).
We observed variability in thesubpopulation identified by theHEDIS criteria as having persistentasthma. Of children initially targetedby the HEDIS criteria as having persistentasthma, approximately onequarter did not requalify in the subsequentyear. This suggests that theHEDIS criteria for persistent asthmado not identify a consistent subpopulationof children over time and supportsthe criticism that the HEDIScriteria for persistent asthmamore accurately target childrenwith problematic asthma control.
Measures of disease severity, bydefinition, should characterize asubject's intrinsic intensity of disease.However, after initial presentation,accurate classification ofasthma severity, independent oftherapy, is complex. Many of thecharacteristics of disease that weuse to describe severity maychange or be absent after appropriateintervention. Therefore, theconcept of asthma control hasbeen introduced to more accuratelydescribe the status of diseasein the presence ofintervention. Asthma control canchange rapidly in response to triggersor therapy. Asthma is a highly variable disease, particularlyamong children whose level of control varies significantlyover time and whose manifestations andappropriate treatments vary with age.13,14 The HEDIScriteria for persistent asthma are more appropriatelyused to identify persons with suboptimal asthma controlthan to define a subject's underlying severity ofdisease.
Multivariable analyses confirmed that the HEDISperformance measure, the dispensing of a controllermedication among subjects meeting the HEDIS criteriafor persistent asthma, is associated with the risk offuture asthma-related adverse events. In addition, theassociation between the HEDIS performance measureand the risk of an ED visit is dependent on the underlyingreliever agent dispensing rate. However, our resultssuggest that the strength of this relationship is weakenedas the lag between classification and outcomeassessment is increased. Modifying the HEDIS criteriafor persistent asthma to include only children with consistentcontroller medication dispensing (≥4 controllermedications per 12 months) did not significantly affectthe relationship.
Previous investigations in asthma have demonstrateda limited ability to predict future events based on measurablecharacteristics. Using regression models toexamine persistent high expenditures, Monheit15demonstrated that demographic factors, health status,comorbidity, and prior utilization are all related to thepersistence of high expenditures. However, only a smallproportion of the variation in the probability of persistenceis explained by these models (pseudo for themodel, 0.19). Two recent analyses of trends in asthma-relatedhealthcare charges also emphasize the variabilityin healthcare utilization among asthmatic patients.Stempel et al16 examined hospital admissions, ED visits,and medication use by grouping patients into quartilesbased on asthma-related charges and by following theirmovement between quartiles each year. Of the 52 135patients identified, 24 351 (46.7%) were in the top quartilefor a single year only, and 14 727 (28.2%) remainedin the top quartile during a 3-year period. Identificationand targeting of high users during a given 1-year periodwill miss a significant number of high users in subsequentyears. In analyzing health expenditures,Monheit15 observed that, among the top 5% of spendersin a given year, only 30% retain that position in the subsequentyear, with just under half (45%) remaining inthe top decile. There is a tendency for most high usersto move to lower percentiles, with few persons demonstratingpersistently high expenditures.17,18
Analyses using risk-adjustment models suggest thatthe amount of variance explained by such models islinked to the temporal proximity of the input data to theperiod being targeted. As the lag between the period ofidentification of a particular warning flag and the periodof prediction increases, the variance explained by suchmodels decreases.19 In addition, analyses predictingcosts based on automated data, such as pharmacy-based(RxRisk) or diagnosis code-based (DiagnosticCost Groups/Hierarchical Condition Categories system)measures, showed significant differences in concurrentvs prospective models.20 These authors detected a significantdecline in variability explained by a model predictingcosts () when the explanatory variables wereseparated in time by a 1-year period from the cost variablethat they were trying to predict.
We must note limitations to our analysis. Althoughthe analysis cohort was drawn from 3 geographically variedMCOs, serving a large overall base population, thegeneralizability of our results to the other populations,such as those without commercial insurance, is uncertain.Our analysis population had a low enrollment ofsubjects covered by Medicaid, who may have differentunderlying patterns of healthcare utilization. Finally, ouranalysis was limited to children, and how our resultswould extrapolate to adult populations is unknown.
The HEDIS performance measure, the dispensing ofa controller medication among subjects meeting theHEDIS criteria for persistent asthma, is associated withdecreased risk of future asthma-related adverse events.Therefore, targeting individuals who have suboptimalasthma control as defined by the HEDIS criteria isimportant. However, the HEDIS criteria for persistentasthma more accurately identify persons with suboptimalcontrol than persons with persistent disease severity,because disease control can vary more rapidly overtime. Our results suggest that the variable nature ofasthma may affect how the HEDIS performance measureshould be used for assessing quality of care. Themeasure may be improved if the period of labeling subjectswith persistent (uncontrolled) asthma overlapswith the period of assessing whether controller medicationhas been dispensed; subjects would meet the criteriafor persistent asthma in the first and second years ofassessment. Requiring children to meet the HEDIS criteriafor persistent asthma for 2 consecutive yearswould decrease the number of children included butwould more appropriately target those individuals withpersistent uncontrolled disease. The inherent trade-offbetween a decrease in the sensitivity of the measure toidentify children with persistent asthma and a potentialimprovement in its specificity will need to be examined.
We gratefully acknowledge Katherine Croom, BS, and Nancy Laranjo,BA, for their insight, technical help, and programming.
From the Channing Laboratory, Brigham and Women's Hospital, Harvard MedicalSchool (ALF, VJC, STW) and Department of Ambulatory Care and Prevention, HarvardMedical School and Harvard Pilgrim Health Care (JAF), Boston, Mass; Center for HealthStudies, Group Health Cooperative of Puget Sound, and Department of Pediatrics,University of Washington, Seattle (PL); Regenstrief Institute for Health Care, Indianapolis,Ind (TSI); and Hines VA Medical Center and the Center for Healthcare Studies and Divisionof General Medicine, Feinberg School of Medicine, Northwestern University, Chicago, Ill(KBW).
Funding for this study was provided by Pediatric Asthma Care Patient OutcomesResearch Team II grant HS008368-01 from the Agency for Healthcare Research andQuality, Rockville, Md, and the National Heart, Lung, and Blood Institute, Bethesda, Md.Dr Fuhlbrigge is supported by Mentored Clinical Scientist Development Award 1 K08HL003919-01 from the National Heart, Lung, and Blood Institute.
Address correspondence to: Anne L. Fuhlbrigge, MD, MS, Channing Laboratory,Brigham and Women's Hospital, 181 Longwood Avenue, Boston, MA 02115. E-mail:firstname.lastname@example.org.
Expert Panel Report:
Guidelines for the Diagnosis and Management of Asthma: Update on Selected
1. National Asthma Education and Prevention Program. Bethesda, Md: National Heart, Lung, and Blood Institute, NationalInstitutes of Health, Public Health Service, US Dept of Health and Human Services;2003. Available at: http://www.nhlbi.nih.gov/guidelines/asthma/. AccessedFebruary 26, 2005.
Am J Respir Crit Care Med.
2. Fuhlbrigge AL, Adams RJ, Guilbert TW, et al. The burden of asthma in theUnited States: level and distribution are dependent on interpretation of the nationalasthma education and prevention program guidelines. 2002;166:1044-1049.
Am J Respir Crit Care Med.
3. Lieu TA, Quesenberry CP, Sorel ME, Mendoza GR, Leong AB. Computer-basedmodels to identify high-risk children with asthma. 1998;157(pt 1):1173-1180.
J Allergy Clin Immunol.
4. Greineder DK, Loane KC, Parks P. A randomized controlled trial of a pediatricasthma outreach program. 1999;103(pt 1):436-440.
Am J Manag Care.
5. Glauber JH. Does the HEDIS asthma measure go far enough? 2001;7:575-579.
Health Serv Res.
6. Finkelstein JA, Lozano P, Streiff KA, et al; Patient Outcomes Research Team.Clinical effectiveness research in managed-care systems: lessons from the PediatricAsthma Care PORT. 2002;37:775-789.
7. Fuhlbrigge A, Carey VJ, Adams RJ, et al. Evaluation of asthma prescriptionmeasures of health system performance based on emergency department utilization.2004;42:465-471.
Ann Intern Med.
8. Kerr EA, Mittman BS, Hays RD, Siu AL, Leake B, Brook RH. Managed care andcapitation in California: how do physicians at financial risk control their own utilization?1995;123:500-504.
Arch Pediatr Adolesc
9. Adams RJ, Fuhlbrigge A, Finkelstein JA, et al. Use of inhaled anti-inflammatorymedication in children with asthma in managed care settings. 2001;155:501-507.
Br J Gen Pract.
10. Aveyard P. Assessing the performance of general practices caring for patientswith asthma. 1997;47:423-426.
Ann Allergy Asthma Immunol.
11. Glauber JH, Fuhlbrigge AL. Stratifying asthma populations by medication use:how you count counts. 2002;88:451-456.
J Infect Dis.
12. Hirschhorn LR, Trnka Y, Onderdonk A, Lee ML, Platt R. Epidemiology ofcommunity-acquired -associated diarrhea. 1994;169:127-133.
J Allergy Clin Immunol.
13. Calhoun WJ, Sutton LB, Emmett A, Dorinsky PM. Asthma variability inpatients previously treated with β2-agonists alone. 2003;112:1088-1094.
Eur Respir J.
14. Zhang J, Yu C, Holgate ST, Reiss TF. Variability and lack of predictive abilityof asthma end-points in clinical trials. 2002;20:1102-1109.
15. Monheit AC. Persistence in health expenditures in the short run: prevalenceand consequences. 2003;41(suppl):III53-III64.
J Allergy Clin
16. Stempel DA, McLaughlin TP, Pendergraft TB, Stanford RH, Fuhlbrigge AL.Patterns of asthma control: A 3-year analysis of patient claims. In press.
17. Newhouse JP. Risk adjustment: where are we now? 1998;35:122-131.
18. Welch WP. Regression toward the mean in medical care costs: implications forbiased selection in health maintenance organizations. 1985;23:1234-1241.
19. Dudley RA, Medlin CA, Hammann LB, et al. The best of both worlds? potentialof hybrid prospective/concurrent risk adjustment. 2003;41:56-69.
20. Sales AE, Liu CF, Sloan KL, et al. Predicting costs of care using a pharmacy-basedmeasure risk adjustment in a veteran population. 2003;41:753-760.