An Assessment of the CHIP/Medicaid Quality Measure for ADHD

Justin Blackburn, PhD; David J. Becker, PhD; Michael A. Morrisey, PhD; Meredith L. Kilgore, PhD; Bisakha Sen, PhD; Cathy Caldwell, MPH; and Nir Menachemi, PhD, MPH

Attention-deficit/hyperactivity disorder (ADHD) is the most commonly diagnosed behavioral disorder among children and youth,1 at an estimated prevalence of 5% to 11%, and is a growing public health concern.2 Children with ADHD can be inattentive, disruptive, and impulsive, and have been shown to experience more injuries3-5 and incur higher overall healthcare costs.6-9 ADHD diagnosis and treatment is often parent- or teacher-initiated and typically consists of psycho-stimulant medications and/or behavior therapy—both of which require careful clinical monitoring to ensure proper dosing and adherence.10-12 Currently, there is no consensus on the relevant population-based outcomes of ADHD treatment to assess quality of care.13 Clinicians and parents have individual treatment goals,14 and previous studies have focused on symptom assessment from the child and their parents and teachers.15,16
As part of the Children’s Health Insurance Program (CHIP) Reauthorization Act,17 an initial core set of quality measures for CHIPs and state Medicaid agencies was developed to foster quality improvement efforts over time.18-27 Evaluation of these quality measures, which are collectively known as the Child Core Set, is ongoing, with measures being retired if they fail to meet criteria of importance, scientific acceptability, feasibility, and usability, as set forth by the National Quality Forum (NQF).28 Among these measures is one for follow-up care for ADHD, calculated from administrative data. The measure is designed to capture the extent to which publicly insured children (ie, CHIP, Medicaid) newly treated for ADHD have adequate medication adherence and follow-up visits indicative of physician monitoring. A recent review of child mental health quality measures by Zima and colleagues gave a “D” rating to this measure, asserting there was no conclusive supporting evidence.29 Similar measures are a focus of the Agency for Healthcare Research and Quality and CMS' Pediatric Quality Measures Program.30
Our purpose was to describe the ADHD quality measure in the Child Core Set using data from a single state. First, we identified the proportion of children meeting the measure. We evaluated whether population characteristics, such as race/ethnicity, age, or rural status, were challenges to meeting the measure. Second, we identified potential barriers to meeting the measure. We measured the specific components that were the most difficult to meet, such as timeliness of follow-up visits, and we explored whether exclusion criteria would be useful to understand the measured population. Third, we measured the association of ADHD treatment with selected outcomes. Although cognizant of the difficulties of accurately capturing the effectiveness of ADHD treatment using administrative data, we determined whether meeting the measure conferred any benefit on outcomes, such as injury rates and healthcare costs.
Data and Study Population
This study used 1999 to 2012 claims data from Alabama’s stand-alone CHIP program, “ALL Kids,” to calculate the measure of ADHD quality treatment. Throughout this time period, ALL Kids coverage was available in 12-month enrollment periods to Alabama children younger than 19 years with family incomes between 100% and 200% of the federal poverty level (FPL). In October 2009, income eligibility was expanded to include up to 300% FPL. Annual premiums and co-payments for services were determined primarily by FPL category. Family incomes at 100% to 150% FPL were categorized as the “low-fee group.” Depending on year, this group faced co-payments ranging from $0 to $3 for a physician visit. Those children at 150% to 200% FPL (“fee group”) faced co-payments of $5. Children received exemption from cost sharing (“no-fee group”) if they met federal criteria, such as Native American heritage. The October 2009 expanded-eligibility group (“expansion group”) had incomes of 200% to 300% FPL and resembled the cost-sharing structure of the fee group.
Enrollee characteristics were derived from ALL Kids claims and enrollment data, which are maintained through a contract with Blue Cross/Blue Shield of Alabama, as well as from enrollment files provided by ALL Kids. Race/ethnicity was self-reported during the enrollment application. We grouped children into 3 racial/ethnic categories: white, black, and other (includes Hispanic/Latino children of any race/ethnicity). To identify children in rural areas, we used rural urban commuting area codes based on enrollees’ zip codes.
Construction of the ADHD Quality Measure
The ADHD Measure, “Measure ADD: Follow-up Care for Children Prescribed Attention-Deficit/Hyperactivity Disorder (ADHD) Medication,” was constructed according to the CMS technical manual as follows.18 First, children aged 6 to 12 years who received medications specified by the Healthcare Effectiveness Data and Information Set (HEDIS) for the treatment of ADHD were identified from claims data. Eligible children must have been continuously enrolled in the 120 days prior and 300 days following their initial ADHD prescription date (the index prescription start date [IPSD]) (eAppendix Figure [eAppendices available at]). This includes a minimum of 120 days prior to the IPSD without claims for ADHD medications (the “negative medication history”), 30 days after the IPSD (the Initiation Phase), and 300 days following the IPSD (the Continuation and Management Phase [C&M Phase]). Information about follow-up visits and medication use were used to determine whether children met the measure’s criteria.
The first component required a follow-up visit with a practitioner with prescribing authority within the first 30 days after the IPSD. The second component required at least 2 follow-up visits with practitioners with prescribing authority, as well as medication claims equating to at least 210 of 300 days (70%) with medication coverage. We calculated the measure for children with IPSDs from March 1, 1999, through December 31, 2011, in order to ensure complete observable follow-up time for the measure calculation. As specified by CMS, children were excluded if they received inpatient treatment for mental health or substance abuse at any point during follow-up.18 Additionally, we excluded 24 outlying children with greater than 38 mental health outpatient follow-up visits, representing extreme values of less than 1% of all children.
We calculated the number of children who failed to meet the specific components of the measure outlined in the CMS technical specification manual criteria. As specified, the measure was calculated and reported as 2 components with different denominators: rate 1 covering the first 30 days following the IPSD and rate 2 covering 300 days following the IPSD. Rate 2 is calculated only for those meeting rate 1 with a follow-up visit within 30 days. Although we calculated both measures as specified, we focused on children with 300 days of continuous enrollment after the IPSD (regardless of follow-up within 30 days) in order to comprehensively assess characteristics of meeting and not meeting the measure. Thus, our denominator included children with an IPSD who met the age requirement, negative medication history requirement, and mental health and substance abuse hospitalization exclusion criteria, and were continuously enrolled for 300 days.
We then determined which component of the measure was not met by holding the denominator constant and calculating the proportion of children that: 1) had no 30-day follow-up, 2) had <210 days of medication coverage, and 3) did not have 2 follow-up visits between days 31 and 300 of follow-up. For example, if an eligible child had no follow-up visit within 30 days, he or she did not meet the measure. However, we still determined if the child had medication coverage of 210 days or 2 follow-up visits in days 31 to 300 following the IPSD. Furthermore, we calculated alternative measure specifications to illustrate the sensitivity of the measure to its definitions. For example, although the Initiation Phase measure specifies follow-up within 30 days, we calculated rates at 45 and 60 days. Likewise, we calculated different medication coverage periods (ie, 210, 150, 90, and 30 days).
We used claims data to construct covariate and outcome variables, including health expenditures and injury-related utilization. We used primary International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) diagnosis codes from the HEDIS Mental Health Diagnosis Value Set to determine if a child had any mental health diagnoses during the 120-day negative medication history. We examined total expenditures and expenditures for emergency department (ED) visits, hospitalizations, all medical costs (excluding pharmacy claims), and pharmacy claims during the measurement period. Health expenditures were defined as payments made on behalf of ALL Kids, excluding co-payments. All expenditures were inflation-adjusted to 2012 dollars using the Consumer Price Index (all items).
ED visits were defined as claims with Revenue Center codes 450 to 452, 456, or 459, or Current Procedure Terminology–4 procedure codes from 99281 to 99285 that did not result in hospital admission. Hospitalization was defined by place of service codes. To identify children with injuries, we used primary ICD-9-CM diagnosis codes 800 to 999 in any position to create a binary indicator for injuries treated in the ED or inpatient setting. Outpatient injury visits are considered unreliable, and similar methods have been used for comparisons of ADHD treatment.31
Statistical Analysis
Bivariate comparisons of measure specifications used the 2-sample test of proportions. Expenditure category means were compared with t tests. Multivariable comparisons were estimated with logit regression models to predict differences in characteristics of meeting the measure as well as the association of meeting the measure with any injury, ED, or hospitalization, clustered at the individual level to account for correlated errors from children with multiple IPSDs. We report marginal effects, which represent the percentage point change in the likelihood of the outcome for each predictor, holding all others factors constant at their mean values. For health service expenditures, we estimated Poisson models with a log-link function to account for skewness of the data. Marginal effects from this model reflect the predicted incremental costs or savings resulting from meeting the measure.
Between 1999 and 2012, ALL Kids had an average enrollment of 61,251 children annually. The average annual rate of children with ADHD medication claims was 7.6%, and this increased from 1.7% (389 children) in 1999 to 11.1% (9604 children) in 2012 (Figure). Among children receiving ADHD medication each year, on average, 10.8% (544 children) were eligible for inclusion in measure calculation on the basis of age, negative medication history, continuous enrollment, and no mental health/substance abuse hospitalizations. Eligibility among children receiving ADHD medications declined over time from the peak of 17.0% in 2000 to 7.8% in 2012 (not shown).
We calculated the CMS measure, and reported it as 2 rates: rate 1, the Initiation Phase, and rate 2, C&M Phase (Table 1). For rate 1, we identified 10,822 eligible IPSDs among 9693 children who met the definition of age, negative medication history, continuous enrollment through 30 days, and no mental health/substance abuse hospitalizations, of whom 4142 (38%) had a follow-up within 30 days. Of the 4142 IPSDs meeting rate 1, 2897 IPSDs among 2801 children met the continuous enrollment requirements for calculation of rate 2. Of those, 1083 (37%) had 2 additional follow-up visits and 210 days of medication coverage during the C&M Phase (days 31-300 from IPSD), thereby fulfilling the measure criteria. In our expanded denominator, we identified 7615 IPSDs among 6924 children that met all the measure eligibility requirements for rate 1 and were continuously enrolled for a minimum of 300 days. For this specification, the same 1083 IPSDs that met rate 2 were the numerator, but 4718 additional IPSDs were included despite not receiving an initial follow-up visit within 30 days, which was a requirement for the rate 2 denominator. Allowing an additional 15 days would increase rate 1 by 13 percentage points and increase the number of IPSDs eligible for rate 2 by 1421.
The reasons for failing to meet the measure are described for the expanded denominator of IPSDs (Table 1). Although 691 children were included with multiple IPSDs ranging from 2 (n = 429; 62%) to 7 (n = 1; 0.1%), henceforth we describe the unit of analysis as “children,” rather than IPSDs, for ease of interpretation. The lack of complete medication coverage was a primary reason for failing to meet the measure (65%). However, the majority of children continued to file ADHD medication claims beyond the initial prescription (90%), and 60% did so for at least 150 days.
Characteristics of children meeting and not meeting the measure are shown in Table 2. Among black children eligible for measure calculation, 9.8% met the measure compared with 15.8% of white children. Otherwise, characteristics between children meeting and not meeting the measure did not differ substantially.
Children meeting the measure were observed to have a greater frequency of ED visits (26.1%) than those who did not (21.9%) (P = .002), but they were not significantly more likely to have an ED- or inpatient-treated injury: 14.4% versus 12.8%, respectively (P = .154) (Table 3). Mean annual prescription drug costs were higher among children meeting the measure ($1678 vs $916; P <.001), as were medical costs, excluding pharmacy claims ($1698 vs $1160; P <.001), and therefore, overall expenditures ($3376 vs $2076; P <.001).
After adjustment for covariates in the logit model, the likelihood of meeting the measure declined monotonically with age (Table 4). Relative to 6-year-olds, 12-year-old children were 13.0 percentage points less likely to meet the measure (P <.001). Children with a prior mental health diagnosis were 5.8 percentage points more likely to meet the measure (P <.001). Compared with white children, black children were 5.5 percentage points less likely to meet the measure (P <.001). Meeting the measure was associated with a 2.2 percentage point–increased likelihood of an ED visit or hospitalization for an injury (P = .032). Similarly, meeting the measure was associated with a 4.3 percentage point–increased likelihood of an ED visit during the measurement year (P = .001). We observed no relationship between meeting the measure and hospitalization (P = .214). In an additional analysis of 2019 children who had an additional year of follow-up after completion of the measure calculation, there were no statistically significant differences observed in the rates of injury, ED visits, or hospitalizations between those meeting versus not meeting the measure (available on request).
The average annual predicted total expenditures per child were $2261 for the measurement year (eAppendix Table). Children meeting the measure had total expenditures that were $909 higher (P <.001). In particular, prescription drug expenditures were $514 higher and medical expenditures were $375 higher (P <.001 for each) for children meeting the measure.
We used Alabama CHIP data from 1999 to 2012 to report trends in eligibility for inclusion in the CMS measure; disparities in measure adherence by age, race/ethnicity, and income level; sensitivity of meeting the measure to the definitions of its components; and the association with injuries and health expenditures. Our historical calculations reported here are slightly lower than other states’ most recently reported data, including those of Alabama.32 Using these data, we concluded there are substantial concerns with the measure in its current state based on the NQF major criteria for measure evaluation: importance, scientific acceptability, feasibility, and usability.28 Specifically, we found the measure lacks sufficient details, which limits its usefulness as a tool to improve ADHD treatment among publicly insured children. We present 5 main conclusions from this analysis, which may be useful in improving the measure.
First, the results indicate that only a small fraction of children with ADHD medications are eligible for inclusion in the measure, and that number is declining. In recent years, more than 10% of ALL Kids enrollees received ADHD medications, yet less than 2% of enrollees were eligible for measure calculation. Although the percent of enrollees aged 6 to 12 years has been increasing, a faster rate of growth is observed among children over age 12, and they are precluded from the measure. Additionally, the continuous enrollment requirement precluded many children from eligibility in the calculation of the measure. During this time period, the immediate renewal rate in ALL Kids was approximately 60%.33 Although program retention may vary among states, one objective of the Child Core Set Measures is to enhance state-by-state comparisons.34 Given that many children receiving ADHD medication are not eligible for calculation of the measure based on age or enrollment criteria, there are concerns that these children are not representative of the typical child on ADHD medications or that quality of care is not measured equitably. Thus, the population is important, but the measure appears to lack generalizability to the intended population, and so its scientific acceptability to capture changes in overall performance in the care of ADHD among children is decreasing.
Second, there are systematic differences in the characteristics of children who meet the measure, including age, prior mental health diagnoses, and race/ethnicity. Racial/ethnic disparities within ADHD have been previously observed; specifically, minorities have different perceptions about symptoms,35 are less likely to be diagnosed with ADHD,36,37 and have lower rates of ADHD medication use.38 Future consideration of the quality of care for ADHD among children in minority and other subgroups is needed in order to increase the usability of the measure’s goal of improving quality equally among all publicly insured children.
Third, meeting the measure is sensitive to both the allowable medication coverage criterion and to the time window for the initial follow-up visit. Failing to meet the measure was driven by lack of medication coverage and lack of 30-day follow-up visits. The proportion of children who received a follow-up visit within weeks of the 30-day window increased the number of those who met the measure by as much as 20 percentage points, suggesting the measure fails to capture clinically relevant, albeit untimely, follow-up care.
Conversely, the number of children taking ADHD medications without an initial follow-up visit is concerning because medication therapy is optimized when dosing is properly managed.39-43 For children, the range of dosing needed to achieve therapeutic effects is broad.39 Therefore, titrating the stimulant medication dose is widely accepted to achieve the desired response while balancing adverse side effects.39-44 Although having a 30-day follow-up visit does not guarantee proper dosing, at least some contact with a medical provider may be beneficial in monitoring side effects, including the potentially serious ones like aggression or emotional instability.44 Concerns with this measure have been present since the initial draft of the Child Core Set.45 For example, providers were concerned that only an in-person, and separately billable, visit counted as a follow-up. The CMS technical manual does permit C&M Phase follow-up visits to be conducted via telephone; however, like many public health insurance agencies nationwide, ALL Kids does not reimburse providers for such telephone-based services.
Fourth, our analysis raises issues regarding the validity of the measure to capture clinically relevant and evidence-based care quality. Given that the negative medication history period is 120 days, some fraction of children are not completely naïve to ADHD medication. It is unclear whether these children need the same level of clinical monitoring as first-time users, but it is well known that many children take medication “holidays” coinciding with school breaks, such as during the summer.46 Although clinical guidance regarding planned medication holidays is lacking and the long- and short-term effects of them are not well understood,10 the measure does not adequately account for this behavior.43 Furthermore, children directed by a physician to discontinue medication after less than 210 days fail to meet the measure. Thus, we believe that simply identifying the rate of children meeting or not meeting the measure may be insufficient for quality improvement.
Finally, as with any administrative data measure, claims data are unable to capture the presence of symptoms or school performance, which are key outcomes of ADHD treatment. Using information available from claims data in accordance with the NQF feasibility criterion, we calculated health service utilization, costs, and injury rates among children meeting and not meeting the measure as an attempt to assess whether the care received under the measure confers any measurable benefit. These outcomes are associated with ADHD diagnosis,3-6,47,48 but it is an open question whether injury rates are expected to decline among treated versus untreated children. We observed that injury rates were higher for those meeting the measure, as were total costs, largely driven by medical and pharmaceutical costs associated with measure criteria, including ADHD medication.
Quality measures must balance the use of currently available data with ideal measure components. Given the importance of improving the quality of care, measures with minimally acceptable evidence may be used; however, it is imperative that these suboptimal measures evolve as health data change and new data, such as electronic medical records, become more widely available.49 We find that the current ADHD quality measure lacks specific information to guide state public health insurance agencies on where to target interventions to improve the care. For example, failure to meet this measure may be related to a variety of factors, including access to care, medication adherence, scheduling constraints, or attitudes toward ADHD among parents and children. Furthermore, the lack of measurable outcomes that are relevant on a clinical and/or population scale further inhibit the usefulness of this measure. To improve the usability of this measure, we recommend development of additional components of the measure that will allow for more specific feedback to agencies to improve care and demonstrate a benefit of care consistent with the measure.
Our assessment should be interpreted in light of its limitations. First, a single state analysis may not be representative of the experiences of other states or agencies with different policies and provider networks. Second, we acknowledge the possibility of selection bias in our analysis of injuries and healthcare costs among those meeting the measure. This could occur if children with more severe manifestations of ADHD are more or less likely to receive follow-up care and demonstrate adherence with medications. Alternatively, parents more likely to follow up may simply be high consumers of healthcare. Thus, estimates could be biased toward or away from the null. Finally, the measure calculation is based on treatment for ADHD, which may not be of uniform severity. Although claims data are unable to capture disease severity, we have controlled for previous mental health diagnoses and utilization in regressions for some risk adjustment. However, without more detailed information, we cannot be certain to have controlled for confounding by disease severity.
Monitoring the delivery and quality of care under state public health insurance agencies is an important process. State reporting of quality measures, including this measure for ADHD, has not yet been tied to incentives for improvement or penalties for performance failure. Before this can occur, thorough validation of all measures within the Child Core Set is necessary to ensure fairness of interagency comparisons. Assessment of the measures to ensure public health and clinical relevance is crucial, and as such, we found several concerns with the current ADHD measure reported by public health insurance agencies. Therefore, we recommend further empirically driven revisions to the measure. We recommend an improved understanding of the details behind why children fail to meet the measure, as well as improving clinical process and outcome metrics in order to better measure the true quality of ADHD treatment among publicly insured children. 
Print | AJMC Printing...