Much has been written about trends in Medicare Part D formulary design and consumers’ choice of plans, but little is known about the magnitude of claims rejections or their clinical and administrative implications. Our objective was to study the overall rate at which Part D claims are rejected, whether these rates differ across plans, drugs, and medication classes, and how these rejection rates and reasons have evolved over time.

We performed descriptive analyses of data on paid and rejected Part D claims submitted by 1 large national longterm care pharmacy from 2006 to 2010. In each of the 5 study years, data included approximately 450,000 Medicare beneficiaries living in long-term care settings with approximately 4 million Part D drug claims. Claims rejection rates and reasons for rejection are tabulated for each study year at the plan, drug, and class levels.

Nearly 1 in 6 drug claims was rejected during the first 5 years of the Medicare Part D program, and this rate has increased over time. Rejection rates and reasons for rejection varied substantially across drug products and Part D plans. Moreover, the reasons for denials evolved over our study period. Coverage has become less of a factor in claims rejections than it was initially and other formulary tools such as drug utilization review, quantity-related coverage limits, and prior authorization are increasingly used to deny claims.

 Examining claims rejection rates can provide important supplemental information to assess plans’ generosity of coverage and to identify potential areas of concern.

Our findings suggest that medication-specific and planwide rejection rates could be useful information, especially for policy makers and clinicians.

		Policy makers could monitor rejection rates across plans to identify potential access problems for Medicare beneficiaries, to inform the Part D appeals process, and to make adjustments to regulatory guidance.

		More specific to the nursing home population, policy makers could consider using rejection rates in decisions about plan assignment for dually eligible residents or which plans are eligible to serve dually eligible individuals.

		Similarly, providers and pharmacies could monitor rejection data in establishing formularies and prescribing practices more generally.

The US government relies on private plans to administer the Medicare Part D prescription drug benefit, including the establishment of premiums, pharmacy networks, and formulary design. Within limits, Part D plans (PDPs) have flexibility to shape enrollees’ drug use through levers such as coverage, cost-sharing, and utilization management techniques including prior authorization, step therapy, and quantity limits. The underlying expectation of this approach is that informed consumers will choose the plan that best suits their needs and that price competition across plans will optimize government payments for drugs.

Although Part D includes special protections for nursing home residents, the program’s administrative reliance on private plans and the emphasis on consumer choice is similar across institutional and community settings. In addition to nursing home residents having relatively high levels of medication use and physical and cognitive frailty as compared with other Medicare beneficiaries, a key difference in the nursing home setting is that almost two-thirds of long-stay residents are dually eligible for Medicare and Medicaid.

 Part D’s implementation shifted medication coverage for duals from Medicaid to Medicare and randomly assigned these individuals to plans with premiums at or below regional benchmark rates set by the Centers for Medicare & Medicaid Services (CMS). Under Part D, nursing homes and the pharmacies with which they contract no longer function under a state’s Medicaid policies for dually eligible residents but instead work across multiple plans, each of which may have different formulary designs and administrative procedures.

 The transition to Part D in the nursing home sector has not always been smooth, with early concerns about coverage adequacy and administrative burdens expressed by physicians, pharmacists, and administrators working in nursing homes.

Little has been written about the extent of claims rejections or their clinical and administrative implications. One previous study using data on claims in rejected status at the end of 2006 found considerable variation in the reasons for rejection across medications and in the relative rejection rates across PDPs.

 However, no published information exists about the overall rate at which Part D claims are rejected; whether these rates differ across plans, drugs, and medication classes; and how these rejection rates and reasons have evolved over time. To address these questions, we examined data on paid and rejected Part D claims submitted by 1 large national long-term care pharmacy (LTCP) over the initial 5 years of Medicare Part D.

We obtained data on all paid and rejected Part D claims from Omnicare, Inc (Covington, Kentucky), the nation’s largest long-term care pharmacy, operating in 47 states and serving approximately half of US nursing home residents. A claim is defined as a demand of payment for a particular drug, for a particular individual, on a particular date. Data include Part D claims for nursing home and assisted living residents, although more than 8 in 10 claims are estimated to be for nursing home residents. Data capture all claims filed in the month of March for each of 5 study years (2006-2010) (although we purposely avoided using data from January because of potential transition challenges at the beginning of the year, the choice of March was somewhat arbitrary. March 2010 also was the latest month available when the study began). All observations include the claim date; unique identifiers for residents and facilities; the National Drug Code (NDC) of the product; the plan to which the claim was submitted; and up to 3 reasons for denial of rejected claims. For ease of presentation, we group rejections into 3 broad categories (see 

): 1) Product not covered—capturing instances where the product is not covered; 2) utilization management techniques&mdash;capturing instances where coverage applies only after plans approve necessary documentation from pharmacies and clinicians (prior authorization), where less expensive medications must first be tried and failed before more expensive medications are dispensed (step therapy), and instances where plans limit the number (or amount) of drugs covered within a certain time period (quantity limits and refill too soon); and 3) administrative rejections&mdash;including instances where claims have non-matched pharmacy or member identification numbers and missing/invalid information about the prescriber, patient, or prescription itself. Importantly, administrative rejections can stem directly from coverage restrictions (eg, missing or inadequate justification for a “dispense as written” prescription order is the second-most prominent code in this category).

We define the total number of claims as the sum of paid claims and the subset of rejected claims that remain unpaid at the end of the month. We are able to flag claims that were rejected multiple times (around 1 in 4 rejected claims) and claims that were rejected and then paid during the 1-month windows of the 5 study years. We count these flagged claims only once. We define the total number of rejections to include all rejected claims, regardless of whether the claim was paid subsequently. If a claim was rejected multiple times for the same reason(s), we count these as 1 rejection. If a claim was rejected multiple times for different reasons, we count each unique set of reasons separately. We are unable to observe the life cycle of claims outside our 1-month windows (eg, if a claim was rejected in March and paid in a subsequent month, we have no record of the later payment). To calculate the rejection rate, we divide the total number of rejections by the total number of submitted claims. We present the rejection rate by year, PDP, and product. We consider generic and brand formulations of the same molecule as distinct products for our analyses; similarly, we treat different formulations of the same molecule (eg, tablets, solutions, and extendedrelease formulations) separately. We exclude 248,026 rejections (7% of all rejections) that were rejected due to problems transmitting claims electronically, such as “host processing error” or “system unavailable.” Based on conversations with the data provider, these rejections typically are resubmitted automatically, either to be resolved or rejected for another reason. We also exclude from our analyses the small number of claims that could not be matched to a drug name (1.6% of rejected claims; 0.1% of paid claims) and that were missing a rejection reason (0.02%).

Our data include approximately 450,000 unique individuals and 4 million total claims in each of the 5 study years (

). The overall rejection rate ranged from 14% to 19% over the 2006 to 2010 time frame, increasing slightly in recent years. The percent of rejections due to products not being covered declined considerably over the study period, from 21% of all rejections in 2006 to 10% in 2010. In contrast, rejections due to utilization management techniques such as prior authorization, drug utilization review, and other coverage restrictions grew in prominence, from 33% of all rejections in 2006 to 44% in 2010. Within the utilization management category of rejections, the most prominent subcategories of rejections were “refill too soon” (43% of utilization management rejections), “Drug Utilization Review Reject Error” (30%), “plan limitations exceeded” (17%), and “prior authorization required” (8%) (eAppendix A). Administrative rejections were consistently high over the study period (approximately 43% of all rejections, on average). Specific codes comprising most administrative rejections were non-matched pharmacy numbers (31% of administrative rejections); missing/ invalid information for requests to “dispense as written” (ie, requests to fill prescriptions for brand-name drugs, even though generic substitutes might be available) (8%); nonmatched plan member numbers (7%); and missing date of service (6%).

details product-level rejections for our most recent year of data (2010), with the top and bottom panels describing rejections for the most prescribed medications and medications with the highest rejection rates among drugs with at least 5000 claims, respectively. Among the 20 most prescribed drugs (13 of which are generics), the rejection rate was between 13% and 19%, with denials generally divided between administrative rejections and utilization management, and rarely due to lack of coverage. For the 20 most commonly rejected drugs (9 of which are generics), the rejection rates (23%-62%) and reasons for denial varied more widely. Lack of coverage factored more prominently into these rejections, especially for some alternate formulations. Administrative rejection codes accounted for more than half of denials for 10 out of 20 medications.

 shows rejection rates over time for 7 classes commonly used in long-term care settings—antidepressants, angiotensin receptor blockers, atypical antipsychotics, cholinesterase inhibitors, long-acting opioids, nebulized inhalants, and osteoporosis medications (

 lists drugs by class). Other than the rejection rates for nebulized inhalants, which declined from 2008 to 2010, class-level rejection rates declined initially and then increased in subsequent years. Some increases were relatively large—for instance, after falling to a low of 16% in 2007, the rejection rate for long-acting opioids increased to 28% in 2010. Rejection reasons varied across classes (see 

displays rejection rates and the reasons across plans with higher claims volume in 2010. If a company had multiple PDPs nationwide, the information is aggregated across these plans. The overall rejection rates varied (6%-30%), as did the distribution of rejection reasons. Among plans with higher rejection rates, administrative rejections were relatively prominent. The 3 plans with the highest proportion of rejections for administrative reasons (74%, 68%, and 65%) were among the 3 plans with highest rejection rates overall (30%, 24%, and 29%, respectively). Translated to an administrative rejection rate (overall rejection rate x proportion of rejections for administrative reasons), the rate at which claims were rejected because of administrative reasons generally was between 2% and 9% of claims (not shown). Relatively high rates of administrative rejections arose in smaller and larger plans alike. The rates of claims rejections in the other 2 categories did not convey anything consistent about plans’ overall rejection rates.

Over the initial 5 years of Medicare Part D, nearly 1 in 6 drug claims for beneficiaries living in nursing homes and other long-term care settings was rejected. Although one might have expected the rejection rate to decline over time after the initial transition to the program and as pharmacies and clinicians grew more accustomed to working across private plans, this has not occurred. After an initial decline, the rejection rate has increased slightly each year. At the same time, reasons for denials have evolved considerably. Lack of coverage has become less of a factor in claims rejections than it was initially; increasingly, other formulary tools such as drug utilization review, quantity-related coverage limits, and prior authorization are used to deny claims. Although these results are consistent with broader trends concerning PDP formularies, somewhat more surprising is the persistence of administrative claims rejections throughout the study period, a feature that likely reflects the complexity of working across multiple plans and policies in a given year, as well as changes in coverage policies over time.

Not surprisingly, rejection rates and reasons varied considerably across products. Medications most commonly used by nursing home residents generally have below average denial rates (around 1 in 7 claims), with coverage issues rarely noted. Given the relatively low cost of many of these drugs (9 out of 10 most commonly used drugs are generics), these features are unsurprising. Among drugs with the highest rejection rates, explanations for denials are complex and vary across products, which include generics, brand drugs with generic alternatives, and brand drugs without generic alternatives.

Among the most rejected generics, half of the denials for pantoprazole sodium extended release tablet (a proton pump inhibitor for conditions such as gastroesophageal reflux disease) were due to the product not being covered. The primary reason for rejections of fluticasone propionate nasal spray (for asthma and allergy symptoms) stemmed from claims having missing/invalid information about days-supply, a problem also affecting other solutions and inhalation products. Three-fourths of rejections for potassium chloride solution (for nutritional deficiencies) were due to non-coverage, a feature likely reflecting a CMS ruling that the formulation did not meet Part D coverage requirements and the subsequent shift by PDPs to reimburse only non-liquid formulations of the drug.

Denials for brand name drugs with generic equivalents likely reflect PDPs’ efforts to promote use of cheaper generic alternatives. The high rejection rate for Flomax (tamsulosin) (for symptoms of an enlarged prostate) largely stemmed from claims having missing/invalid “dispense as written” information, an issue that presumably arose following the availability of generic tamsulosin in early 2010. Among the handful of brand drugs without generic equivalents, an example worth highlighting is Procrit (epoetin alfa), an expensive blood modifier used to treat anemia due to chronic renal failure and other select conditions. More than three-fourths of Procrit’s rejections were due to prior authorization issues, something not unexpected given the drug’s high cost and potentially harmful side effects.

A striking feature of our product-level findings is that 6 out of 20 medications with the highest rejection rates in 2010 are used to treat pain (oxycodone/acetaminophen, oxycodone hydrochloride, morphine sulfate extended release, fentanyl patch, propoxyphene napsylate/acetaminophen, and Lidoderm (lidocaine). With the exception of propoxyphene napsylate/acetaminophen (an opioid combination of questionable effectiveness and safety in older adults

) and Lidoderm (an expensive topical anesthetic patch used to treat pain from shingles and other sources of neuropathic pain), denials for pain medications arise primarily from administrative rejections and utilization management techniques. An illustrative example is that nearly 1 in 3 claims was denied for the fentanyl patch in 2010—a rate nearly double its 2006 level; and administrative rejections and utilization management accounted for almost all of these rejections. Although rejection rates for pain medications could reflect their considerable abuse and diversion potential, concerns must be weighed against the need for timely access to effective medications for individuals living with chronic pain.

Rejection rates and reasons also vary considerably across PDPs. In the context of a facility’s residents being enrolled in multiple plans, this variation presents administrative challenges for facility and pharmacy staff and may undermine the predictability of coverage across residents with similar conditions. Although it is difficult to convey a clear, concise story in the context of the PDP rejection rates, plans’ higher administrative rejection rates generally imply higher rejection rates overall. Further context for administrative rejections relates to continuing challenges that providers and pharmacies face in updating residents’ Medicaid eligibility status, delays which can result in enrollment gaps and incorrectly collected copayments.

Nonetheless, the cross-plan variation in the extent to which these issues lead to rejected claims implies that there may be room for improvement at plans with especially high administrative rejection rates.

It is unclear what the optimal or expected rejection rate should be in Part D generally or in the long-term care setting specifically. Besides administrative errors that must be corrected before payment, plans have legitimate reasons to deny claims; and the optimal rejection rate is not zero. Prior  authorization, for instance, could add valuable safeguards when prescribing is of questionable efficacy or appropriateness (eg, in the historically problematic area of psychotropic drug use among elderly nursing home residents

). Similarly, step therapy edits could push clinicians to prescribe less costly medications that are clinically similar or, in the case of brand-to-generic comparisons, equivalent. Nonetheless, use of these strategies creates a tension between the push to lower costs and the importance of ensuring access to needed medications.

The clinical impact of claims rejections on long-term care residents is unclear. We do not have information about beneficiaries’ diagnoses or health outcomes and are unable to assess the potential impact of denials on individuals’ health or on the overall appropriateness of prescribing. Similarly, we do not have information about the absolute or relative administrative burden or costs associated with rejected claims. It is likely that some denials are rectified relatively easily, while others require more coordinated effort. Because of the limited time frame (1 month each year) for which we have data, we are unable to estimate the extent to which individuals switch or discontinue medications because of claims denials (or, conversely, obtain their needed medication in the end). Prescribing behavior also may adjust to PDPs’ formularies and administrative practices (eg, if clinicians reduce prescribing a particular medication because it is difficult to obtain), something we are unable to observe. Important context for these points is that regulations require that nursing homes adhere to residents’ clinical care plans regardless of financial coverage, meaning that clinicians must either work within plans’ extant formulary constraints or leave facilities and/or pharmacies to shoulder the financial cost of medications. Finally, although the same PDPs serve individuals across institutional and community-based settings, our data are from individuals living in long-term care settings and come from a single (albeit very large) LTCP. Thus, generalizing our findings to the Part D program as a whole or even to the entire long-term care sector should be done with caution.

Nonetheless, our results have several implications for policy. Our findings suggest that medication-specific and planwide rejection rates could be useful information for policy makers, beneficiaries, and clinicians. Policy makers could monitor and possibly report rejection rates across plans to identify potential access problems for Medicare beneficiaries, to inform the Part D appeals process, and even to make adjustments to regulatory guidance if necessary. Given uncertainty about the optimal or appropriate rejection rate in Part D, related oversight likely would focus on identifying outliers at the plan, product, or class levels for further investigation. For example, CMS currently requires that plans cover at least 1 formulation of all medications in selected medication classes.

 CMS could consider rejection rates in decisions about whether to incorporate special protections around formulary coverage or the use of utilization management for these or other drugs. Although beneficiaries could use rejection rates in evaluating plans (eg, to compare plans with comparable coverage on drugs of interest), previous research suggests consumers already are overwhelmed by available information and could benefit from having fewer, more targeted data points.

 More specific to the nursing home population, policy makers could consider using rejection rates in decisions about plan assignment of dually eligible residents (eg, not assigning individuals to plans where medication access looks especially problematic) or even about which plans are eligible to serve dually eligible individuals altogether. Similarly, providers and the pharmacies they work with could monitor rejection data in establishing formularies and prescribing practices more generally.

Throughout the brief history of the Medicare Part D program, comparing generosity of coverage across private PDPs has been a key point of interest for consumers, policy makers, and researchers.

Claims rejections have been an unobserved feature of these comparisons to date. Our research shows that examining rejection rates across plans and medications can provide important supplemental information to assess plans’ generosity of coverage and to identify potential areas of concern. Going forward, information about claims rejections should be more systematically monitored and analyzed for oversight purposes, not only to identify administrative challenges that arise in the Part D marketplace but also to ensure the program is working well for Medicare beneficiaries as a whole.

From Department of Health Care Policy (DGS, HAH), Harvard Medical School, Boston, MA: Department of Health Services, Policy, and Practice (LK), Alpert Medical School of Brown University, Providence, RI; Hebrew SeniorLife (SLM), Institute for Aging Research, Boston, MA; Omnicare, Inc (BJZ), Livonia, MI.

 This work was supported by a grant from the National Institute on Aging (NIA), grant number R01 AG034085. Dr Stevenson also was supported by an NIA Career Development Award (K01 AG038481). Dr Huskamp also was supported by a Robert Wood Johnson Foundation Investigator Award in Health Policy Research. Dr Mitchell was supported by NIA grant number K24AG033640.

Articles

POLICY

This study presents data on paid and rejected claims submitted by 1 large long-term care pharmacy over the initial 5 years of Medicare Part D.

Medicare Part D Claims Rejections for Nursing Home Residents, 2006 to 2010

To describe 1 pediatric integrated delivery system’s experience with the influenza A (H1N1) pandemic in 2009 to illustrate the benefits of coordination, scale, scope, and flexibility in handling large volumes of patients in many locations.

Through multidisciplinary planning across a large, multisite pediatric delivery system, an effective 3-tier plan was developed to handle anticipated increased volumes associated with the fall 2009 influenza pandemic in the Philadelphia region.

Patient demand for services increased to record-setting levels, particularly for emergency department visits and phone calls. The 3-tier plan of response allowed for graded and appropriate response to volumes that more than doubled in many locations. Measured by wait times and leftwithout- being-seen rates, the system appeared to match capacity to demand effectively. Lessons learned in terms of successes and challenges are useful for future planning.

The experience of 1 pediatric delivery system in handling increased volume due to pandemic influenza may hold lessons for other organizations and for policy makers seeking to improve the preparedness, quality, and value of healthcare. These experiences do not imply the need for vertical integration with ownership, but do support tight coordination, communication, integration, and alignment in any management structure.

A 3-tier plan of response was developed by an integrated pediatric delivery system to handle anticipated increased volumes associated with the fall 2009 influenza A (H1N1) pandemic in the Philadelphia region.

		The 3-tier plan allowed for graded and appropriate response to volumes that more than doubled in many locations.

		Measured by wait times and left-without-being-seen rates, the system appeared to match capacity to demand effectively.

		The experiences of this system support tight coordination, communication, integration, and alignment of resources across multiple locations and provider types.

An integrated delivery system (IDS) is a network of physicians and hospitals that provides a continuum of healthcare services.

 Although integration is promoted primarily for general system improvement and accountable care,

 its utility may be particularly relevant to managing care delivery during a pandemic, a period when surge capacity is tested.

 The Agency for Healthcare Research and Quality defines surge capacity as “A healthcare system’s ability to rapidly expand beyond normal services to meet the increased demand for qualified personnel, medical care and public health in the event of large-scale public health emergencies or disasters.”

At the 2006 Academic Emergency Medicine Consensus Conference, a special breakout session titled “Science of Surge Capacity” was convened to define the essential components of surge capacity and outline key considerations when planning for large-scale public health emergencies.

Three essential elements were described in the resulting conceptual model: 

(equipment, pharmaceuticals, and supplies), and structure (both physical 

 and management infrastructure). In situations that exceed available human or physical resources, the latter element—an organization’s ability to effectively match resources to patient care needs&mdash;becomes paramount.

Like other institutions, particularly since 2001, The Children’s Hospital of Philadelphia (CHOP), an IDS dedicated to pediatric care, has focused attention on preparedness and surge capacity across its geographic network with particular concern for these 3 elements. CHOP’s experience with the influenza A (H1N1) pandemic of 2009 illustrates the benefits of an IDS in the face of a pandemic. Using the Academic Emergency Medicine Consensus Conference conceptual model of staff, stuff, and structure as a framework for our experience, we detail the planning and execution, operational experiences, and lessons learned. It is our hope that these insights will inform health system leadership, clinicians, and policy makers as they plan for future infectious and noninfectious surges of activity.

The Children’s Hospital of Philadelphia has centralized management of primary, specialty, and tertiary pediatric care facilities. For the fiscal year that included fall 2009, CHOP had a 459-bed main hospital, a 70-bed emergency department (ED), 28 primary care centers, and 9 specialty care facilities (

). Employing more than 9900 personnel, the CHOP network serves approximately 30% of the 1.6 million children in the Delaware Valley region via more than 1.1 million patient visits each year, including 85,690 ED visits and 28,106 hospitalizations. CHOP owns all hospital and practice locations, and employs all staff directly (except 670 faculty physicians employed by affiliated University of Pennsylvania practice plans). CHOP also operates the After Hours Program (AHP), a service designed to have specially trained nurses provide telephone triage, assessment, and advice using computerized, standardized protocols for a parent concerning their ill child. The AHP handles more than 100,000 calls per year.

In February 2009, prior to the influenza A (H1N1) outbreak in Mexico, administrative and clinical leadership from emergency medicine, general pediatrics, and primary care began routine planning for high wintertime nonurgent volume. Soon after this group convened, H1N1 emerged worldwide and spread in spring 2009, ultimately affecting enough children and adults to be classified as a pandemic.

 In Philadelphia, the first H1N1-infected patient was identified in April. From April to June, CHOP ED and inpatient areas saw higher volumes than typical for the season, and there was a spike in influenza A—positive respiratory viral tests (

). Based on guidance from CHOP and Centers for Disease Control and Prevention infectious disease experts, as well as on international experiences with H1N1, CHOP clinicians anticipated that a second wave of infection would emerge in the late summer or fall concordant with the return of children to school.

 In response, in May 2009, CHOP’s Strategic Planning office initiated planning activities to prepare CHOP’s multisite network for a second, even larger surge of H1N1-related activity. The office convened a special meeting of representatives from inpatient and outpatient clinical areas, Environmental Services, Facilities, the Patient Access and Revenue Cycle division, and Security. CHOP’s Emergency Preparedness team took a central role in plan development and coordination of effort across the institution. At their direction, key areas, including the main hospital, ED, pediatric intensive care unit (PICU), primary care centers, and AHP developed 3-tier (green-yellow- red zone) plans for handling successive levels of high volume (

). Such “traffic-light” graded response schemata are commonly used in clinical and operational care to summarize graded responses.

During the 3-week period from October 19 to November 10, 2009, the entire CHOP system experienced H1N1-related high volumes, with specific locations experiencing several days of double the typical volumes for visits or calls. This 3-week period (the surge) coincided with Centers for Disease Control and Prevention influenza surveillance reports of high levels of H1N1 activity in region 3, which includes Philadelphia.

 Patients from CHOP’s 4 urban practices are frequent users of the CHOP ED, often for nonurgent visits. As such, most ambulatory planning focused on these centers and the AHP phone triage system that services them (Figure 2). During the surge, these urban primary care centers operated in the yellow zone, with visit volumes exceeding prior peak winter volumes on a daily basis (

). Practices met demand by maximizing clinic work time, reducing administrative time, extending office hours, and replacing nonurgent follow-up visits and older child preventive care slots with acute/same-day visits. Additional strategies included expanding office hours through early opening, scheduling of patients during lunch hours, and extending weekend hours. Practices reassigned resident precepting duties, freeing attending physicians to meet real-time demands in the ED on an as-needed basis. Outpatient volume never officially triggered the red zone. Practices canceled nonurgent or preventive care visits sporadically at only 2 sites. The Children’s Hospital of Philadelphia’s suburban primary care practices added appointments as needed but did not utilize a formal 3-tier plan. Urban and suburban specialty clinics did contribute physicians to increase ED functional capacity, but did not have specific plans for increasing office capacity. Only CHOP clinicians provided services.

For the urban practices, numbers of daytime phone calls surpassed prior peak winter call numbers by 50% to 100% (Figure 3). Home access to the electronic medical record allowed part-time nursing and physician staff to respond to many nonurgent patient calls, AHP staff provided additional daytime coverage of incoming phone calls, and residents on primary care rotations were diverted to help answer calls when phone volume outpaced available respondents. Additionally, automated messages directed patients to online educational sites that addressed common influenza-related questions. Suburban practices had limited trunk lines and phone tree capabilities, and so did not have as much flexibility to support messaging, rerouting of calls, or daytime assistance by AHP staff. As a result, some suburban patients faced prolonged wait times and/ or had difficulty getting through to a CHOP provider.

The AHP operated in the yellow zone throughout the surge, handling 12,348 calls in October 2009 and 10,154 calls in November 2009 (vs 8128 in September 2009 and 8930 in December 2009). From October 18 to November 15, the AHP handled an average of 368 calls per day (SD 33.6), whereas in the corresponding 4 weeks before and after the surge, it handled a mean of 236 calls (SD 14; 

 <.001) per day, respectively. During the busiest week (October 25-31, 2009), AHP providers handled a mean of 415 calls per day. The AHP met the increased demand by creating on-hold messages to answer common influenza-related and non—influenza-related questions, streamlining call scripts, splitting and staggering shifts to improve staffing during high call volume times, shifting report writing to slower periods, and using physicians to handle some calls. After Hours Program staff members had access to primary care appointment slots within 24 hours.

In preparation for the pandemic, ED leadership outlined plans to open additional space, ensure adequate staffing, and improve work flow (Figure 2). To increase capacity for low-acuity patients (with and without suspected H1N1), a newly renovated 14-room ambulatory subspecialty clinic space near the ED was staffed on weekends and from 6 to 11 pm on weekdays by paid physicians from general pediatrics, primary care, adolescent medicine, and emergency medicine, as well as by paid ambulatory nurses. In addition, Department of Pediatrics attending physicians (regardless of specialty) were asked to volunteer to provide care to low-acuity patients during 6 newly created 6-hour ED shifts each day. Volunteer physicians signed up for these shifts via a webbased Google document,

 on average covering 5 to 6 extra 6hour shifts each day. These 2 additional options for nonurgent care increased ED peak bed count to 80, a 38% increase over regular peak bed capacity. During the surge, higheracuity ED patients were cared for in the ED’s 10bed extended care unit. In anticipation of staff illness, the ED and residency program created robust sickness relief programs to ensure adequate staffing.

During the 3-week surge period, ED patient visits rose 48% compared with the previous year, with 7793 children arriving for ED care (

). In the corresponding fall 2008 calendar period, 2597 fewer patients were seen and mean daily visits were only 225 compared with a mean of 339 during the surge (

 <.001). The 2009 surge exceeded all other previous high winter volume periods.

Of the patients seeking care during the surge, 603 (23%) were cared for in the evening and weekend subspecialty clinic space. Most of the increased ED volume was for infl uenzarelated illness or influenza-like symptoms, but few children who did not require hospitalization were tested for influenza (consistent with prior practice patterns at CHOP). Diagnostic testing using the multiplex-PCR assay was performed on the majority of patients requiring hospitalization. Rapid influenza diagnostic testing was not adopted due to poor sensitivity and specificity.

 The ED operated in its red zone for virtually the entire 3week surge period. Details of CHOP’s ED logistics during the pandemic are discussed at length by Scarfone et al.

The average time patients waited to be seen by an ED physician during the surge was less than that during the prior winter’s period of maximal infl uenza activity (

). Overall, 400 (5.1%) patients arriving for ED care during the surge left without being seen by a physician; none had been triaged as critical and only 16 had been triaged as acute (n =1761, yielding a leftwithoutbeingseen rate of 0.9%). Although that was twice the 2009 average left-without-being-seen rate, fewer patients left without being seen during the fall 2009 surge than during the high-respiratoryvolume month of February 2009 (eAppendix), when the leftwithoutbeingseen rate was 8%. For admitted patients and those discharged to home, the total ED length of stay was either unaffected or comparable to that of prior high-volume periods. The ED did not go on divert status.

 During the surge, ED admission rates were lower than average, but absolute numbers of admissions and inpatient occupancy were high. The OctoberNovember 2008 average admission rate was 18.0% (SD 2.9%, daily range 12.8%25.3%), reflecting the typical monthly rate of 18% to 21%. In comparison, the average admission rate during the 2009 surge was 14.7% (SD 2.1%, daily range 11%20%; 

The daily peak inpatient census during the 3-week surge averaged 423 (SD 22.1). That was 12% higher than the census during the same calendar period in fall 2008 (mean daily peak census 374 [SD 22]; 

 <.001), but only 2.9% higher than the census during February 2009 (mean daily peak census 411 [SD 23.2];

 = .06). During the surge, the main hospital was more than 90% occupied at peak daily census during 83% of weekdays and more than 95% occupied during 33% of weekdays. This additional patient volume was met by the creation and as needed deployment of an additional general pediatrics team; this team was supervised by a volunteer attending physician from the Department of Pediatrics. Pediatric residents on elective rotations were added to these teams as needed. The yellow zone plan for an additional general pediatrics team was needed for 5 days, and the red zone plan was not triggered.

Peak daily occupancy in the 46-bed main hospital PICU exceeded 90% on virtually all days of the surge (94%) and more than 100% occupancy on 15% of days (PICU treatment rooms were used for patients). The yellow zone plan for the PICU, which required increased staffi ng by nurses, fellows, and nurse practitioners, was active from October 21 to November 22, 12 days beyond the 3week networkwide surge period. Additional nursing hours (mean of 80 hours per day, representing a 10%15% increase over baseline nursing hours) for the ED and PICU were created through overtime, canceling educational activities, placing orientees in supervised clinical roles, and reassigning nurses from lowercensus units or nonclinical roles. Census levels alone did not trigger the red zone. Acuity was high enough, however, that PICU leadership decided to increase overnight inhouse critical care attending coverage from 1 to 2 during this period.

Death is a rare outcome in pediatrics. During the surge, there was 1 death recorded for a PICU patient with a primary diagnosis of infl uenza. This patient also had developmental delay and chronic medical conditions. There were 9 other deaths in the hospital during the surge, but infl uenza was not a contributing factor to these deaths.

Supplies and Equipment for Primary Care, Emergency Department, and Inpatient Areas.

 Ensuring adequate supplies to provide patient care was identifi ed early in planning as a critical challenge. During the planning phase, the emergency preparedness director worked with the director of resource management to secure gowns, gloves, and masks. Factors such as personal protective equipment (PPE) use based on clinical role, work setting, and number of patient encounters per shift, along with expected patient volumes, were used to estimate total PPE needs. In addition, Emergency Preparedness staff projected demand for chairs, screens, and supply carts for the auxiliary waiting area. The hospital procured PPE supplies through regular contact with distributors and weekly status reports. Distributor commitment to shipping PPE within 8 to 24 hours allowed stockpiling of supplies at a local distributor site. Ultimately, PPE use was 300% to 550% above normal. Although CHOP did not exhaust key supplies, estimates for needs were too low.

 After review of the literature and in consultation with local and national experts, CHOP followed Society for Healthcare Epidemiology of America, Infectious Diseases Society of America, and Association for Professionals in Infection Control and Epidemiology recommendations on transmission-based precautions for H1N1. The network adopted contact and droplet precautions for all patients with influenza-like illness and airborne isolation precautions for patients with known or suspected influenza who were undergoing aerosolgenerating procedures (eg, bronchoscopy, endotracheal intubation and extubation, open suctioning of airways). This strategy offered cost savings, as well as avoidance of supply shortages and unnecessary fit-testing of thousands of healthcare workers for N95 respirators.

To limit H1N1 transmission between patients, several measures were implemented. In the primary care setting, selfscreening stations were placed at the entrance of each clinic. During yellow zone periods, patients were asked to don a mask and perform hand hygiene if they had both fever and respiratory symptoms. In collaboration with Medicaid managed care insurers, CHOP created and sent letters to families describing vaccination and influenza clinics. When the H1N1 vaccine became available in November 2009, CHOP primary care practices used vaccine stations to administer vaccine to patients with scheduled appointments and highrisk patients. Approximately 33% of CHOP’s primary care network patients received the H1N1 vaccine during the winter of 2009 to 2010. Distribution was limited due to vaccine availability during the peak of the local outbreak; supplies increased only after regional disease decreased.

As vaccine availability was limited, Infection Prevention & Control, Occupational Health, and ambulatory specialties defined “tier 1” staff for H1N1 vaccination using 2 criteria: vulnerability of patients and repeated staff exposure to potential infections. Targeted tier 1 staff included 1307 staff working in the ED, neonatal intensive care units, the PICU, the PICU step-down unit, the cardiac intensive care and cardiac care unit, oncology (inpatient and clinic), the special immunology clinic, and the specialized perinatal unit. When the H1N1 vaccine arrived at CHOP during the last week of October, there was only enough supply to vaccinate approximately half of all direct care providers. Numerous staff saw themselves as direct care providers and wanted vaccine, even if they were not tier 1 staff. Prior to 2009, the Emergency Preparedness Working Group asked the CHOP Ethics Committee to develop a conceptual framework for allocation of scarce resources such as vaccine during a pandemic. After 2 formal presentations and additional informal discussions, Ethics Committee guidance was incorporated into plans for the 2009 H1N1 pandemic. When the initial vaccine supply became available, 1229 of the 1307 tier 1 employees (94%) were vaccinated. Ultimately, a total of 4506 healthcare workers were immunized.

MANAGERIAL

Experience of a pediatric integrated delivery system with the surge from the 2009 H1N1 pandemic is described, emphasizing scale, scope, and flexibility at multiple locations.

Pediatric Integrated Delivery System's Experience With Pandemic Influenza A (H1N1)

To assess the differentiated effects of population-based disease management programs (DMPs) for type 2 diabetes on intermediary clinical outcomes in The Netherlands.

Data covering a period from 20 to 24 months between January 2008 and December 2010 were collected from 18 Dutch care groups (primary care provider networks that have bundled payment contracts for delivery of diabetes DMPs). Meta-analysis and meta-regression methods were used to conduct differentiated analyses of these programs’ effects over time on 4 clinical indicators: glycated hemoglobin, lowdensity lipoprotein, systolic blood pressure, and body mass index. Heterogeneous average results were stratified according to various patient and process characteristics to investigate whether differences in these features could explain variation in outcomes.

Between 56% and 71% of patients (N = 105,056) had valid first- and second-year measurements of the study outcomes. Although average changes in these measures over time were small, stratified analyses demonstrated that clinically relevant improvements were achieved in patients with poor first-year health values. Interactions with age, disease duration, comorbidity, and smoking status were not consistent across outcomes; nonetheless, heterogeneity in results decreased considerably when simultaneously correcting for known patient characteristics. Positive effects tended to diminish with longer length of follow-up, while greater measurement frequency was associated with improved results, especially in patients with poor health.

Our data suggest that tailored disease management, in which not only evidencebased guidelines but also patient characteristics directly determine care processes, including self-management support, has great potential to improve the cost-effectiveness of current chronic care delivery.

Correcting for heterogeneous care processes and patients resulted in differentiated insight into the effects of population-based disease management on intermediary outcomes of diabetes care.

		Patients with poorly controlled diabetes benefit more from standardized disease management than patients with better health, for whom self-management support might be equally effective.

		Positive effects of disease management diminish with longer length of follow-up; greater measurement frequency improves results, especially in patients with poor health.

		Tailored disease management, in which care content and dose are determined by patients’ health, demographics, and social characteristics, has great potential for improving cost-effectiveness in chronic care.

Rising costs, an aging population, and the recognition of severe deficiencies in traditional care have motivated the development of innovative models for chronic illness management. Examples include shared care, case management, and stepped care, but perhaps most well known internationally are disease management and the Chronic Care Model.

 Differences aside, these concepts have in common their basic assumption that better treatment today will result in better health and less expensive care in the future. Specific quality improvement efforts tend to focus on (1) reducing fragmentation between providers and settings, (2) stimulating evidence-based practice, (3) promoting active and planned follow-up, and (4) supporting patients’ self-management.

Developed in the United States, disease management programs (DMPs) have quickly spread to other countries such as Canada, the United Kingdom, and Germany, which now has some of the largest DMPs in the world.

 In the primary care—oriented Dutch health system, chronic disease management initiatives developed from the 1990s onward. Initially, uptake of these initiatives remained limited, mainly due to lack of a structured framework and fragmentary funding.

 The 2006 health insurance reform granted health insurers extended power to negotiate with care providers and, in so doing, facilitated the development of a more integrated funding method, the so-called “bundled payment system.”

 Under this system, insurers pay care groups, which are provider networks based in primary care, a single fee for the full range of outpatient services for a specific chronic condition.

 Bundled payments are seen as a way to stimulate primary care providers, predominantly general practitioners (GPs), to engage in multidisciplinary cooperation and deliver integrated,  evidence-based disease management in an ambulatory setting, therefore limiting the need for specialist care.

Although the evidence base for their impact on the quality and outcomes of care is limited, integrally financed DMPs for type 2 diabetes quickly achieved national coverage in The Netherlands. In March 2010, more than 100 groups representing approximately 80% of GPs had a bundled payment contract for diabetes care.

 In the same year, the payment system was expanded to cover the management of chronic obstructive pulmonary disease and vascular risks. Hence, the support for disease management in The Netherlands appears to be a matter more of faith than fact, as is the case in other countries as well. International studies and reviews of DMPs demonstrate highly heterogeneous results and so far have failed to answer the basic question of what works best for whom, not least because of variation in both methodology and nomenclature.

 It is, however, especially the multicomponent and populationbased nature of disease management that makes it difficult to draw unambiguous conclusions concerning effectiveness. Analyzing complex DMPs necessitates a clear framework that links expected outcomes to the characteristics of both the program and its target population, and measures effects over an adequate period of time.

As part of the DISMEVAL (Developing and Validating Disease Management Evaluation Methods for European Healthcare Systems) project, this study’s objective was to assess the effects of the Dutch diabetes DMPs on a range of intermediary clinical outcomes. To investigate heterogeneity in results across different care processes and patients, we designed the study as a population-based, multilevel meta-analysis and meta-regression. Given that experimental comparisons were not possible because of the nationwide rollout of the DMPs and the unsuitability of using historical controls,

 these methods allow for the most in-depth assessment of effectiveness. Such differentiated insight, which goes beyond the “grand means” that currently inform many health system redesigns, can support professionals and policy makers in their efforts to better meet the complex care needs of the growing and inherently diverse population of chronically ill patients.

The bundled payment system for generic diabetes disease management in The Netherlands obliges care groups to provide insurers with a specific number of performance indicators for both processes and outcomes on an annual basis.

 We retrospectively gathered individual patient data on these indicators from a convenience sample of 18 groups, which were set up between 2006 and 2009 and represent nearly all regions of The Netherlands. Nine groups were part of an experimental pilot concerning bundled payments evaluated by the National Institute for Public Health and the Environment

; to include 9 other, nonexperimental groups, 14 care groups were approached (response rate 64.3%).

Across the included groups, 106,623 patients had at least 1 registered visit during the research period, which—depending on the availability of data&mdash;was either 20 or 24 months between January 2008 and December 2010. We excluded type 1 diabetes patients (n = 1567) because they are treated primarily by specialists. Since patient data were drawn from groups’ clinical information systems, plausibility was verified through range checks. Outliers were removed based on cutoff points determined by Dutch diabetes experts (see

Because patient data were not available for the period before introduction of the bundled payment system, we assessed the effects of the diabetes DMPs by comparing the last measurement of each clinical outcome during the first year of the research period (or the first 8 months for the 2 groups with a 20-month research period) with the last measurement of that outcome in the second year. Per outcome-specific analysis, we excluded patients who (1) lacked registrations of the first- or second-year measurements, or both; (2) missed registrations of 1 or more of the patient and/or process characteristics used for stratification; and/or (3) had an observation period between first- and second-year outcome measurement of fewer than 3 months. The maximum length of follow-up per patient was 23 months. The study flow chart is shown in 

Diabetes Disease Management Programs in The Netherlands

In 2007, 10 Dutch care groups started experimenting with a bundled payment system that allows the different components of care for generic type 2 diabetes to be purchased, delivered, and billed as a single product or service. Care groups are legal entities in primary care, which consist of multiple care providers, are most commonly owned by GPs, and form the principal contracting partners for health insurers involved in bundled payment contracts.

 Groups either deliver the various components of a diabetes DMP themselves or subcontract other providers, including GPs, physical therapists, dieticians, laboratories, and/or specialists, to do so. The price of a specific program (ie, care bundle) as well as the fees for individual subcontractors are freely negotiable between care groups on the one hand and health insurers or care professionals on the other. Patients are automatically enrolled by their GP.

The services to be covered in the diabetes DMPs are codified by a national care standard for generic diabetes care.

This standard is based on existing evidence-based guidelines for GPs and includes general modules (eg, information, education and self-management support, smoking cessation, physical activity, nutrition and diet) as well as disease-specific modules.

The latter comprise a defined frequency of GP visits, regular foot and eye examinations, and laboratory testing. To stimulate task redistribution from GPs to practice nurses, who traditionally play an important role in Dutch chronic care, the standard describes care in terms of functions rather than providers, defining what services must be delivered rather than by whom or where.

 For more information on the contents of the Dutch diabetes DMPs, the diabetes care standard can be accessed online: http://www.diabetesfederatie.nl/zorg/zorgstandaard.html.

Definition of Variables and Data Validation

We defined intermediary clinical outcomes as mean changes in glycated hemoglobin (A1C), low-density lipoprotein (LDL), systolic blood pressure (SBP), and body mass index (BMI) between the first- and second-year measurements. In addition, we assessed and compared the proportions of patients within 3 target range groups for glycemic control (A1C 

75 mmol/mol) at these 2 measurement points.

Care processes were described in terms of measurement frequency and range, and duration of care. We codified measurement frequency as the number of registrations of each clinical outcome during follow-up; to describe measurement range, we assessed the number of different outcomes registered, which across care groups could be a maximum of 8 (ie, A1C, total cholesterol, LDL and high-density lipoprotein, SBP and diastolic blood pressure, BMI, and triglycerides). Duration of care was defined as the number of months between the first- and second-year measurements of each clinical outcome.

To describe patients, we used age in years, disease duration in years, health status, comorbidity, and smoking status. Health status was determined by the first-year values of each clinical outcome. Comorbidity was defined as the presence of 1 or more of the 4 most frequently registered co-occurring conditions across the included care groups (ie, angina pectoris, myocardial infarction, stroke, transient ischemic attack). We dichotomized smoking status as previous smoker or nonsmoker versus current smoker.

We used a 2-step approach to population-based, multilevel meta-analysis, in which we clustered the individual patient data by care group.

 During the first step, we conducted paired-sampled

 tests (2 sided; α = .05) using SPSS version18 (IBM Corporation, Armonk, New York) to calculate the group-specific mean differences in clinical values and associated standard deviations. In the second step, these were synthesized with Review Manager (RevMan version 5.1.1; The Cochrane Collaboration) into pooled mean differences and 95% confidence intervals, for which—owing to significant heterogeneity in results&mdash;we used the random-effects metaanalysis model of DerSimonian and Laird.

 Using this model, we weighted the aggregate effects by the inverse of their variances while assuming random treatment effects across care groups.

 To quantify heterogeneity, we calculated the I2 statistic—which can range from 0% to 100%&mdash;on the basis of the χ2 test.

 >75%) heterogeneity, subgroup analyses were conducted to examine the consistency of results across diabetes care processes and patients.

 For this purpose, most continuous variables were categorized into 2 or 3 groups on the basis of either scientific literature (age

) or median values (measurement frequency and length of followup). We dichotomized measurement range as 8 outcomes (the maximum number of clinical indicators that could be registered) versus fewer than 8 outcomes; first-year health status was categorized according to the target range values for clinical outcomes included in the diabetes care standard.

We used multilevel meta-regression analysis of individual patient data, stratified by care group, to further assess the influence of potential effect modifiers as well as to investigate potential 3-way interactions between care processes, patient characteristics, and clinical outcomes.

 To conduct the meta-regressions, we used the PROC MIXED command in the SAS 9.2 software (SAS Institute Inc, Cary, North Carolina), which uses a random-effects iterative method to provide a maximum likelihood estimate of the regression parameters. The meta-regression models were multivariable, though process and patient characteristics were included separately; continuous covariates were included as such. For each clinical outcome, we calculated the intraclass correlation coefficient to assess the percentage of total heterogeneity in effects occurring between care groups. The larger the intraclass correlation coefficient, the greater the proportion of variance that can be attributed to differences between rather than within groups.

 We examined collinearity with the variance inflation factor: a variance inflation factor value of greater than 10 is generally taken as an indication of serious multi-collinearity.

 Explained heterogeneity was expressed as the percent change in between-group variance (τ

With regard to care processes, patients’ SBP was assessed most frequently (n = 4) during the period between the firstand second-year measurements, followed by BMI (n = 3) and A1C (n = 2). LDL was measured least often (n = 1). Across care groups, the average share of patients with the maximum outcome measurement range varied from 44.4% to 86.7%, with a mean of 62.3%. Median length of follow-up varied from 10 to 12 months between groups.

shows the patients’ main characteristics and firstyear clinical values. Age ranged from 15 to 105 years with a mean of 65.7 (± 11.9) years. Diabetes duration varied from 0 to 76 years with a mean of 4.8 (± 5.6) years. Approximately 16% of patients had a known comorbidity; 18.4% were registered as current smokers. Depending on the clinical outcome of interest, roughly 56% to 71% of patients (n = 105,056) had both a first- and second-year measurement. For most outcomes, the mean first-year values for patients were within normal range. Mean first-year BMI (29.7 ± 5.2 kg/m2), however, signified severe overweight to borderline obesity.

shows the overall results of the random-effects meta-analysis across 18 care groups, presented per clinical outcome. Mean A1C increased by 0.17 mmol/mol (95% confidence interval [CI] —0.60 to 0.93) between the firstand second-year measurements. LDL was reduced significantly by a mean of 0.09 mmol/L (95% CI −0.13 to −0.05); SBP decreased significantly by an average of 0.95 mm Hg (95% CI −1.25 to −0.64). There was a small average decrease in BMI of 0.04 kg/m

 (95% CI −0.10 to 0.02). Except for BMI, the effects of the diabetes DMPs on all intermediary outcomes were moderately to highly heterogeneous, with I

 values ranging from 57% for SBP to 98% for A1C.

 shows the proportions of patients within 3 target range groups for glycemic control at the first- and second-year measurements. Glycemic control improved in the vast majority of patients, especially in the group with highly uncontrolled diabetes (A1C 

75 mmol/mol). Diabetes was brought under control in roughly 68% of these patients; in 19%, the values improved to 53 mmol/mol or lower. Of the patients with first-year A1C concentrations between 53 and 75 mmol/mol, approximately 36% improved their glycemic control, 60% remained within the same range, and 4% deteriorated. Of those within target range at the first measurement (A1C 

 53 mmol/mol), 83% maintained their A1C control, whereas A1C control deteriorated up to levels of <75 mmol/mol in 16.5%. Roughly 0.5% saw their A1C increase to 

75 mmol/mol. In terms of mean age and disease duration, the patients within target range at the first-year measurement were most comparable to the overall patient population. Those in the subgroups with initial A1C values above 53 mmol/mol and 75 mmol/mol were significantly younger but had a longer-thanaverage disease duration.

The heterogeneous nature of care processes and patients should be taken into account in both the design and evaluation of disease management programs for diabetes.

Toward Tailored Disease Management for Type 2 Diabetes

To evaluate the effects of a multifaceted quality improvement intervention during 2 time periods on 4 coronary artery disease [CAD] measures in 4 primary care practices. During the first phase, electronic reminders prompted physicians to order indicated medications or record contraindications and refusals (exceptions). In the second phase, physicians also received reports about their performance (including lists of patients not satisfying these measures), and financial incentives were announced.

Adult CAD patients seen within the preceding 18 months were included. The primary outcome was the performance on each measure (proportion of eligible patients satisfying each measure after removing those with exceptions). Secondary outcomes were the proportion with the medication on their medication list, and the proportion with exceptions.

Median performance at baseline was 78.8% for antiplatelet treatment, 85.1% for statin treatment, 77.0% for beta-blocker after myocardial infarction (MI), and 67.1% for angiotensinconverting enzyme inhibitor or angiotensin receptor blocker after MI. Performance improved slightly for 3 measures during the first phase and improved more substantially for all 4 measures during the second phase. For 3 of 4 measures, however, documentation of exceptions increased but not medication prescribing. Most exceptions were judged to be appropriate by peer review.

Physicians responded more to the combination of feedback and financial incentives than they had to electronic reminders alone. High performance was only achieved for 1 of 4 measures and recording of exceptions rather than increases in medication prescribing accounted for most of the observed improvements.

We examined the sequential implementation of commonly used quality improvement techniques to improve outpatient coronary heart disease care in 4 practices.

Physician response to electronic reminders alone to address outpatient coronary heart disease measures was small.

Physicians responded more when the combination of feedback reports and financial incentives was added to reminders.

For 3 of 4 measures, documentation of exceptions increased, but not medication prescribing.

Most physician-recorded exceptions to quality measures were judged to be appropriate by peer review.

Quality improvement techniques that leverage an electronic health record (EHR) have been shown to improve care in many cases.

However, EHR-based quality improvement has not been universally successful, and even in many instances where study results were positive, the magnitude of the improvement was small.

Furthermore, observational data do not suggest that simply having an EHR improves quality in outpatient settings.

 In contrast, we have shown in the UPQUAL study (Utilizing Precision Performance Measurement for Focused Quality Improvement) that interconnected EHR-based tools can improve quality for multiple process of care measures in a large urban, single-site, university-affiliated practice.

 This intervention was designed to improve quality measurement (including capture of contraindications and patient refusals), make point-of-care reminders more accurate, and provide more valid and responsive feedback to clinicians (including lists of patients not receiving essential medications).

In this current study, we applied these principles—improve quality measurement in order to enable more accurate point-of-care reminders and feedback&mdash;to coronary artery disease (CAD) care in 4 suburban primary care group practices (2 family medicine and 2 internal medicine) that belong to the same health system and use the same EHR. We selected CAD care as our study objective because it is a common and important chronic disease and because implementing CAD measures in this setting was more feasible than several other candidate chronic disease and prevention topics. In this health system, point-of-care reminders were implemented first in July 2008 (Phase 1). Starting September to November 2009 (Phase 2), feedback was given to physicians and the medical group publicized to physicians that financial incentives would be tied to performance measures (including the 4 measures studied here). Both the reminders and physician feedback portions of the interventions were planned prior to Phase 1 by the study team. The financial incentives were initiated independently by leadership in the organization that was not directly associated with this study. This sequential implementation provides an opportunity to observe the additional effects of adding the combination of feedback and announcing financial incentives to electronic reminders on measured performance.

We performed this study at 4 primary care practices in the northern suburbs of Chicago, Illinois, that use the same commercial EHR (EpicCare, Epic Systems Corporation, Verona, Wisconsin). Northwestern University’s and Northshore University HealthSystem’s institutional review boards approved the study. All patients eligible for 1 or more quality measures cared for by 33 attending physicians (10 family medicine and 23 internal medicine) and 15 family medicine resident physicians were included. The practices had used the EHR for 5 years before the start of the first intervention examined in the study.

Sequential Implementation of Quality Improvement Techniques

We selected for consideration 4 measures of CAD care quality that were based on national measures: antiplatelet drug and lipid-lowering drug treatment in all patients with CAD, beta-blocker use in patients with prior myocardial infarction (MI), and angiotensin-converting enzyme (ACE) inhibitor or angiotensin receptor blocker (ARB) for diabetes or left ventricular systolic dysfunction.

 Health system clinicians, including cardiologists, discussed and modified the measures for local use, changing the lipid-lowering drug measure to statin treatment in CAD and changing the ACE inhibitor or ARB measure to apply to patients with prior MI only.

Electronic Clinical Decision Support (Phase 1)

Prior to these interventions, there were no other clinical decision support tools in use that addressed these topics. We added electronic point-of-care reminders that appeared during patient encounters when an apparently eligible patient did not have an indicated medication on their current medication list and had no exception recorded. These alerts were minimally intrusive (they did not interrupt clinicians’ work flow and the alert was indicated only by a single yellow highlighted tab that appeared on the left side of the screen when any clinical reminder criteria were present, and physicians had to select this tab to see the individual reminders). These alerts were displayed using existing EpicCare functionality. These electronic reminders included standardized ways to capture patient reasons (eg, refusals) or medical reasons that were exceptions for individual reminders within the reminder system of the EHR. These reminders were implemented in July 2008. We sent physicians educational e-mails with brief training materials to introduce the new alerts and to show how to record patient or medical exceptions.

Implementation of Feedback Reports and Announcement of Incentives (Phase 2)

Starting in September 2009, on a monthly basis, we gave physicians printed reports indicating their overall performance on each of the 4 measures for all their eligible patients and lists of individual patients who appeared to be eligible for an indicated medication but were not receiving it and had no exception recorded.

In October and November of 2009, the medical group leadership announced to physicians that a small portion of their compensation (1.5% of total compensation, which constituted 25% of the incentive-based compensation) would be tied to their performance on quality metrics, including the 4 metrics covered in this study.

We retrospectively calculated the performance for the 4 CAD measures for each month from September 2007 through March 2011. At each time point, patients were eligible for a measure if they had an office visit with a physician from 1 of the 4 included practices during the preceding 18 months and had a qualifying

 International Classification of Diseases, Ninth Revision, Clinical Modification 

code used on their active problem list, past medical history, or as an encounter diagnosis. We used Structured Query Language to retrieve data from an enterprise data warehouse that contains data copied daily from the EHR. For each time point, all patients were classified for each measure for which they were eligible as: a) satisfied, b) did not satisfy but had an exception, or c) did not satisfy and had no documented exception. The primary outcome for each measure was calculated as the number of patients who satisfied the measure divided by the total number of eligible patients excluding those with an exception. As an equation, the primary outcome = number satisfied / [number eligible — number not satisfied with an exception]. We also analyzed separately for each measure the proportion of eligible patients who satisfied the measure (were given the medication) and the proportion of all eligible patients who did not satisfy the measure and had exceptions.

We performed peer review beginning in September 2009 of medical exceptions recorded in the EHR and continued the review process for exceptions entered within the first 10 months of the intervention. One physician reviewed medical records to collect the reason for the exception and additional clinical information needed to judge the validity of the exception. When the clinical reasoning was unclear, the peer reviewer would request clarification from the treating clinician. Two board-certified internists and 1 board-certified family medicine physician met regularly to review the exceptions and judged them as appropriate, inappropriate (including when no contraindication to the medication was evident on physician chart review or cases where clarifying information was requested from the primary care physician but was not provided), or of uncertain appropriateness by consensus. When a consensus was not reached or the appropriateness was uncertain, 1 physician reviewed the medical literature, requested advice from specialists when needed, and the group discussed the case again until consensus was reached. Practice physicians received e-mail or telephone feedback for medical exceptions that were judged to be inappropriate.

We used interrupted time series analysis to examine changes in the primary and secondary outcomes over time for 2 different interventions: before and after July 2008 (the transition between baseline and Phase 1), and before and after September 2009 (the transition between Phase 1 and Phase 2). We calculated the primary and secondary outcomes for each of the performance measures for each month from September 2007 through August 2010. A linear model was fit to each series including a continuous time variable, a dichotomous indicator of the intervention, and the interaction term of time and intervention as covariates. The individual data points used for each time period are depicted in the figures. Next, we determined the autoregressive order of the model residuals by minimizing Akaike’s information criterion.

 Finally, we fit a linear regression model with autoregressive errors (using the appropriate number of autoregressive parameters, if any were necessary) to each series. These fitted models were used to test statistical significance.

 To ensure model validity, we examined several residual diagnostics, the Jarque-Bera and the Shapiro-Wilk tests for normality of residuals, and normal Q-Q and autocorrelation plots.

 Analyses used SAS version 9.2 (SAS Institute Inc, Cary, North Carolina) and R software package version 2.13.1 (R Foundation for Statistical Computing, Vienna, Austria).

The number of patients eligible for the coronary disease measures and their characteristics are provided in 

. The number of eligible patients increased over time from 779 CAD and 218 MI patients in October 2007 to 1099 CAD and 332 MI patients by October 2010. Their characteristics changed little during the 3 years we examined (Table 1).

Median performance during the baseline period was 78.9% for antiplatelet treatment, 85.3% for statin treatment, 77.0% for beta-blocker after MI, and 67.2% for ACE inhibitor or ARB after MI (

). Performance on the antiplatelet measure was increasing significantly during the baseline period. Performance on the other 3 measures did not change significantly during the baseline period (

During Phase 1, overall performance continued to increase for the antiplatelet measure at a rate that was similar to the rate of improvement observed during the baseline period. There were statistically significant, but very small, increases in overall performance for the statin and ACE inhibitor/ARB measures (Table 2 and Figure 1). However, there were no significant increases in the rates of patients given medication for these 2 measures, and there was a decrease in patients given beta-blockers during Phase 1 compared with baseline (Table 2 and Figure 2). Physicians recorded exceptions to these measures during Phase 1 for small percentages of eligible patients (

There were significant increases in measured performance for all 4 measures during Phase 2 compared with Phase 1 (Table 2 and Figure 1). For 3 of the 4 measures, there was an increase in the documentation of exceptions (Table 2 and Figure 3) but not medication prescribing (Figure 2). For the antiplatelet measure there was a significant increase in both medication prescribing and exception documentation (Table 2, Figures 2 and 3).

By March 2011, overall performance was 99.7%, 91.1%, 86.8%, and 78.9% for antiplatelet treatment, statin treatment, beta-blocker after MI, and ACE inhibitor or ARB after MI, respectively.

We performed peer review for 179 medical exceptions. For 13 of these, a physician recorded an exception even though the patient was given the medication (the physician both recorded an exception and ordered the medication). Exceptions of this kind would not be included in the performance calculation, since the numerator criteria were met. Of the remaining 166 exceptions, 145 (87.3%) were judged appropriate on peer review,19 (11.4%) had an inappropriate reason or had no reason found for not prescribing on peer review, and 2 (1.2%) were of uncertain appropriateness (

We used time series analysis to examine the effects of implementing electronic clinician reminders directed at 4 measures of outpatient coronary artery disease care followed later by physician audit and feedback, lists of individual patients not meeting the measures, and the announcement of financial incentives. We made several observations that warrant discussion.

Introducing electronic reminders alone during Phase 1 had very little impact on measured quality. Physicians did interact with the clinical decision support (CDS) system to some degree, as evidenced by recording of exceptions using the CDS system. However, the extent of this usage was modest, and led to very small changes in measured quality. These findings are consistent with older studies of CDS aimed at changing outpatient provider prescribing behavior for coronary artery disease that found small effects, or no effect, from electronic clinician reminders.

After the start of individualized feedback to physicians with lists of patients not satisfying the measures and the announcement that there would be financial incentives based on performance (Phase 2), there were much more prominent changes in performance. During this second intervention period, there was a sharp increase in physicians’ use of exception recording for all 4 measures as well as an increase in patients with antiplatelet medication recorded on their medication lists. As a result of these changes, there were significant increases in measured performance for all 4 measures. This observed change suggests that linking CDS tools to local accountability systems like incentives or providing feedback on performance can lead to greater physician engagement with EHR quality improvement tools than would occur with CDS alone. Prior studies have shown a positive effect of providing audit and feedback to physicians,

We examined the impact of electronic reminders followed by performance reports and financial incentives. Physicians responded more to reports and incentives than to reminders alone.

October 2012

Medicare Part D Claims Rejections for Nursing Home Residents, 2006 to 2010

Pediatric Integrated Delivery System's Experience With Pandemic Influenza A (H1N1)

Toward Tailored Disease Management for Type 2 Diabetes

Implementation of EHR-Based Strategies to Improve Outpatient CAD Care

Change to FIT Increased CRC Screening Rates: Evaluation of a US Screening Outreach Program

Identification of and Intervention to Address Therapeutic Gaps in Care

EMR-Based Medication Adherence Metric Markedly Enhances Identification of Nonadherent Patients

Financial Incentives and Physician Commitment to Guideline-Recommended Hypertension Management

Identifying Frail Older People Using Predictive Modeling

Application of New Method for Evaluating Performance of Fracture Risk Tool