Nikki M. Carroll, MS

: To report outcomes of a pilot online curriculum educating clinicians about off-label prescribing, finding unbiased sources of pharmaceutical information, and responding to patient inquiries about direct-to-consumer advertised medications.

: We developed 3 case-oriented Internet modules to educate healthcare professionals about off-label prescribing, accessing and appraising unbiased drug information, and talking with patients about direct-to-consumer advertising. Modules included reflective questions and opportunities to asynchronously ask questions of faculty. Pilot testing occurred among 52 Kaiser Permanente Colorado prescribing clinicians from May 2007 through September 2007. We assessed self-reported changes in knowledge, intent-to-change practice, and barriers to practice change. We also examined curriculum participant pre—post changes in prescribing of selected highly advertised medications.

: Most participants reported changes in knowledge after curriculum completion, many made intent-to-change practice statements at curriculum completion, and several reported changes in practice 3 to 6 months after curriculum completion. Six categories of barriers to practice change were noted. Prescribing of targeted medications rose by 0.8 prescription per month in the intervention group and by 6.3 prescriptions per month in the control group, but this difference was not significant (

: Targeted curricula on aspects of the pharmaceutical development, approval, and marketing process can lead to subjective changes in clinician knowledge and can facilitate intent-to- change practice. Larger studies are needed to assess the effect of such interventions on prescribing patterns.

		Interactive reflective online educational curricula can help educate and prompt clinicians to make practice changes in prescribing heavily marketed medications.

		Clinicians may not avail themselves of online curricula because of competing demands, multiple educational opportunities, and lack of recognition of the effects of pharmaceutical advertising.

		Curricula on off-label prescribing, accessing and appraising pharmaceutical information, and talking with patients about direct-to-consumer advertising should not be used in isolation but may be useful in multifaceted efforts toward more cost-effective prescribing.

	Pharmaceutical spending represents an important and large part of US healthcare expenditures.

 As costs of healthcare continue to outpace inflation,

 the amount of money spent on pharmaceutical advertising

 and its influence on patients and clinicians

 have come under increasing attention, especially in cases when newer heavily advertised medications offer marginal, if any, therapeutic advantage over more established agents.

	In 2004, Pfizer, Inc (New York, NY) entered into a $430-million settlement with a consortium of state attorneys general relating to off-label marketing of Neurontin (gabapentin).

 The consortium allocated some settlement funds to a grant for developing programs designed to improve the cost-effectiveness of prescribing by (1) educating health professionals about pharmaceutical development and approval processes, (2) increasing clinician awareness of and ability to evaluate pharmaceutical industry marketing techniques, and (3) providing strategies for accessing and evaluating drug information.

	Kaiser Permanente Colorado (KPCO) cares for 440,000 members in the Denver-Boulder, metropolitan area via a group model health maintenance organization (HMO) and another 40,000 members in the Colorado Springs, area via a preferred provider organization (PPO) model. Kaiser Permanente Colorado was 1 of 24 initial grantees to receive educational curricular development grant funding under the Neurontin settlement. Using the results of a needs assessment that we conducted among HMO and PPO prescribing clinicians and KPCO pharmacy and other leaders,

 we focused our curriculum on the following 3 topics: (1) off-label prescribing, (2) accessing and appraising unbiased drug information, and (3) talking with patients about direct-to-consumer advertising. This case study describes our curriculum and the results from the pilot phase of its deployment in KPCO.

	We developed 3 Internet-based modules (1 for each topic) that were accredited for a total of four (4) 

AMA PRA Category 1 Continuing Medical Education

 Credit(s)™. Modules consisted of online self-paced presentations (PowerPoint; Microsoft, Redmond, WA). The presentations included didactic information (eg, facts and figures on prescribing rates, results from physician and patient surveys, and screenshots of evidencebased pharmaceutical information Web sites) that was supported by case examples using selected, heavily marketed antibiotics, antidepressants, and diabetes medications (target areas of focus for KPCO identified from the needs assessment as areas in which potential participants desired to improve prescribing). Modules also contained multiple-choice questions and interspersed reflective questions asking participants about past experiences with the module topics and about their intent-to-change practice after reviewing the material (eg, “Recall the last time a patient asked you about a specific medication advertised directly to the public. Which medication was it? What did you end up prescribing at that visit?”). We also included a series of tips on topics (such as a checklist for appraising articles on pharmaceutical effectiveness) that were also made available as downloadable pdf files for future use. Participants could pose questions (asynchronously) to module faculty (DWP and MAR), who responded by e-mail within 72 hours. At the completion of each module, participants were asked to complete an evaluation form and were given the option to access the other modules. The curricula are publicly available from the Federation of State Medical Boards (http://www.fsmb.org/re/open/modules.html). Copies of each of the modules are available on request from the author.

	The evaluation at the end of each module asked participants to rate whether the module met each of its learning objectives (using a 4-point Likert-type scale) and to subjectively rate their improvement in knowledge (improved, somewhat improved, about the same, or less than before the module). To assess participant-perceived value of the modules and to assess the potential influence of the modules, we also asked participants to indicate what changes in practice they intended to make after module completion (open-ended question). Three to six months after completing the modules, participants were invited to complete a follow-up survey asking them to self-report the changes they had made and the barriers they experienced in making intended practice changes (open-ended questions).

	Approximately 1800 potential participants were contacted and recruited in two 3-month waves initially by e-mail, supplemented with hard-copy mail for e-mail nonrespondents. Eligible participants included Colorado Permanente Medical Group physicians and KPCO allied health professionals of all specialties practicing in the Denver-Boulder area (HMO cohort) and physicians and allied health professionals in the Colorado Springs area affiliated with the KPCO PPO network.

	Using administrative data from internal pharmacy databases, we also compared participant (intervention group) pre—post prescribing of brand-name medications from drug classes targeted in the curriculum (included based on the results of the needs assessment

) 6 months after the curriculum compared with 12 months before the curriculum by Wilcoxon signed rank test. Interpretation of pre—post analysis results is subject to bias associated with temporal trends that can be unassociated with the study intervention. Therefore, to assist in elucidating temporal prescribing trends, we conducted the same comparison and analysis for noncurriculum participants (control group) by examining overall organizational prescribing trends for these medications, excluding data from curriculum takers. Prescribing information was assessed only in the Denver-Boulder cohort, for whom these data were more readily accessible. Although the number of study subjects in the intervention group was small (n = 52), we analyzed the number of target medications by comparing intervention group and control group prescribing before and after the curriculum using a generalized estimating equation analysis with a Poisson distribution, an efficient method for modeling count data (number of prescriptions). The interaction of group and time (before and after the curriculum) was entered as the independent variable, with the number of prescriptions for targeted brand-name medications as the dependent variable. The group-×-time interaction term was used as the independent variable to minimize the possible temporal bias that may exist when looking at prescription counts over time. Using a comparison group in the analysis (rather than a pre–post design with only the intervention group) helps to control for this potential temporal bias.

	During the pilot study (May 2007 through September 2007), 52 of approximately 1800 eligible clinicians enrolled in the curriculum. Forty-seven clinicians (33 physicians and 14 allied health professionals) completed at least 1 module; 8 of 47 were from the PPO. Fourteen participants had been in practice 20 years or more and 12 participants for 5 years or less; 14 participants had practiced with KPCO for more than 10 years and 12 participants for less than 2 years. Sixteen physicians (8 each in family medicine and internal medicine) and 2 allied health professionals were primary care providers, 5 physicians and 1 nurse practitioner were obstetrics and gynecology specialists, and the remaining participants worked in psychiatry, general surgery, orthopedic surgery or trauma, ophthalmology, dermatology, endocrinology, palliative care, or hospital medicine. One dental surgeon participated; 6 allied health professionals did not indicate a specialty.

	Thirty participants completed all 3 modules, 4 participants completed 2 modules, and 13 participants completed 1 module. Forty-five participants completed the module on offlabel prescribing, 33 participants completed the module on accessing and appraising unbiased drug information, and 33 participants completed the module on talking with patients about direct-to-consumer advertising. Fourteen participants used the “ask the faculty” e-mail option.

	Forty-five participants indicated that the modules met their educational objectives. We also found some changes in clinician perception of off-label drug promotion. Initially, 10 clinicians reported positive viewpoints and openness to specific off-label use after exposure to off-label drug promotion, 9 clinicians were skeptical, and 4 clinicians were reluctant or refused to prescribe off label. After completing the module on off-label prescribing, 20 clinicians believed that, had they seen the module before off-label promotion of a specific medication, they likely would have had a different reaction to the promotion. Sixteen clinicians (10 of whom were skeptical of or refused the promotion initially) indicated that seeing the module first would not have changed their perception of off-label promotion.

 summarizes the assessment of change in knowledge among participants after completion of each module, the number of participants who made an intent-to-change practice statement after each module, and the total number of intent-to-change practice statements made at the end of each module. It also summarizes the number of clinicians making (by self-report) at least 1 practice change and the number of practice changes made on 3-month to 6-month follow-up. Representative self-reported changes in practice included the following statements: “I have more aggressively pursued proposed off-label medication indications and any underlying clinical studies to support those indications.” “I have also made the effort to hold my ground more frequently in the face of patient pressure. This has been difficult but ultimately, I think, rewarding.”

	Nineteen of 47 participants who completed at least 1 module responded to the follow-up survey. Six participants reported no barriers to making intended practice changes. A total of 22 barriers in 6 general categories (determined based on review and consensus categorization of the qualitative barrier comments among the authors) were identified (

	Forty-three of 47 participants who completed at least 1 module consented to an evaluation of their prescribing of target classes of medications. In the 12 months before engaging in the curriculum, group model HMO participants prescribed a mean of 19.0 target medications per month. In the 6 months following the curriculum, these participants prescribed a mean of 19.8 target medications per month (

 = .60). Among curriculum nonparticipants, a mean of 17.5 target medications per month were prescribed in the 12-month precurriculum period, and a mean of 23.8 target medications per month were prescribed in the 6-month postcurriculum period (

 <.001). Therefore, although curriculum participants did not decrease target medication prescribing, their target medication prescribing did not increase, while nonparticipant prescribing of written by the curriculum participants versus the nonparticipants before and after the curriculum was offered showed no differences in prescribing changes between the 2 groups (prescribing of targeted medications rose by 0.8 prescription per month in the intervention group and by 6.3 prescriptions per month in the control group; 

	The results of this pilot study indicate that prescribing clinicians who participated subjectively improved their knowledge base after completing the curriculum. Some of these clinicians indicated their intent-to-change practice and identified barriers to practice change, and several clinicians reported changes in practice 3 to 6 months after completion of the curriculum. Prescribing of targeted medications did not change among participants after completion of the curriculum; however, prescribing of the same medications significantly increased among nonparticipants during the same time frame.

	Despite broadly publicizing this curriculum via e-mail and postal mail, few KPCO HMO and PPO clinicians participated in this curriculum, limiting the generalizability of our findings. This is likely because of multiple factors such as competing demands for clinician time at work and interest in and availability of several other CME opportunities. Furthermore, topics in this curricula are broadly applicable across multiple conditions; previous surveys have indicated that Kaiser Permanente physicians tend to prefer  disease-specific CME topics focusing on guidelines and evidence, while expressing less interest in programs focused on cross cutting topics such as clinician—patient communication and culturally competent care.

 The lack of participation also may reflect a sense among clinicians that they are immune to the effects of advertising and can discern marketing efforts from dissemination of balanced scientific content.

 shows that clinicians, especially those who have the most room for improvement, may not accurately selfassess and may not be motivated to engage in activities such as the educational curriculum we offered. Group model HMO clinicians particularly may not perceive the need to participate in this type of activity given other organizational efforts toward promoting cost-effective prescribing. Mandatory participation in curricula such as the one we developed may not lead to changes in practice, as clinicians may approach the activity with a “just get through it” mind-set rather than with openness to examining and changing practice.

	The 19 participants who responded to the follow-up survey noted several barriers to practice change. Despite the low response rate, the barriers identified are not unexpected given this topic; in addition, the barriers are consistent with those noted in other CME programs.

 Therefore, the barriers that participants noted in our study are likely generalizable to other settings and to similar curricula.

	Our analysis showed that nonparticipant prescribing of targeted medications increased, while participant prescribing remained fairly constant. Our direct comparison of participants with nonparticipants did not show a statistically significant difference between groups; however, our study may have been underpowered to detect a difference because of few participants. In addition, participation in this curriculum was voluntary, with potential selection bias toward participants who were more predisposed to modify their prescribing. If this is the case, one possible explanation is that our curricula helped participants buck the trend of increased prescribing of heavily advertised medications. Multiple other explanations (including no effect) are possible. Therefore, we can only speculate about the effects that curricula such as ours might have on prescribing of heavily marketed medications. Pending additional studies, the effect on prescribing trends among a broader selection of prescribers remains unknown.

	Despite these limitations, we noted several positive findings that other investigators can apply when developing interventions aimed to improve cost-effective prescribing. The multifaceted needs assessment method we used

 was useful targeted medications significantly increased in the same period. Results from the analysis of the number of prescriptions for designing a  linician-relevant curriculum on the pharmaceutical development, approval, and marketing process. Participants readily answered the open-ended questions in the modules; 30% of participants asked e-mail questions of faculty. This provided a low-cost, easily implemented method of promoting self-reflection and providing faculty interaction during online CME; interactivity and reflection have been shown to be more effective in leading to practice change than strictly didactic CME formats.

 Many participants made intent-to-change practice statements after completing the CME modules; intent-to-change practice statements have been shown to correlate with actual change in practice.

	What might be done to increase participation in curricula such as this not only in future studies but also in real-world practice? We suggest that prework, especially with prescribers who have increased rates of prescribing heavily marketed high-cost medications, should be performed (perhaps on an individual level) to encourage participants to engage in activities of this sort in a mode in which they are ready to learn. For example, brief individual academic detailing discussions

 could be used to highlight the differences between an individual’s prescribing habits and those of respected peers who prescribe in a more cost-effective manner. As part of these discussions, individuals could then be encouraged in a supportive manner to engage in curricula to help improve their prescribing. Embedding concepts of offlabel prescribing, accessing and appraising unbiased drug information, and talking with patients about direct-to-consumer advertising within disease-specific CME programs could help engender physician interest for subsequent broader-based curricula such as ours. Alternatively, repeatedly including and reinforcing these concepts in serial disease-specific CME programs could lead to generalized learning. We have had some success using these approaches with other cross-cutting topics; whether these approaches would influence prescribing over the long term and among a larger audience should be examined.

	The findings of our study suggest that curricula targeted on aspects of the pharmaceutical development, approval, and marketing process and positioned as part of a multifaceted effort might help mitigate the influence of pharmaceutical industry advertising on clinician prescribing behavior. Future studies should include larger samples, target clinicians based on prescribing patterns, engage clinicians in discussions using the clinician’s data to encourage participation, and tailor curricula to specific areas of prescribing to create a sense of need and relevance among participants. Consistent with studies

 on the effectiveness of CME, further educational endeavors should use multiple formats to present the information and provide reinforcement after the curriculum using audit and feedback, reminders, and opportunities for further questions, perhaps by individual academic detailing. Future endeavors might include the provision of incentives for participating in the curriculum and for completing postcurriculum intent-to-change practice statements. We also suggest that future efforts attempt to correlate intent-to-change practice statements with objective changes in practice. Finally, future studies should include prescribing information to facilitate direct comparison of prescribing patterns among curriculum participants versus nonparticipants.

: From Kaiser Permanente Institute for Health Research (DWP, MAR, DAC, NMC), Denver, CO; and Kaiser Permanente Colorado (LAW), Denver.

: This study was supported by a grant from the Attorney General Consumer and Prescriber Education Grant Program, which is funded by the multistate settlement of consumer fraud claims regarding the marketing of the prescription drug Neurontin. The grantors had no involvement in the design or conduct of the study; collection, management, analysis, or interpretation of the data; or preparation, review, or approval of the manuscript.

: The authors (DWP, MAR, DAC, LAW, NMC) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

: Concept and design (DWP, MAR); acquisition of data (MAR, DAC, LAW, NMC); analysis and interpretation of data (DWP, MAR, DAC, LAW, NMC); drafting of the manuscript (DWP, MAR, LAW, NMC); critical revision of the manuscript for important intellectual content (DWP, MAR, DAC, NMC); statistical analysis (DAC, LAW, NMC); obtaining funding (DWP, MAR); and supervision (DWP).

: David W. Price, MD, Kaiser Permanente Institute for Health Research, 10065 E Harvard Ave, Ste 300, Denver, CO 80231. E-mail: david.price@kp.org.

1. Price D, Raebel M, Conner D, Wright L.

 Assessing prescriber and  organizational leader educational needs about heavily marketed drugs.

Health, United States, 2007, With Chartbook on Trends in the Health of Americans

. Hyattsville, MD: US Dept of Health and Human Services, Centers for Disease Control and Prevention, National Center for Health Statistics; 2007. http://www.cdc.gov/nchs/data/hus/hus07.pdf. Accessed August 31, 2009.

3. National Institute for Health Care Management Research and Educational Foundation

Prescription drugs and mass media advertising, 2000

. Washington, DC: NIHCM Foundation; 2001. http://www.nihcm.org/~nihcmor/pdf/DTCbrief2001.pdf. Accessed October 17, 2008.

Direct marketing of pharmaceuticals to consumers. 

. Physicians and the pharmaceutical industry: is a gift ever just a gift? 

7. Steinman MA, Bero LA, Chren MM, Landefeld CS

. Narrative review: the promotion of gabapentin: an analysis of internal industry documents. 

8. Price DW, Overton CC, Duncan JP, et al.

 Results of the first national Kaiser Permanente continuing medical education needs assessment survey. 

. The impact of commercial support on medical education: the physician’s perspective. 

10. American Association of Medical Colleges

The Scientific Basis of Influence and Reciprocity: A Symposium. Washington

, DC: American Association of Medical Colleges; January 2008. http://www.aamc.org/publications. Accessed August 31, 2009.

 Self-assessment in the health professions: a reformulation and research agenda. 

12. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L

. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. 

Journal clubs and case conferences: from academic tradition to communities of practice. 

14. Thomson O’Brien MA, Freemantle N, Oxman AD, Wolf F, Davis DA, Herrin J

. Continuing education meetings and workshops: effects on professional practice and health care outcomes [update in 

15. Davis D, O’Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A

. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? 

 Continuing medical education and the physician as a learner: guide to the evidence. 

Commitment to change: a strategy for promoting educational effectiveness. 

18. Wakefield J, Herbert CP, Maclure M, et al

. Commitment to change statements can predict actual change in practice.

19. O’Brien MA, Rogers S, Jamtvedt G, et al.

 Educational outreach visits: effects on professional practice and health care outcomes. 

. From theory to practice: identifying authentic opinion leaders to improve care. 

21. NHS Centre for Reviews and Dissemination

. 1999;5(1):1-16. http://www.york.ac.uk/inst/crd/EHC/ehc51.pdf. Accessed December 2, 2008.

Articles

This article summarizes the outcomes from a pilot online curriculum on off-label prescribing, appraising pharmaceutical information, and talking with patients about advertised medications.

Pharmaceuticals From Development to Practice: Learnings From an Online Curriculum

To describe the proportion of patients receiving drugs with a narrow therapeutic range who lacked serum drug concentration monitoring during a 1-year period of therapy and to identify patient characteristics associated with lack of monitoring.

Ambulatory patients (n = 17 748) at 10 health maintenance organizations who were receiving ongoing continuous drug therapy with digoxin, carbamazepine, divalproex sodium, lithium carbonate, lithium citrate, phenobarbital sodium, phenytoin, phenytoin sodium, primidone, quinidine gluconate, quinidine sulfate, procainamide hydrochloride, theophylline, theophylline sodium glycinate, tacrolimus, or cyclosporine for at least 12 months between January 1, 1999, and June 30, 2001, were identified. Serum drug concentration monitoring was assessed from administrative data and from medical record data.

Fifty percent or more of patients receiving digoxin, theophylline, procainamide, quinidine, or primidone were not monitored, and 25% to 50% of patients receiving divalproex, carbamazepine, phenobarbital, phenytoin, or tacrolimus were not monitored. Younger age was associated with lack of monitoring for patients prescribed digoxin (adjusted odds ratio, 1.86; 95% confidence interval, 1.39-2.48) and theophylline (adjusted odds ratio, 1.58; 95% confidence interval, 1.23-2.04), while older age was associated with lack of monitoring for patients prescribed carbamazepine (adjusted odds ratio, 0.59; 95% confidence interval, 0.44-0.80) and divalproex (adjusted odds ratio, 0.50; 95% confidence interval, 0.38-0.66). Patients with fewer outpatient visits were also less likely to be monitored 

A substantial proportion of ambulatory patients receiving drugs with narrow intervals between doses resulting in beneficial and adverse effects did not have serum drug concentration monitoring during 1 year of use. Clinical implications of this finding need to be evaluated.

The goal of therapeutic drug monitoring is to guide dosing by means of drug concentration measurements. Therapeutic drug monitoring is useful for drugs that lack correlation between dose and pharmacodynamic properties, for drugs that have nonlinear correlations between dose and effect, and for drugs that have a narrow therapeutic range (NTR) between the dose necessary to achieve beneficial effects and the dose that causes serious adverse effects when there is a direct concentration-effect relationship. Individualizing the drug dosage and the dosing interval can minimize the toxicity and maximize the therapeutic benefit of NTR drugs.

The first step toward individualizing therapy is to evaluate the drug concentration in the body, often from a serum sample. Although there is controversy about whether monitoring should be routine, as well as about the frequency and timing of therapeutic drug monitoring to maximize beneficial drug effects,

there is general agreement that therapeutic drug monitoring is useful in avoiding drug toxicity. For example, drug concentration monitoring is recommended as a quality-of-care indicator for patients taking phenytoin, phenobarbital, divalproex sodium, and carbamazepine.

 Concentration monitoring of many NTR drugs is routinely available at healthcare laboratories, but little is known about the frequency of monitoring among ambulatory patients.

We undertook a study to assess drug concentration monitoring in ambulatory patients receiving NTR drugs. The objectives of this study were to describe the proportion of patients dispensed NTR drugs who did not have drug concentration monitoring during a 1-year therapy period and to identify patient characteristics associated with lack of monitoring. The results of this study provide information to guide thoughtful establishment of quality-of-care indicators related to NTR drug monitoring in ambulatory patients.

This retrospective cohort study of ongoing NTR drug therapy was designed to assess rates and correlates of serum drug concentration monitoring among ambulatory members of 10 health maintenance organizations in geographically diverse US locations. We studied monitoring practices for the following NTR drugs: carbamazepine, cyclosporine, digoxin, lithium, phenytoin, phenobarbital, primidone, procainamide hydrochloride, quinidine, tacrolimus, theophylline, and divalproex (valproic acid). The first study objective was to assess the proportion of ambulatory patients receiving continuous ongoing therapy with an NTR drug who did not have at least 1 drug serum concentration monitored within a 1-year period. The second study objective was to evaluate possible associations between lack of NTR drug monitoring and patient age, sex, outpatient visits, chronic diseases, diagnoses, selected socioeconomic status variables, and hospitalizations. Finally, the accuracy of using automated data in identifying drug concentration monitoring was assessed.

The participating organizations comprise the HMO Research Network Center for Education and Research in Therapeutics, which has been described elsewhere.

 In brief, the center includes staff, group, network, independent practice association, and mixed-model health maintenance organizations that serve racially and ethnically diverse populations and in 2000 provided healthcare for approximately 7 million people in more than 1000 locations. The institutional review board of each participating organization approved this study.

The study sample was drawn from a data set of 2 020 037 individuals, consisting of approximately 200 000 randomly selected health plan members from each of the 10 organizations. The sampling scheme and demographic distribution of this population have been previously described.

We identified patients who received an NTR drug of interest between January 1, 1999, and May 31, 2000, and who had continuous health plan membership with pharmacy benefits during the study period, disregarding gaps of less than 60 days. The study sample was limited to prevalent users who had continued drug dispensings of and ongoing therapy with an NTR drug. To avoid inadvertent inclusion of new drug users in the study cohort, prevalent ongoing therapy was defined as beginning with the second dispensing of that drug in the data set and continuing from the second dispensing date for 12 months or longer. Continued dispensings were present when no interval between prescription refills was greater than the dispensed days' supply plus 1.5 times the dispensed days' supply. A dispensing gap was ignored if it was less than 1.5 times the dispensed days' supply. For example, a patient dispensed a 30-day supply met the criterion of continued dispensings if no more than 75 days (30 days + [1.5 Ãƒâ€” 30 days]) elapsed between the date of one dispensing and the date of the subsequent dispensing. NTR drug dispensings were identified using National Drug Code numbers. For all eligible patients, we used automated health plan data to ascertain dates of health plan membership and drug dispensings.

Identification and Categorization of Drug

Lack of laboratory monitoring was defined as failure to perform drug concentration monitoring within the 1-year period of ongoing therapy, allowing up to a 1-month grace period (ie, monitoring within 13 months was considered as monitoring within 1 year). To accommodate patients entering the cohort between January 1, 1999, and May 31, 2000, monitoring data were collected for January 1, 1999, through June 30, 2001. Whether the laboratory test was performed was assessed from the presence of an administrative claim for the test. The laboratory test date was the date the test was performed or the result was reported. Laboratory test codes were defined at each site according to that site's method of coding the test (eg, 

 codes). One site was unable to contribute drug concentration monitoring data for carbamazepine, digoxin, cyclosporine, and phenytoin.

Medical records were randomly selected for review to assess the accuracy of administrative data. Because of resource limitations, this review included medical records of patients dispensed cyclosporine, digoxin, and carbamazepine. Medical record abstraction was considered the gold standard, and abstraction data were compared with administrative data to determine the sensitivity, specificity, positive predictive value, and negative predictive value of administrative information about drug concentration monitoring.

The health maintenance organization automated databases were used to identify the patient age (on the date of the first NTR drug dispensing), sex, outpatient visits, chronic diseases, diagnoses, and hospitalizations. The presence of specific chronic diseases was determined using the Chronic Disease Score method by Clark et al.

To evaluate characteristics that predicted lack of monitoring, hospitalizations and outpatient visits that occurred within 6 months before the study period were identified. Outpatient visits include clinic appointments, visits for laboratory testing only, and emergency department visits. Geocoding was used to provide surrogate patient-level measures of socioeconomic status. Residential street address was combined with census-block level data from the 2000 US Census data to construct proxies of patient race, education, and poverty.

The number of unique patient-drug therapy combinations and the number of patients were tabulated. Descriptive statistics were computed to characterize patients, drug dispensings, and drug concentration monitoring for each drug cohort. The proportions of patients taking each NTR drug who received drug concentration monitoring and those who did not receive drug concentration monitoring were tabulated overall and by health plan, age group, sex, outpatient visits, hospitalizations, Chronic Disease Score, race, education, and poverty categories. Statistical significance of differences was tested using the Wilcoxon rank sum test or the χ

The association between patient characteristics and drug concentration monitoring was evaluated using generalized estimating equation logistic regression modeling, with site as a cluster variable.

The initial generalized estimating equation model for each drug included the following variables with 

< .05 in the univariate analysis: age group, sex, outpatient visits (in increments of 5 visits), hospitalizations, the presence or absence of selected diagnoses, and the 3 linear socioeconomic status variables. The Chronic Disease Score was not included in any of the initial models because it was correlated with outpatient visits. We used the backward selection method for all generalized estimating equation models. Customary residual and effect statistics were examined to assess model fit and to evaluate outliers. Analyses were performed using SAS version 8.2 or 9.1 (SAS Institute Inc, Cary, NC).

Almost 18 000 patients (n = 17 748) met the eligibility criteria for the study; these patients represented 18 821 individual patient-drug therapy combinations (5.7% of patients received > 1 study drug). Patient characteristics are given in 

. Patients prescribed digoxin (n = 7153), divalproex (n = 2209), carbamazepine (n = 1876), phenytoin (n = 2232), theophylline (n = 2195), and lithium (n = 1487) accounted for 91% of the study patients. The median number of drug dispensings during 1 year ranged from 6 for digoxin to 12 for cyclosporine (data not shown).

The proportions of patients who were not monitored were low (14%) for cyclosporine and lithium (

). However, for patients prescribed digoxin, theophylline, procainamide, quinidine, or primidone, 50% or more were not monitored. Between 25% and 50% of patients prescribed divalproex, carbamazepine, phenobarbital, phenytoin, or tacrolimus did not have a serum concentration evaluated within the 1-year period.

Nine participating sites contributed data for all study drugs. At 8 of these sites, 36% to 48% of patients lacked monitoring; at the 9th site, 62% of patients were not monitored. Because the 10th site did not contribute data for 4 NTR drugs, the percentage of patients without monitoring for all NTR study drugs could not be determined.

A consistent association between patients' median age and lack of monitoring was not observed in the univariate analysis. Patients prescribed digoxin (77 vs 74 years, 

= .02), or theophylline (65 vs 61 years, 

< .001) who were less likely to be monitored were younger, while patients prescribed carbamazepine (45 vs 49 years, 

< .001) who were less likely to be monitored were older. For all study drugs, patients who were less likely to be monitored had a lower median number of outpatient visits. This difference was significant for digoxin (10 vs 8 visits, 

< .001), and phenobarbital (5 vs 4 visits, 

< .001) difference (but unlikely a clinically significant difference) in the median percentages of monitored patients who had at least a high school education (89% [5th-95th percentiles, 64%-98%]) compared with patients who had at least a high school education who lacked monitoring (88% [5th-95th percentiles, 62%-98%]). Because patient race was correlated with poverty and with education (Spearman rank correlation coefficient [r] range for race and poverty for individual drugs, -0.43 to -0.53; and r range for race and education for individual drugs, 0.36-0.45) and because poverty and education were correlated (r range for poverty and education for individual drugs, -0.31 to -0.63), race and poverty were not considered further in the analysis.

Logistic regression analysis was used to evaluate associations between patient characteristics and lack of monitoring for the 6 prescribed drugs that accounted for 91% of the study patients (

). When other characteristics were adjusted for, the patient characteristics most consistently associated with monitoring status continued to be age and outpatient visits, while the numbers of chronic diseases and recent hospitalizations were not associated with lack of monitoring for any of these drugs. Younger patients prescribed digoxin were more likely than the oldest patients prescribed digoxin to lack monitoring. Other factors associated with monitoring status of digoxin were female sex, the number of outpatient visits, and a diagnosis of arrhythmia or heart failure. These factors were associated with a lower likelihood of lacking monitoring (ie, patients with these factors were more likely to be monitored).

For patients prescribed theophylline, those in the youngest age group were more likely to lack monitoring than those in the oldest age group (Table 2). Fewer outpatient visits and a diagnosis of chronic obstructive pulmonary disease were associated with lacking theophylline monitoring.

In contrast to the relationship between younger patient age and the presence of monitoring that was observed for digoxin and theophylline, patients prescribed carbamazepine or divalproex who were in the oldest age group were more likely to lack monitoring (Table 2). For carbamazepine and divalproex, a seizure diagnosis or a mental health diagnosis was associated with a lower likelihood of lacking monitoring, as was an increasing number of outpatient visits.

Because patients prescribed carbamazepine or divalproex who did not have a seizure diagnosis or a mental health diagnosis had characteristics that were different from those of individuals with these diagnoses (

), we analyzed these patient groups separately. For carbamazepine, the association between being in the oldest age group and having a lower likelihood of monitoring was significant in patients with a mental health diagnosis (odds ratio, 0.24; 95% confidence interval, 0.11-0.54 for the youngest age group). Similarly, for divalproex, the association between being in the oldest age group and having a lower likelihood of monitoring was significant in patients with a mental health diagnosis (odds ratio, 0.47; 95% confidence interval, 0.32-0.69 for the youngest age group; and odds ratio, 0.53; 95% confidence interval, 0.36-0.77 for the second youngest age group), as well as for the youngest age group of patients who had neither a seizure diagnosis nor a mental health diagnosis (odds ratio, 0.42; 95% confidence interval, 0.22-0.83 for the youngest age group).

The sensitivity of claims data compared with medical record data for drug concentration monitoring was excellent (89% [47/53] for digoxin, 93% [70/75] for carbamazepine, and 95% [63/66] for cyclosporine). The specificity varied (94% [34/36] for digoxin, 78% [14/18] for carbamazepine, and 50% [11/22] for cyclosporine). The positive predictive value, the percentage of patients who had serum drug concentration monitoring according to administrative data that was also documented in medical records, was high (96% [47/49] for digoxin, 81% [60/74] for carbamazepine, and 85% [63/74] for cyclosporine). The negative predictive value, the percentage of patients who did not have serum drug concentration monitoring according to administrative data that was also not documented in medical records, was good (85% [34/40] for digoxin, 74% [14/19] for carbamazepine, and 79% [11/14] for cyclosporine).

Surprisingly high proportions of patients prescribed digoxin, theophylline, phenytoin, carbamazepine, procainamide, quinidine, primidone, divalproex, and phenobarbital (all medications with an NTR) do not have drug concentrations monitored at least yearly. In contrast, lithium and cyclosporine drug serum concentrations were monitored in most patients. Monitoring is recommended for these drugs because life-threatening toxicity can result when concentrations are elevated, therapeutic efficacy can be lost when concentrations are low, and correlations between drug dosages administered and concentrations achieved can be poor.

The most consistent patient characteristic associated with lack of monitoring was a low number of outpatient visits, a finding that is intuitive. For example, it is feasible that the high monitoring rate observed with cyclosporine is related to the fact that patients prescribed this drug have a high number of outpatient visits (median, 11 visits in a 6-month period) (Table 1).

Age was also often associated with lack of monitoring, but the association between age and monitoring varied by drug, with younger age being a predictor of lack of monitoring for patients prescribed digoxin and theophylline, and with older age being a predictor of lack of monitoring for patients prescribed carbamazepine and divalproex. The absence of a specific seizure diagnosis or mental health diagnosis for patients prescribed carbamazepine or divalproex increased the risk for lack of monitoring, as did the absence of an arrhythmia or heart failure diagnosis for patients prescribed digoxin and the absence of a chronic obstructive pulmonary disease diagnosis for patients prescribed theophylline.

Dosages of digoxin used to manage arrhythmias are often higher than dosages used to manage heart failure. Therefore, one could argue that monitoring is more important for patients with arrhythmias because their risk of drug toxicity is greater. Yet, we found that the presence of either diagnosis was similarly protective against lack of monitoring (Table 2).

Patients prescribed carbamazepine who have seizures or a diagnosis such as bipolar disorder are at higher risk of drug toxicity because they are treated with higher drug dosages than patients prescribed carbamazepine for an indication such as trigeminal neuralgia.

It is feasible that clinicians are less concerned about efficacy and toxicity monitoring in patients prescribed carbamazepine for diagnoses other than seizures or bipolar disorders. The same rationale can be applied to patients prescribed divalproex who do not have a seizure diagnosis or a mental health diagnosis. Unfortunately, this study was not designed to evaluate whether a relationship existed between dosage and monitoring, nor was it designed to differentiate between drug concentration monitoring that was not performed because it was not ordered vs because the patient did not obtain the ordered test.

Our work did not evaluate the appropriateness of drug concentration measurements relative to the need, timing, and interpretation of monitoring.

We acknowledge the importance of these factors in evaluating quality of care, but we believe that the information herein is meaningful even without information about the appropriateness of monitoring. Our findings document a widespread absence of drug concentration monitoring. Monitoring drug concentrations is viewed as a quality measure associated with avoiding preventable drug-related morbidity and disease exacerbations by organizations, including the National Committee for Quality Assurance in its Health Employer Information Data Set.

The findings of our study can be used by organizations to improve rates of drug concentration monitoring through implementing guidelines that target the patient risk groups we identify herein as lacking monitoring.

We found good sensitivity, specificity, positive predictive value, and negative predictive value of administrative data compared with medical record data across the organizations that participated in this study. However, the generalizability of our results is limited by the fact that quality of care and the accuracy of administrative data vary across organizations. The quality of diagnostic and procedure coding depends on institutional experience with and emphasis on coding, as well as contractual arrangements between health plans and institutions.

There is a dearth of information about whether drug concentration monitoring reduces adverse outcomes in patients. The evidence we present herein of low rates of drug concentration monitoring should prompt investigations of clinical and economic outcomes in ambulatory patients who are prescribed NTR drugs and who do not receive drug concentration monitoring. Patient outcomes should be used to guide the refinement of existing NTR drug monitoring quality-of-care indicators in ambulatory patients.

We thank Parker Pettus, MS, for leading the data management and computer programming required for this study. We thank Kimberly Lane, MPH, Andrea R. Paolino, MA, and Deanna Merrill, PharmD, MBA, for project management and medical record abstraction coordination. We acknowledge the programmers from each of the health maintenance organizations. We also acknowledge the helpful input provided by Laurie Crounse, MPH, and David McClure, MS, on the analytic plan for this work. Finally, we thank Paul H. Barrett, MD, MSPH, for his thoughtful review of a draft of the manuscript.

From the Clinical Research Unit (MAR, NMC) and Pharmacy Department (EAC), Kaiser Permanente of Colorado, and University of Colorado School of Pharmacy (MAR, EAC), Denver; Meyers Primary Care Institute, Fallon Foundation, and University of Massachusetts Medical School, Worcester (SEA), and Harvard Pilgrim Health Care and Harvard Medical School (SRS, RP) and Channing Laboratory, Brigham and Women's Hospital and Harvard Medical School (KAC, RP), Boston; Henry Ford Health System, Detroit, Mich (JEL); Kaiser Permanente Northwest Center for Health Research and Oregon Health Sciences University, Portland (AF); Lovelace Clinic Foundation, Albuquerque, NM (MJG); HealthPartners Research Foundation, Minneapolis, Minn (WWN); and Group Health Cooperative Center for Health Studies and Department of Epidemiology, University of Washington, Seattle (RLD).

This study was supported by cooperative agreement U18 HS 11843 and the HMO Research Network Center for Education and Research in Therapeutics, funded by the Agency for Healthcare Research and Quality, Rockville, Md.

The contents of this article are solely the opinions of the authors and do not necessarily represent the official views of the Agency for Healthcare Research and Quality, which did not participate in the design or conduct of the study; the collection, management, analysis, or interpretation of the data; or the preparation, review, or approval of the manuscript. Dr Raebel had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Address correspondence to: Marsha A. Raebel, PharmD, Clinical Research Unit, Kaiser Permanente of Colorado, PO Box 378066, Denver, CO 80237-8066. E-mail: marsha.a.raebel@kp.org.

Nikki M. Carroll, MS

Articles

Pharmaceuticals From Development to Practice: Learnings From an Online Curriculum

Monitoring of Drugs With a Narrow Therapeutic Range in Ambulatory Care