Our July 2008 supplement on Biologics and Benefit Design, funded by Amgen Inc, (Am J Manag Care. 2008;14[suppl 8]:S241-S268), included an error of omission on the Participant Disclosures page. 

The following disclosures for Robert Berglund should have been noted on S244:

Articles

CORRECTION STATEMENT

Participant Disclosure (Am J Manag Care. 2008;14[suppl 8]:S241-S268)

Objective: To determine the agreement between patient-reported symptoms of chest pain, dyspnea, and cough and the documentation of these symptoms by physicians in the electronic medical record.

Methods: Symptoms reported on patient-provided information forms between January 1, 2006, and June 30, 2006, were compared with those identified by natural language processing of the text of clinical notes from care providers. Terms that represent the 3 symptoms were used to search clinical notes electronically with subsequent manual identification of the context (eg, affirmative, negated, family history) in which they occurred. Results were reported using positive and negative agreement, and kappa statistics.

Results: Symptoms reported by 1119 patients age 18 years or older were compared with the nonnegated terms identified in their clinical notes. Positive agreement was 74, 70, and 63 for chest pain, dyspnea, and cough, while negative agreement was 78, 76, and 75, respectively. Kappa statistics were 0.52 (95% confidence interval [CI] = 0.44, 0.60) for chest pain, 0.46 (95% CI = 0.37, 0.54) for dyspnea, and 0.38 (95% CI = 0.28, 0.48) for cough. Positive agreement was higher for older men (P >.05), and negative agreement was higher for younger women (

Conclusions: We found discordance between patient self-report and documentation of symptoms in the medical record. This discordance has important implications for research studies that rely on symptom information for patient identification and may have clinical implications that must be evaluated for potential impact on quality of care, patient safety, and outcomes.

Three symptoms (chest pain, dyspnea, and cough) reported by patients were compared with symptoms identified by natural language processing of the text of clinical notes from care providers.

The electronic medical record documentation of symptoms differed from the symptoms self-reported by patients.

Research studies that rely on using symptom information must take this discordance into account.

This discordance also may have a potential impact on quality of care, patient safety, and outcomes.

Physical symptoms account for half of all outpatient visits in the United States

 The verbal characterization of the symptoms conveyed by the patient and recorded by care providers is central to the practice of clinical medicine, and increasing importance is attached to patient-centered clinical care.

 With the increasing adoption of the electronic medical record (EMR), free text of the clinical history can now be subjected to automated analysis in ways that are impossible or uneconomic with paper-based records.

 example of a symptomatic condition in which history taking is central to management is chronic stable angina pectoris,

 the number 1 cause of death in the entire Western world.

 In addition to its mortality burden, angina is associated with serious morbidity such as myocardial infarction and heart failure. Optimal methods of identifying patients with stable angina remain unclear. Many patients with typical symptoms are not diagnosed with angina,

 and age, sex, and ethnicity may influence physicians’ recommendations for diagnostic testing such as coronary angiography and the resulting 

International Classification of Diseases codes

Our preliminary findings indicate that natural language processing (NLP) of the free text of the EMR identifies patients with chronic angina pectoris

 who were missed by traditional diagnostic coding approaches. Natural language processing is a range of computational techniques aimed at extracting useful information from unstructured text. In the context of the EMR, NLP offers a promising method to automate the collection of a richer set of information for quality improvement and safety that would otherwise require manual chart abstraction.

Early identification of patients at risk for myocardial infarction is critical to its prevention and improved prognosis,

and is possible using patient-reportable information.

 Because the diagnosis of angina pectoris relies on the patient conveying the symptoms to the physician,

 the ambiguous nature of verbal communication as well as the nature of coronary artery disease (ie, its presentation varies according to patient race and sex) can make the diagnosis challenging. These challenges can lead to inconsistency and incompleteness in both the diagnosis and the information recorded in the EMR.

 Currently, the nature of the processes that lead to the creation of the EMR is poorly understood; however, prior research indicates that these processes have critical implications for clinical care and may have a significant impact on patient outcomes.

The primary objective of the current study is to determine whether there was disagreement between patient-reported symptoms of chest pain, dyspnea, and cough and the documentation of these symptoms by care providers in the EMR. Although in this study we focus on heart disease, which constitutes one of the top national healthcare priority areas,

 our findings have significant implications for any condition whose diagnosis and treatment relies on verbal presentation of symptoms.

Sources of Symptom Documentation For this study we identified 2 primary sources of symptom information: patient-provided information forms and clinical notes. Both are part of the Mayo Clinic EMR.

Electronic Medical Record and Clinical Notes

. The Mayo Clinic maintains an EMR for each inpatient and outpatient. A large part of the EMR consists of clinical notes, which represent transcriptions by trained medical transcriptionists of the dictations recorded by care providers after each contact with the patient. Clinical notes have been in use electronically since 1994; however, all services at the Mayo Clinic switched to electronic notes in 2005. The content of these notes complies with a nationally accepted standard, Health Level 7 Clinical Document Architecture, and consists of standard sections including chief complaint, history of present illness, review of systems, and impression/report/ plan, among others. This standard is used by most major EMR vendors in the United States.

. Many healthcare providers, including the Mayo Clinic, ask their patients to fill out forms detailing their prior health and social history, current symptoms, medications, and allergies. The Mayo Clinic’s form is reproduced in the eAppendix (available at 

. com). Figure 1 shows item 14 of this patient-provided information form, which asks the patient to indicate if he/she has ever experienced a symptom listed on the form. Subsequently, the patient’s responses are captured in a structured format by scanning the form into a database.

We used a subset of the database of patient-provided information entered during 6 months between January 1, 2006, and June 30, 2006, as the convenience sample consisting of 121,891 patients who filled out the form at least once during the 6-month period. The patients in this sample were not restricted to any geographic location and represent general Mayo Clinic ambulatory and hospitalized populations. We selected records that indicated either a positive or a negative response to questions about chest pain, chest pressure, shortness of breath/dyspnea, and cough. (These codes are internal identifiers provided here for reference and have no relation to any standardized nomenclature of medical concepts.) The analysis presented here is based on a combined symptom for chest pain that includes both the chest pain and chest pressure responses.

We manually examined clinical notes containing evidence of chest pain, dyspnea, and cough according to the procedure illustrated in Figure 2. For each of the symptoms, we randomly selected 200 patients who marked the symptom on at least 1 of their forms and 200 patients who did not mark the symptom on any of their forms. No restrictions other than age ≥18 years were applied. Only records of patients who had 1 or more clinical notes dictated during the study period (January 1, 2006, through June 30, 2006) were used in this study.

Each clinical note was searched automatically to identify and electronically mark the terms representing each of the symptoms and their orthographic variants and synonyms using search keywords arranged into natural language queries (Figure 3). The keywords and methods were similar to those previously reported.

 For example, a natural language query for “chest pain” identifies portions of clinical note text where 1 of the terms describing PAIN (eg, pressure) either precedes or follows 1 of the terms describing the LOCATION (eg, chest). Thus, this query will find either “chest pressure” or “pressure in the chest.” Subsequently, the text of the notes was manually examined by 2 nurse abstractors to determine the context in which each term appeared. The range of possible context labels that the abstractors could choose is displayed in column 1 of Table 1. The “conditional” context label was selected when the term was mentioned in a conditional context (eg, “I recommended nitroglycerin if he should develop chest pain”). Terms manually identified as “negated,” “family history,” “conditional,” or “unknown” were excluded from subsequent analysis.

In addition to identification of the context in which the query terms appeared, the abstractors manually identified all symptoms of chest pain, dyspnea, and cough in a random sample of 100 clinical notes. These manually identified symptoms were compared with those identified with automated natural language queries to determine their sensitivity.

Angina refers to a complex of symptoms, 1 of which may be chest pain or discomfort. Similarly to dyspnea, orthopnea and paroxysmal nocturnal dyspnea (PND) refer to a specific kind of shortness of breath. The patient-provided information form used for this study contains a question that covers this type of dyspnea (see “awakened with shortness of breath” in Figure 1). For the sensitivity analysis, we added the terms “angina,” “orthopnea,” and “PND” to the natural language queries to determine whether these related terms had a measurable effect on the agreement between symptoms reported by patients and care providers.

For the primary analysis and the symptom-reporting consistency, data were analyzed in terms of positive and negative agreement rather than sensitivity and specificity because neither the patient self-report nor the clinical notes could be considered a perfect criterion standard. Traditional kappa statistics are sensitive to imbalances in the marginal totals of 2 × 2 comparisons.

 Positive and negative agreement measures have been proposed as a way to ensure the correct interpretation of kappa values

 and have been used to assess the agreement between patient-reported information and the medical record.

 The computation of these measures is illustrated in Table 2, which represents a 2 × 2 table containing the counts where the patient responses were in concordance or discordance with the symptoms found in clinical notes for the same patient. Thus, the positive agreement is a ratio of the concordances in positive responses to the difference between the concordances in positive and negative responses added to the total number of samples according to the following formula: 100 2 (2

)]). The negative agreement is a ratio of the concordances in negative responses to the difference between the concordances in positive and negative responses subtracted from the total number of samples according to the following formula: 100 2 (2

 In addition we also report on the false-negative rate, which in our case shows the proportion of times when the patient reported a symptom but the symptom did not appear in the physician’s note [ie, false-negative rate = 100 2 (

))]. We report standard measures of sensitivity and specificity instead of positive and negative rates for the assessment of reliability of identification of negation by the NLP system because we used a manually created reference standard. Stratified analyses in sex and age subgroups were performed as well.

Of the 121,891 patients who filled out a patient-provided information form during 6 months between January 1, 2006, and June 30, 2006, 6569 patients (5.39%; 95% confidence interval [CI] = 5.26, 5.52) reported chest pain, 6166 patients (5.06%; 95% CI = 4.94, 5.18) reported chest pressure, 13,924 patients (11.42%; 95% CI = 11.24, 11.60) reported dyspnea, and 11,670 patients (9.57%; 95% CI = 9.41, 9.74) reported cough. Combining chest pain and chest pressure as synonymous terms yielded 10,518 patients (8.63%; 95% CI = 8.47, 8.79) who reported either chest pain or pressure.

Random sampling resulted in a study population of 1119 patients, with 373 patients in the chest pain group, 391 patients in the dyspnea group, 337 patients in the cough group, and 18 patients (1.6%) in more than 1 group. Of these 1119 patients, 499 (44.6%) were male, 127 (11.3%) were between age 18 and 34 years, 185 (16.5%) were between age 35 and 49 years, 337 (30.1%) were between age 50 and 64 years, and 470 (42.0%) were age 65 years or older. Twelve percent of the clinical notes for the 1119 patients were marked as a “Hospital Admission” or “Hospital Dismissal” note, thus representing an estimate of the notes from the hospitalized population. Only 6 patients filled out more than 1 patientprovided information form during the study period. Overall, the clinical notes in this study sample represent a variety of clinical specialties including primary care (20%), cardiovascular services (13%), physical medicine and rehabilitation (5%), hematology (4%), neurology (4%), endocrinology (4%), surgery (4%), psychology/psychiatry (4%), gastroenterology (3%), and emergency medicine (2%), among other less prevalent specialties.

The distribution of contexts illustrated in Table 1 shows that the terms denoting chest pain, dyspnea, and cough occur in negated contexts between 18% and 29% of the time. These observations are consistent with those of Chapman et al

; in their study, 27% of the findings and diseases were manually identified as negated in the process of creating a reference standard. We examined the precision of the NLP methodology for identification of the context (negated vs affirmative) in which symptoms were mentioned in the free text of clinical reports. Complete results are presented in Table 3. The sensitivity of the NLP negation algorithm was 83% (95% CI = 78, 88), 84% (95% CI = 80, 87), and 88% (95% CI = 84, 90) for cough, dyspnea, and chest pain, respectively. The specificity was 91% (95% CI = 90, 93), 93% (95% CI = 91, 94), and 92% (95% CI = 91, 94), for cough, dyspnea, and chest pain, respectively. The kappa values were 0.69 (95% CI = 0.63, 0.74) for cough, 0.74 (95% CI = 0.71, 0.78) for dyspnea, and 0.78 (95% CI = 0.75, 0.81) for chest pain.

In the random samples of 100 notes for each of the 3 symptoms, the abstractors identified 34 men tions of chest pain, 46 mentions of dyspnea, and 24 mentions of cough. Of the 34 chest pain mentions, 31 were also identified by the natural language queries yielding 91% (95% CI = 82, 100) sensitivity. Of the 46 dyspnea mentions, 45 were identified by the queries yielding 98% (95% CI = 94, 100) sensitivity. All 24 mentions of cough also were identified by automatic queries.

Agreement Between Clinical Notes and Patient-provided Information Forms

The results are summarized in Table 4. Overall, the positive agreement for chest pain, dyspnea, and cough was 74, 70, and 63, respectively, and the negative agreement was 78, 76, and 75, respectively. The kappa values were 0.52 (95% CI = 0.43, 0.60) for chest pain, 0.46 (95% CI = 0.37, 0.54) for dyspnea, and 0.38 (95% CI = 0.28, 0.48) for cough. Additional analysis was carried out within different sex and age strata to determine whether the concordance and discordance are influenced by these factors. Positive agreement was slightly lower for female patients across all 3 symptoms, and tended to be lower for younger subgroups. Negative agreement varied and tended to be lower for males and older subgroups. Kappa values ranged from 0.44 to 0.55 for chest pain, 0.27 to 0.52 for dyspnea, and 0.31 to 0.56 for cough.

Including nonnegated mentions of angina resulted in identification of 5 additional patients who self-reported chest pain and 3 additional patients who did not, yielding positive agreement of 76 and negative agreement of 75. Including nonnegated mentions of orthopnea and PND resulted in identifying 1 additional patient who did not self-report dyspnea, yielding positive agree ment of 70 and negative agreement of 75.

Distribution of Clinical Specialties for Discordant Results

We determined the distributions of clinical services that reflect the clinical specialty of care providers for the discordant results for each of the 3 symptoms (Table 5). The distributions were computed by counting the number of clinical notes originating in each of the top 10 most prevalent clinical services. Consistent with the overall distribution of clinical services in our sample, cardiovascular services and primary care/family medicine were the top 2 most prevalent specialties for the patients in the discordant groups. The majority of the notes for patients reporting chest pain that was not documented in the EMR were distributed among primary care, cardiovascular services, and endocrinology. A similar distribution was found for dyspnea. For cough, the majority of the notes were distributed between primary care, cardiovascular services, and physical medicine and rehabilitation services. The majority of the notes for patients who had documentation of chest pain in the EMR but did not report it on patient-provided information forms were distributed among primary care, psychology/ psychiatry, hematology, and cardiovascular services. A similar distribution was found for cough, with a large proportion of notes from the hematology service. For dyspnea, the majority of the notes originated from primary care and cardiovascular services.

Electronic medical records offer a way to make improvements in the quality and safety of patient care by addressing the information challenge of organizing and making patient charts accessible and interoperable across healthcare providers.

 The primary and most important use of the EMR is to facilitate and streamline care delivery; however, its secondary uses for clinical research and quality and safety assurance also are important.

 From the standpoint of the primary use of the EMR, we currently do not have the data to determine whether the discordances between the symptoms reported by patients and those documented by care providers had any significant clinical consequences. However, we do present evidence that effective secondary uses of the EMR require examination of the communication approaches of care providers and patients.

Our findings indicate substantial discordance between patient reporting and care provider documentation of the symptoms. The 2 sources of the symptoms may complement each other and have implications for clinical studies and quality measurements that rely on the medical record for identification of symptoms. For example, to ensure completeness of identifying and recruiting participants with specific symptoms for clinical studies, it may be necessary to use the information reported by patients on self-entry forms in addition to other sources such as the clinical notes. It also may be necessary to use both sources of information for applications such as postmarketing medication safety surveillance, where it may be necessary to detect alarming trends in symptoms to prevent more serious adverse drug reactions. The current study is the first step toward understanding the implications of symptom documentation practices for both primary and secondary uses of the EMR.

The results in Table 3 indicate that NLP is a valid tool for identification of symptoms that occur in negated contexts in the free text of clinical reports. These results are promising because they show that the NLP methods used in this pilot study may be applied retrospectively to an existing cohort of patients with known outcomes to determine whether there was an association between the outcomes and symptom documentation. We also found that the methods used to identify the mentions of chest pain, dyspnea, and cough are highly sensitive. Automated natural language queries missed chest “tightness,” “sense of bruising in the lateral and posterior chest wall,” “anginal symptoms,” and “dyspneic.” Capturing these types of cases would require only minor modifications to the algorithms used for term identification. Inclusion of the terms “angina,” “orthopnea,” and “PND” resulted in very minor changes in the positive and negative agreement, not affecting the conclusions of the primary analysis. It is likely that these more specialized terms are used in conjunction with more generic terms such as “chest pain” and “dyspnea” and thus do not yield additional patients.

In a previous study of the Mayo Clinic EMR, St. Sauver et al showed that positive reports of cardiovascular disease risk factor information (blood pressure; triglycerides; cholesterol; history of heart rhythm, heart valve, and arterial problems) by patients are largely inaccurate, while negative reports were unlikely to be noted on the medical record by care providers, thus showing a low level of agreement between patient-reported risk factors and physician documentation.

 The study by St. Sauver et al, however, did not address symptom reporting and documentation, as that was not in the scope of their research questions. Although patient self-report is a secondary source of information on lab results and past medical diagnoses, with physician-documented medical record being the primary source (thus perhaps contributing to the low agreement found by St. Sauver et al

), the situation is reversed for patient symptoms in cases where the patient is the primary source and the care provider is the secondary source. The results of the current study are consistent with those reported by St. Sauver et al and contribute to constructing a more complete picture of clinical process and documentation.

Although we found discordance between patient-reported and physician-reported information, the causes of such discordances can only be hypothesized. Indeed, discordance may reflect the nature of the patient-physician interaction or other factors such as clinical specialty. For example, for discordances on chest pain reporting and documentation, one might expect to find the majority of clinical notes originating from noncardiovascular services. Our data shows that although this is true, a relatively large proportion of the notes (24%) originated from cardiovascular services. The results are similar for dyspnea and cough. The majority of the notes for patients who reported chest pain not documented in the EMR originated in 3 services: cardiovascular, endocrinology, and primary care. Direct examination of the patient-physician interaction would be necessary to determine the reasons for the discordance; however, the reasons may differ by specialty. Furthermore, the discordance also may reflect the specifics of the documentation system. The Mayo Clinic uses an integrated EMR where all notes and forms (including the patient-provided information forms) from all services for a given patient are available to any clinician working with the patient. Clinicians may decide to avoid repeating a symptom in their clinical note that is already documented elsewhere. Thus, the nature and the consequences of the interaction between the individual characteristics of the documentation systems, the care providers, and the patients warrant further investigation within the framework of patient-centered care.

Of interest are those instances where the patient reported a symptom not documented in the record (ie, false negatives); 31% of patients with chest pain, 38% with dyspnea, and 45% with cough did not have a positive mention of the symptom in the clinical notes. We could hypothesize that the outcomes of such patients may differ from those for whom symptoms were documented in the EMR. Conversely, when the patient does not report a symptom documented in the EMR, such discordance may reflect differential elicitation of symptoms by care providers by specialty. As these considerations remain speculative, further study is needed to examine the reasons for discordances and to determine whether these affect clinical outcomes. The present study shows that the discordances exist, which has implications for clinical research that relies on the EMR to ascertain symptoms.

The generalizability of the study depends on the availability of the EMR; however, the adoption of EMRs across the United States is growing.

 Many medical centers already have their patient notes in electronic form, and systems for accessing the information in the notes can be constructed even without a fully integrated EMR system. Although the data sources used in this study are specific to the Mayo Clinic, collecting patient-reported information before a visit and physician documentation of the visit are standard healthcare practices, and similar resources are in use at other institutions across the United States. Thus, our results are generalizable to other institutions that are equipped with an EMR and can guide the improvement of existing EMR systems by suggesting new avenues for improved elicitation and capture of patientspecific information central to patient care and research.

In this study, we did not look for an association between the literacy level or proficiency in the English language. We did not have these data available for this study, but we recognize that these are important variables to consider because the forms require a certain level of knowledge of the English language. Care providers’ demographic characteristics (as well as the interaction of these characteristics with patient characteristics) also may be important predictors of concordances and discordances in documentation. These characteristics were not available for the current study but will be considered in future work.

The use of the state-of-the-art EMR environment at the Mayo Clinic is a distinct strength of this study, enabling an investigation of the discrepancies between patient-reported symptoms and provider documentation. This study presents a novel approach that relies on automated and thus easily scalable investigation of the free text contained in provider documentation; this approach can be used in future large-scale studies of the interaction between the patient and the care provider. Despite the fact that some natural language term variants (particularly those due to misspellings and nonstandard descriptions) are likely to be missed by the automated natural language queries used in this project, our methodology is highly sensitive for identification of chest pain, dyspnea, and cough.

We thank Ellen Koepsell and Diane Batzel for their dedicated efforts in annotating clinical notes.

: From the Department of Pharmaceutical Care and Health Systems, University of Minnesota (SVP), Minneapolis; the Department of Research and Evaluation, Kaiser Permanente (SJJ), Pasadena, CA; and the Department of Health Sciences Research (CGC) and the Division of Cardiovascular Diseases (VLR), Mayo Clinic, Rochester, MN.

: This work was supported by Public Health Service grants RO1-72435, GM14321, and AR30582, and the NIH Roadmap Multidisciplinary Clinical Research Career Development Award Grant (K12/NICHDHD49078).

: The authors (SVP, SJJ, CGC, VLR) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

: Concept and design (SVP, SJJ, CGC); acquisition of data (SVP, VLR); analysis and interpretation of data (SVP, SJJ, VLR); drafting of the manuscript (SVP, VLR); critical revision of the manuscript for important intellectual content (SVP, SJJ, CGC); statistical analysis (SVP, VLR); provision of study materials or patients (SVP); obtaining funding (SVP); administrative, technical, or logistic support (SVP, CGC); and supervision (SVP, CGC, VLR).

 Serguei V. Pakhomov, PhD, Department of Pharmaceutical Care and Health Systems, University of Minnesota, 308 Harvard St, SE, 7-125F Weaver-Densford Hall, Minneapolis, MN 55401. E-mail: 

. National Ambulatory Medical Care Survey: 2004 summary. 

. New mind/body tactics target medically unexplained physical symptoms and fears. 

3. Committee on Quality of Health Care in America, Institute of Medicine

Crossing the Quality Chasm: A New Health System for the 21st Century

. Washington, DC: Institute of Medicine; 2001.

4. Sager N, Lyman M, Bucknall C, Nhan N, Tick LJ

. Natural language processing and the representation of clinical data. 

A broad-coverage natural language processing system. In: Overhage JM, ed. Proceedings of the 2000 AMIA Annual Symposium

. Bethesda, MD: American Medical Informatics Association; 2000:270-274.

. Automated detection of adverse events using natural language processing of discharge summaries. 

7. Javitz HS, Ward MM, Watson JB, Jaana M

8. Hemingway H, McCallum A, Shipley M, Manderbacka K, Martikainen P, Keskimaki I

. Incidence and prognostic implications of stable angina pectoris among women and men. 

9. Thom T, Haase N, Rosamond W, et al; American Heart Association Statistics Committee and Stroke Statistics Subcommittee

. Heart disease and stroke statisticsÃ¢â‚¬â€

METHODS

Effective use of electronic medical record technology requires examination of the communication approaches of both care providers and patients.

Agreement Between Patient-reported Symptoms and Their Documentation in the Medical Record

: To update the most recent meta-analysis comparing percutaneous coronary interventions (PCIs) with medical therapy (MT) in patients having stable coronary artery disease (CAD) by including 2 new large trials that double the total number of patients.

: Meta-analysis was used to update previous meta-analyses of PCIs in stable CAD. Eleven previously analyzed randomized controlled trials (RCTs) and 2 new RCTs were included.

: Summary estimates of relative risk (RR) are obtained by applying fixed-effects and random-effects models. Statistical tests for assessing between-study heterogeneity and biases are performed. Cumulative estimates and results from influence analysis are reported.

: No difference between PCIs and MT alone was found for risk of mortality. There was a 12% increase in the RR of cardiac death or myocardial infarction (MI) associated with PCIs, as well as a 22% increase in the RR of nonfatal MI associated with PCIs. Cumulative analysis favored MT over PCIs as early as 1997, but recent study results have increased confidence in this finding. Because of heterogeneity between studies, no certain conclusions are drawn for the use of PCIs in preventing follow-up PCI or coronary artery bypass graft surgery. 

: Recent RCTs comparing PCIs with conservative MT in stable CAD increase confidence in previous findings that the use of PCIs does not offer marginal benefit over that of the use of MT alone for mortality risk, cardiac death or MI, and nonfatal MI.

The weight of evidence has not shown percutaneous coronary interventions (PCIs) to have marginal benefit in the treatment of stable coronary artery disease over that of medical therapy alone; recent large trials increase our confidence in this finding. 

Original favorable coverage decisions by Medicare were not based on whether PCIs provided marginal benefit over that of medical therapy; no randomized controlled trials were available for evaluation at that time.

Early diffusion of PCIs was likely influenced by Medicareâ€™s decision to pay the same rate as open heart surgery and by early evaluation suggesting that PCIs could save money.

It has been 31 years since the 1977 development of percutaneous transluminal angioplasty by Gr&#252;ntzig

 for use in coronary arteries. The early diffusion of percutaneous coronary interventions (PCIs, previously percutaneous transluminal coronary angioplasty

) benefited from favorable Medicare pricing.

 Early Medicare payment for coronary angioplasty was based on coding to 

International Classification of Diseases, Ninth Revision

, Clinical Modification procedure code 36.0 (&#8220;removal of a coronary obstruction&#8221;), which was reimbursed through the inpatient hospital diagnosis related group system as open heart surgery (diagnosis related group 108), despite the lower costs of PCIs.

 cites 1982 costs for coronary artery bypass graft (CABG) surgery as $20,000 and costs for PCIs as $3000. Zweifel

 notes that, while new technology lowers the unit cost of production among industry in healthcare, it drives up the total cost. The use of PCIs in the United States grew from 1800 cumulative procedures from June 1979 through May 1981, to 32,300 procedures in 1983, to more than 200,000 procedures in 1988. From 2000 through 2005, more than 7 million coronary artery angioplasties, arthrectomies, or stent insertions have been performed in the United States.

 Using the mean national Medicare hospital payment rate for fiscal 2006 of $13,793, the procedures could represent hospital cost ranging from $54 billion to $97.4 billion. Estimates of national commercial insurance mean costs for PCIs, including physician fees, approximate $24,400 per case, giving a total cost estimate of $96 billion to $170 billion for the 6-year period from 2000 to 2005. (The $24,400 value is based on the BlueCross BlueShield of Tennessee treatment cost estimator. The treatment cost estimator is a collaborative project of Consortium Health Plans, Blue- Cross BlueShield Association, and Milliman USA. Based on the ClaimsQuest cost modeling system, the treatment cost estimates are episode treatment group&#8211;based. For the $96-billion to $170-billion estimate, there is considerable uncertainty surrounding the values. The source of the uncertainty is accounting for the number of procedures versus the number of patients; that is, a single patient could have both angioplasty and stent procedures performed within a single stay, for which there could be a single-case rate payment, a per diem rate&#8211;based payment, or a payment based on the fees for individual procedures and implanted stents.) Despite the high level of investment and experience with this technology, the marginal benefit of use in stable coronary artery disease (CAD) remains controversial, with study results often in conflict.

of randomized controlled trials (RCTs) of PCIs versus medical therapy (MT) have shown no marginal benefit for PCIs over that of MT alone in death rates, myocardial infarction (MI), or need for subsequent revascularization in patients with nonacute CAD. Since the publication of the most recent meta-analysis,

 2 large RCTs have been added to the literature,

more than doubling the total number of patients studied. The results of these new studies potentially have an important role in the development of policy regarding PCIs. The new studies have been portrayed in the lay press as &#8220;landmark,&#8221;

 implying that there is a historical change of course from the findings of previous studies. We update the meta-analysis of Katritsis and Ioannidis by including the 2 new studies 

and by addressing the consistency, heterogeneity, and bias of previous findings.

We employed eligibility criteria, studied outcomes data, and meta-analytic methods of Katritsis and Ioannidis,

 whose principal outcome measure was relative risk (RR). In addition to summary outcome measures and heterogeneity test results, we present cumulative estimates and influence analysis, and test for the existence of biases.

Relative risks across studies were combined using the Mantel- Haenszel fixed-effects model and the DerSimonian and Laird random-effects model.

In general, both methods give greater weight to studies with smaller variances. Weighting is shown graphically in forest plots, in which the size of the box is proportional to study weight (which is also shown in the plots) and in which the center of the box represents the size of the treatment effect estimated from that study (point estimate). The confidence interval (CI) for the treatment effect from each study is also shown. The pooled treatment effect is indicated by the middle of the diamond. An estimate less than 1.0 favors PCIs, while an estimate of 1.0 or greater favors MT alone. Influence analysis was performed by omitting 1 study at a time from the Mantel-Haenszel or DerSimonian and Laird model. A study is influential if, when omitted, the combined meta-analytic estimate differs significantly from that achieved when the study is included. Cumulative effects are shown in the cumulativeeffects plot, in which the first circle represents the results of the first study only, while subsequent circles represent the weighted results of all previous studies in addition to the current study. Between- study heterogeneity was tested using &#967;

-based Q statistic. Publication and other biases were investigated using funnel plots28 and formally using Egger&#8217;s regression asymmetry test.

We account for any biases detected by applying the &#8220;trim and fill&#8221; method developed by Duval and Tweedie.

Problems in the computation of RR and standard errors of RR arise when a study contains no events (the meta-analysis studies

 report no deaths in the intervention group, so that the estimated RR ratio is zero and its standard error cannot be estimated. To overcome this problem, we add 0.5 to each observed frequency (ie, to each cell of the 2 &#215; 2 contingency table for the trial). When the treated and the control groups of the study contain zero events, presenting effect size estimates as RR ratios is meaningless. We have one such case (by Hambrecht et al

) for death due to all causes, and we discard it from that portion of the meta-analysis.

The primary end points are the following clinical outcomes: death (due to all causes), cardiac death or MI, nonfatal MI, and follow-up procedures that include PCIs and CABG surgery. 

There were 509 deaths (250 in the PCI group and 259 in the MT group), 855 patients had cardiac death or MI (452 in the PCI group and 403 in the MT group), 520 patients had nonfatal MI (287 in the PCI group and 233 in the MT group), and 1628 patients underwent follow-up PCIs or CABG surgery (726 in the PCI group and 902 in the MT group). Pooled estimates for the 13 RCTs 

No difference between PCIs and MT alone was found for risk of mortality. There was a 12% increase in the RR of cardiac death or MI associated with PCIs, as well as a 22% increase in the RR of nonfatal MI associated with PCIs. 

One trial, the Arbeitsgemeinschaft Leitende Kardiologische Krankenhaus&#228;rzte (ALKK),

 reports an RR and a 95% CI that are below 1.0 for mortality (RR, 0.36; 95% CI, 0.15-0.88), while another trial, the Medicine, Angioplasty, or Surgery Study (MASS-II),

 reports an RR and a 95% CI that are above 1.0 for mortality (RR, 3.63; 95% CI, 1.03-12.82). The remaining trials report RRs and CIs for mortality that include 1.0 (

). The pooled RR for mortality is 0.97 (95% CI, 0.82-1.14). For cardiac death or MI, the ALKK46 reports an RR and a 95% CI that terminate at 1.0 from below, while the MASS-II43 reports an RR and a 95% CI that terminate at 1.0 from above (

). The pooled RR for cardiac death or MI is 1.12 (95% CI, 0.99-1.27). No individual trial reports an RR and a CI that do not include 1.0 for nonfatal MI; the pooled RR is 1.22 (95% CI, 1.04-1.44) (

). Evidence from individual RR ratios is mixed for follow-up PCI or CABG surgery, with a pooled RR estimate of 1.00 (95% CI, 0.80-1.26) (

). Additional results are given in the online appendix (

This study adds to the current body of literature on PCIs by considering between-study heterogeneity, bias, consistency of outcomes among studies, inclusion of 2 recent large studies, and historical aspects of PCI acceptance by payers and healthcare providers. Consideration of these features affirms and increases confidence in the previous findings. We also find that contrary to perception, except for the first RCT,

 cumulative effects support the finding that PCIs offer no marginal benefit over MT for the 4 outcomes studied herein. Consideration of how PCIs became covered by Medicare and were accepted by healthcare providers lends credence to the conclusion that PCIs provide no marginal benefit over MT alone. Policy may be further informed by these findings because they may prompt additional service review by payers, employers, and consumers in addition to providers.

A meta-analysis by Katritsis and Ioannidis

 of 11 RCTs comparing PCIs with conservative treatment in patients with stable CAD was updated to include 2 recent large trials.

The following 4 outcomes were considered: (1) For death, the combined estimate of RR is 0.97, implying no difference between PCIs and MT. (2 and 3) For cardiac death or MI and for nonfatal MI, the combined estimates are greater than 1.0 (1.12 and 1.22, respectively), favoring MT. (4) When the clinical outcome is follow-up PCI or CABG surgery, randomeffects estimation accounting for between-study heterogeneity demonstrates a wide 95% CI for the pooled RR, which prevents inferences about the superiority of PCIs versus MT. No evidence of between-study heterogeneity is found for death, cardiac death or MI, and nonfatal MI (Q statistics, 

 = .86, respectively). Because of the small number of studies, the test of homogeneity has low power to detect excess variation (variation beyond sampling error) and cannot be interpreted as evidence of homogeneity. However, fixed-effects and random-effects estimates are remarkably close for these outcomes, implying the absence of between-study heterogeneity. The null hypothesis for unitary RR is rejected for nonfatal MI (

The fixed-effects estimate of RR for follow-up PCI or CABG surgery in patients treated with PCIs compared with MT is 0.81. However, because of substantial between-study heterogeneity (&#967;

 <.01), this estimate is unreliable. By ignoring between-study variability, the fixed-effects model underestimates the standard error of the summary estimate and creates a misleadingly narrow CI. The substantial between-study variability is taken into account in the computation of the randomeffects estimate (RR, 1.00) and is reflected in a wider 95% CI of 0.80 to 1.26. Consequently, no conclusions can be made based on the more conservative random- effects estimate. Four trials, the Clinical Outcomes Utilizing Re vascularization and Aggressive Drug Evaluation (COURAGE), 

 Second Randomized Intervention Treatment of Angina (RITA-2),41 and ALKK,

 favors MT. Because the sources of heterogeneity are unclear, we cannot produce an overall estimate that is conclusive.

) account for 77.5% of total weight for mortality, 83.2% for cardiac death or MI, 81.4% for nonfatal MI, all fixed-effects models, and 41.7% of the total weight for follow-up PCI or CABG surgery; a random-effects model was used because of substantial between-study heterogeneity. The ALKK

 favors PCIs for the outcomes of death, cardiac death or MI, and follow-up PCI or CABG surgery, accounting for 6.5%, 6.4%, and 5.9%, respectively, of the total weight. The MASS-II

 favors MT alone and represents 1.2% and 3.2% of the total weight for death and for cardiac death or MI, respectively. COURAGE,

 favor PCIs for follow-up PCI or CABG surgery, accounting for 41.7% of the total weight, while MASS

 favors MT, accounting for 7.5% of the total weight.

), trials are presented in chronologic order from the earliest (Sievers et al

 [2007]), and cumulative summary estimates through the publication of each study are shown at that study&#8217;s publication date. Publication of the Veterans Affairs ACME Investigators trial in 1997 (ACME-2)

 provided insight that there was unlikely to be a difference in performance between PCIs and MT alone for the 4 outcomes studied. The cumulative meta-analysis plot shows that, although the new trials (OAT

) have not changed the estimated effects of PCIs on mortality, they significantly reduce the CI for the pooled estimate. Increased confidence in the otherwise known findings established by the contributions of OAT

 may engender additional support for a policy cognizant of the lack of marginal benefit of PCIs over that of MT alone in nonacute cases. Cumulative analysis for cardiac death or MI, nonfatal MI, and follow-up PCI or CABG surgery also provides support for these conclusions.

The influence analysis shows that for death and for cardiac death or MI, the studies that have the largest influence are ALKK,

 Omission of other studies makes little or no difference. Follow-up PCI or CABG surgery is dominated by the COURAGE

Factors that may lead to a biased overall effect size estimate are summarized by Egger et al.

 Publication bias (in which smaller studies showing negative effect [or no effect] of the treatment are less likely to get published) is usually the most serious concern, but we did not observe it herein. Quality bias was observed for nonfatal MI (funnel plots in eAppendix), which occurs when there are important differences in methodological quality between small and large studies. The lack of important information about disease and study characteristics in the study by Sievers et al

 provides additional evidence for this explanation. Published as an abstract, the study provides no information about the enrollment period, percentage of men and subjects with diabetes mellitus, existence of symptoms, or mean ejection fraction.

The existence of biases in estimating the combined RR for nonfatal MI does not affect the conclusion that MT alone is favored compared with PCIs. To show this, we performed a sensitivity analysis that excluded the 4 studies 

with large RRs. The overall fixed-effects and random-effects estimate is 1.19 (95% CI, 1.01-1.41). The Duval and Tweedie nonparametric trim and fill method accounts for the bias by estimating the number and RRs of missing trials and adjusts the meta-analysis to incorporate the theoretical missing trials. The combined estimate using all 16 trials (13 actual trials and 3 theoretical trials created by the method of Duval and Tweedie) is 1.18 (95% CI, 1.01-1.40) (

). These estimates are close to those reported earlier, providing indirect evidence that bias did not seriously influence the summary estimate.

The US Food and Drug Administration released the first catheter marketed for PCIs from &#8220;investigational device exemption status&#8221; in late 1980, before publication of any RCTs demonstrating effectiveness.

A 1983 survey of Medicare contractors to whom the coverage decision was delegated, also conducted before publication of any RCTs, found that 61.1% covered single-vessel PCIs, 19.2% covered PCIs of 2 or more vessels without qualification, and another 19.2% covered PCIs of 2 or more vessels with qualification.

 The Medicare coverage policy was that single&#8211;coronary artery PCIs were covered, while PCIs of 2 or more coronary arteries were not covered, although procedure effectiveness and safety had not been established. As Medicare inpatient hospital payment transitioned from cost-based to prospective payment, the effect of the new prospective payment system on medical technology adoption was a major concern of the Office of Technology Assessment.

 Technology was classified as the following 4 types: (1) cost-raising, quality-enhancing new technology; (2) operating cost-saving innovations; (3) capital cost-saving innovations; and (4) service or procedure disadoption. New diagnostic and therapeutic procedures &#8220;that reduce LOS [length of stay] or intensity of care&#8221; were classified as technology type 2. The diffusion of PCI technology occurred in response to prices set by the Centers for Medicare & Medicaid Services, then known as the Health Care Finance Administration (HCFA) for payment of Medicare cases. It is likely the appeal to the HCFA and other payers was that of an inexpensive substitute for CABG surgery. In 1982, Sawi

, on page 172, notes the following: &#8220;[M]ost private carriers reimburse for PTCA [percutaneous transluminal coronary angioplasty] to some extent either knowingly or otherwise. In some cases, the charges for PTCA are &#8216;buried&#8217; in other, chargeable, items, such as catheter laboratory charges, coronary angiograms, etc.&#8221;

A possible explanation for HCFA&#8217;s action to approve the use of PCIs was the fear of discouraging technologic innovation as payment structures changed. Per cutaneous coronary inter ventions were accepted without adequate science to support their use. Payers accepted them largely based on the potential of PCIs to save costs, and healthcare providers accepted them based on favorable pricing. After 2 decades of PCI use, science was able to inform practice. After 3 decades of use, it can be said with confidence that the evidence rejects the hypothesis that the use of PCIs in stable CAD has a marginal benefit over that of MT alone. The downstream consequences of multiple decisions favoring PCIs, made without adequate science to support them, are the following: (1) The estimates of costs for PCIs are likely to have ranged from $15 billion to $28.5 billion in 2006. (2) It is unclear whether there will be adequate study in the near term to assess the value of PCIs in unstable CAD, although 3 studies (Medicine vs Angiography in Thrombolytic Exclusion [MATE],49 Thrombolysis in Myocardial Ischemia (TIMI IIIB),

 and Veterans Affairs Non&#8211;Q-Wave Infarction Strategies in Hospital [VANQWISH]

 have shown no marginal benefit over that of MT alone. (3) The use of PCIs in stable CAD will not suddenly drop to zero; that is, investment without positive return in terms of health over that provided by MT alone will continue for a period. (4) The question of how the fixed costs invested to perform PCIs will be managed will be a much more difficult policy question to address than the decision to cover PCIs in the first place.

The policy implications of covering very expensive health technologies include the following: (1) the need to be able to quantify and compare effectiveness and costs, (2) an understanding of the ethical dimensions of equity and the transfer of income from those who will receive the new technology, and (3) a consideration of the relative societal value of investing in the medical technology compared with, for example, investment in education. When such large sums are considered, the effect on other economic sectors, jobs, and overall global competitiveness in the face of such an investment must be added to the equation.

Our analysis is limited in that the relative effectiveness of PCIs in treating angina is not considered. The presence of bias and between-study heterogeneity limits our ability to draw certain conclusions for the use of PCIs in preventing follow-up PCI or CABG surgery, although the evidence does not favor PCIs. The use of PCIs in unstable CAD is also not considered in the present analysis. 

: BlueCross BlueShield of Tennessee, Chattanooga (WTC, PK, JWB, RSM, KP, SM).

: The authors (WTC, PK, JWB, RSM, KP, SM) are employees of Blue- Cross BlueShield of Tennessee and report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

: Concept and design (WTC, JWB, RSM, KP, SM); acquisition of data (PK); analysis and interpretation of data (WTC, PK); drafting of the manuscript (WTC, PK, JWB, RSM, KP, SM); critical revision of the manuscript for important intellectual content (WTC, PK, JWB, RSM, KP); statistical analysis (PK); administrative, technical, or logistic support (WTC, PK, RSM, KP); and supervision (WTC, PK).

: William T. Cecil, MBA, BlueCross BlueShield of Tennessee, 801 Pine St, 1E, Chattanooga, TN 37402. E-mail: 

. Transluminal dilation of coronary-artery stenosis. 

. Percutaneous coronary interventions (previously called angioplasty, percutaneous transluminal coronary [PTCA], or balloon angioplasty). 

http://www.americanheart.org/presenter.jhtml?identifier=4454

5. Office of Technology Assessment, US Congress

Tools for Evaluating Health Technologies: Five Background Papers, BPâ€“Iâ€“I-142

. Washington, DC: US Government Printing Office; February 1995.

Percutaneous transluminal coronary angioplasty. In: Strategies for Medical Technology Assessment

. Washington, DC: Office of Technology Assessment, US Congress. September 1982:168-174.

. National Hospital Discharge Survey: 2000 annual summary with detailed diagnosis and procedure data. 

. National Hospital Discharge Survey: 2002 annual summary with detailed diagnosis and procedure data. 

. National Hospital Discharge Survey: 2004 annual summary with detailed diagnosis and procedure data. 

. Studies say newer stents for arteries show promise. 

http://www.nytimes.com/2007/09/05/business/05stent.html?_r=1&scp=1&sq=Studies%20say%20newer%20stents%20for%20arteries%20show%20promise&st=cse&oref=slogin<http://www.nytimes.com/2007/09/05/business/05stent.html?_r=1&amp;scp=1&amp;sq=Studies%20say%20newer%20stents%20for%20arteries%20show%20promise&amp;st=cse&amp;oref=slogin

19. Bucher HC, Hengstler P, Schindler C, Guyatt GH

. Percutaneous transluminal coronary angioplasty versus medical treatment for nonacute coronary heart disease: meta-analysis of randomized controlled trials. 

21. Boden WE, Oâ€™Rourke RA, Teo KK, et al; COURAGE Trial Research Group

. Optimal medical therapy with or without PCI for stable coronary disease. 

. Doctors, insurers may differ on stents. Tennessean. April 2, 2007.

Systematic Reviews in Health Care: Meta-analysis in Context

Meta-analysis of Controlled Clinical Trials

. New York, NY: John Wiley & Sons Inc; 2002.

29. Egger M, Davey Smith G, Schneider M, Minder C

Meta-analyses of percutaneous coronary interventions in stable coronary artery disease are updated to include 2 recent large randomized controlled trials.

August 2008

Participant Disclosure (Am J Manag Care. 2008;14[suppl 8]:S241-S268)

Agreement Between Patient-reported Symptoms and Their Documentation in the Medical Record

A Meta-analysis Update: Percutaneous Coronary Interventions

Managed Care Enrollment and Chronically Disabled Women With Breast Cancer

Medicare HMO Impact on Utilization at the End of Life

Strategies for Identifying and Channeling Patients for Depression Care Management

Flunking Asthma? When HEDIS Takes the ACT