John C. Butler, MD

To identify the outcomes desired by patients (and their family members) with abdominal or back pain and to compare patient and physician opinions regarding the importance of each outcome.

After identifying 21 potentially important outcomes from the lit&shy;erature and telephone interviews with patients and family members, we asked 40 patients, 11 family members, and 11 primary care physicians in telephone interviews to rate the importance of each outcome to patients on a scale of 1 to 5 scale (5 = most important), stratified by pain location.

Mean patient ratings of the 21 outcomes ranged from 3.3 to 5, with the average rating across all items higher for patients with back pain than those with abdominal pain (4.50 vs 4.09; 

= .049). Physicians rated the importance of these outcomes to patients significantly lower than the patients did for both abdominal pain (4.1 vs 3.5; 

= .0003). Family member ratings were similar to those of the patients (4.3 vs 4.2; 

= .8), where&shy;as physicians rated the importance to patients to be an average of 0.6 points lower than the ratings of patients for abdominal pain and 0.8 points lower for back pain.

Many outcomes are important to patients and their family mem&shy;bers, but they mostly represent quality-of-life events rather than the symptom and function measures heretofore focused on by researchers. Physicians appear to rate most of these outcomes somewhat lower in importance.

Patient outcomes and their measurement for comparative effectiveness research, performance measurement, and patient care are increasingly important. Early measures of patient outcomes were usually selected from the perspective of a clinician, but there was a shift to a more patient-centric perspec&shy;tive 15 years ago as measures of health status and quality of life.

 By 2010, patient outcomes had attained such priority nationwide that the Patient-Centered Outcomes Research Institute (PCORI) was established to emphasize incorporating patient viewpoints and outcomes in every aspect of healthcare research.

This large shift has required the development of patient-re&shy;ported outcomes measures (PROMs). Subsequently, the National Institutes of Health established the Patient-Reported Outcomes Measurement Information System (PROMIS) in 2004 with the goal of “providing clinicians and researchers access to efficient, precise, valid, and responsive…measures of health and well-be&shy;ing.”

 Although this system was primarily created for clinical re&shy;search, it is rapidly becoming the standard source of PROMs for all purposes. The PROMIS Assessment Center currently contains measure sets that are publicly available in 19 domains for assessing the physical, mental, and social health of both adults and children.

However, the PROMIS tools largely measure specific patient functions and symptoms and seem to assume that such specific measures are the only outcomes that matter to patients and their families. The PROMIS website does not describe any involve&shy;ment of patients in the selection of its outcome measures, and no publications describe attempts to learn from patients what outcomes they care most about. Although patients probably do care about relief from symptoms and improvement in function, there may also be other outcomes that are equal or more import&shy;ant to them.

Therefore, we developed this study to discover what outcomes patients and their family members cared about among those who had experienced back or abdominal pain problems that were se&shy;rious enough to require advanced diagnostic imaging studies. We focused on patients with actual experiences, thinking they should be most aware of outcomes important to them. After identifying potentially important outcomes through preliminary open-ended interviews, we surveyed patients, their family members, and pri&shy;mary care physicians to learn their rating of the importance of each outcome and then compared these ratings among patients, family members, and physicians.

We conducted this study among the patients receiving care from an 800-physician multi-specialty medical group in the Minneap&shy;olis-St. Paul metropolitan area. In order to facilitate access to health plan claims data for these patients for a later phase of the study, patient recruitment was limited to the 60% with insurance from the affiliated health plan; this population includes patients on prepaid medical assistance and a racial and ethnic profile sim&shy;ilar to that of the Twin Cities metropolitan area.

In order to create a list of potentially important patient outcomes, we randomly identified adults who had undergone either a mag&shy;netic resonance imaging (MRI) or computed tomography (CT) scan for abdominal or back pain about 1 year prior. We first sent these individuals a letter explaining the study and providing an opportunity to opt out of further recruitment contacts. If they did not opt out, trained interviewers called them to assess their willingness to participate in a telephone interview, during which they were asked about all of the outcomes they had wanted from the care of their pain problem. Interviewers next read them a list of other possible outcomes that had been generated by the re&shy;search team (which included a patient co-investigator) and asked them to rate the importance of each outcome on a scale of 1 to 5 (5 = extremely important, 1 = not at all important). Finally, they were asked again about any other outcomes they might have thought of. We gave completed interviewees a $40 gift card—this incentive was mentioned in both the pre-notification letter and introductory script. Out of 7 patients contacted within 3 tele&shy;phone attempts, 6 agreed to participate and completed the inter&shy;view. Two of those 6 said there was a family member who was very familiar with their pain problem and would be willing to be contacted, so we also recruited them and completed interviews. The responses from these 8 people were so consistent that no further pilot interviews seemed needed.

Following the pilot, in hopes of completing 40 patient telephone interviews, we identified an additional random pool of 83 adult patients who had experienced a first CT or MRI scan of the back or abdomen for pain about 1 year prior. We followed the same protocol used in the pilot; the interview script—revised after the pilot calls&mdash;confirmed eligibility, obtained demographic infor&shy;mation, and asked about the most important outcomes desired from the medical care of their problem. Then we asked subjects to rate a revised list of 21 outcomes (see 

for the list in the order in which items were asked) on an importance scale of 1 to 5 (again, 5 = extremely important, 1 = not at all important). Finally, we asked them whether there was a family member who was familiar with their problem who might be willing to complete a similar interview. If so, we made contact arrangements and the family member interview followed the same process. Interviews lasted an average of 15 minutes and all respondents were sent a $40 gift card.

We identified and recruited practicing adult primary care physi&shy;cians through the help of a co-investigator (CV) at the Institute for Clinical Systems Improvement for interviews by our practic&shy;ing primary care physician co-investigator (JB). Because the phy&shy;sicians had not had a specific personal experience to reference, we provided them with common case scenarios for patients with back or abdominal pain as context for their answers. We asked what outcomes they thought would be most important to the scenario patients; then they were asked to rate each of the 21 out&shy;comes (see Table 1) identified from patients on the same 1-to-5 scale from the perspective of their patients. They were also asked about the most important outcomes from their perspective as a clinician.

Associations between type of pain and patient attributes were tested using contingency tables and Pearson’s χ

 and Fisher’s ex&shy;act tests. Independent samples 

tests were used to test differenc&shy;es in outcome importance ratings between abdominal and back pain patients, as well as differences in outcome importance rat&shy;ings between patients and physicians. The study has 80% power (α = 0.05, 2-sided test) to detect a difference of 0.9 standard deviations (SDs) in the mean outcome rating from abdominal and back pain patients. An α of 0.05 was considered statistically significant, 2-tailed tests were used, and there was no correction for multiple testing. Analyses were conducted using SAS version 9.1 (SAS Institute Inc, Cary, North Carolina).

Of 83 patients with call attempts, we could not reach 33, and 1 of those contacted was ineligible as they reported not having the scan (see 

). Of the 49 eligible patients who could be contacted, 41 completed interviews—but the data from 1 was inadvertently lost&mdash;for a participation rate of 84% of eligible pa&shy;tients contacted, or 49% of all those attempted. Twelve patients nominated a family member who had been actively involved in the care of their pain problem for interviews, and 11 of those 12 completed them. Responding patients were similar to nonre&shy;sponders by age, race, Medicare insurance, pain type, and scan type. The major difference was that responders were much more likely to be females (92.5% vs 46.2%; 

<.001) and to be on Med&shy;icaid (7.5% vs 0%; 

The characteristics of participants are listed by pain type in 

. The majority were female and white. Aside from the type of scan, none of the other characteristics differed signifi&shy;cantly by pain type. Ten of the family members were spouses and 1 was a daughter. Family member characteristics were similar to those of the patients except that family members were more likely to be male (7 of 11 [63%]) and married (10 of 11 [91%]) than patients as a whole.

Seventeen primary care physicians were identified as potential participants in the second-stage interview, but 4 of them referred us to colleagues who agreed to participate and completed inter&shy;views along with 7 others. Each physician was from a different clinic: 4 were from the Twin Cities metro area and 7 were from the rest of the state; 2 were internists and 9 were family physi&shy;cians; 3 were female and 8 male; and 6 had been in practice for 26 to 43 years, 2 for 12 to 15 years, and 3 for 4 to 6 years.

In the initial open-ended phase of the pilot patient inter&shy;views, the main outcome identified by most patients and family members was to learn the cause of the pain. Seven of the other outcomes on the list were also mentioned spontaneously before respondents were given the list to rate. Only 1 (“minimize dis&shy;comfort from the tests used to assess the pain”) was raised that was not on the list.

contains the mean importance ratings by patients for the 21 outcomes and the difference between those ratings by pa&shy;tients with abdominal pain versus those with back pain. There was considerable overlap between the 2 groups, although patients with abdominal pain gave lower importance ratings for nearly all outcomes than did patients with back pain; the overall mean rat&shy;ing was 0.4 points lower than those with back pain (4.09 vs 4.50; 

= .049). Higher ratings were given by patients with abdomi&shy;nal pain for only 2 outcomes: to find the cause and to get rapid and complete relief. The overall mean rating of the 21 outcomes from family and patients was similar (4.32 for family, 4.28 for pa&shy;tients; 

= .86). Most (8 of 11) family members were connected with patients who had experienced abdominal pain. Importance ratings by family members of abdominal pain patients were the same or directionally higher than those of abdominal pain pa&shy;tients for 16 of 21 outcomes, and there were no statistically sig&shy;nificant differences.

provide graphic comparisons between patient and physician importance ratings for each of the outcomes, or&shy;dered by the magnitude of the difference between them. For ab&shy;dominal pain, physicians rated the importance to patients to be an average of 0.6 points lower than the ratings of patients (3.53 vs 4.09; 

= .04); for back pain, that difference was 0.85 points lower (3.65 vs 4.50; 

= .0003). Each of the 4 individual out&shy;comes with statistically significantly lower physician ratings for abdominal pain was also significantly lower for back pain. There were also another 9 of the outcomes for back pain with statis&shy;tically significant physician-patient differences. Moreover, all of the items with 

<.01 or less were in the back pain list. The out&shy;comes with greatest agreement between patients and physicians were finding the cause, getting rapid and complete relief, and re&shy;turning to work and productivity as soon as possible. For most outcomes, the 3 younger physicians provided lower ratings than the 8 with longer practice experience. Patient ratings of abdomi&shy;nal pain outcomes tended to show more variability than physician ratings, with 15 of the 21 SDs for abdominal pain outcomes be&shy;ing higher for patients than physicians. However, for back pain, the pattern was reversed, with 7 of the 21 SDs for back pain outcomes being higher for patients than physicians.

When the physicians were asked about the most important outcomes from their perspective, 5 highlighted making a diagno&shy;sis, 6 thought it was needed to guide treatment, and 4 wanted to rule out something serious. Four physicians also noted a reduc&shy;tion in symptoms and 3 identified restoration of normal function as soon as possible. Single physicians mentioned humane care, prognosis, and good access to care.

These patients and their families rated a wide variety of out&shy;comes from their care as important. Although there were differ&shy;ences between the average importance ratings by type of pain, the differences were mostly small and all of the outcomes we asked about had relatively high ratings that were similar between patients and family members. Finally, the primary care physician interviewees tended to rate each of these outcomes as having lower importance than did patients or their families, even though the physicians were asked to rate the importance of each out&shy;come from the patients’ perspective rather than from their own. The similar shape of the importance difference curves in Figures 2 and 3 across all of these outcomes suggests that these dif&shy;ferences are real. Such differences highlight the importance of asking patients—both individually and collectively&mdash;about what outcomes are important to them.

The results of this study raise important questions about what approach should be used to identify PROMs. The current fo&shy;cus on using measures that have been identified by researchers and those limited to specific functions and symptoms will likely not capture the same outcomes that patients themselves might identify as important. McClimans, in his study, has objected to the whole idea of standardized questions, instead proposing a theoretical framework for thinking about PROMs that under&shy;stands them “as posing genuine questions to patients—questions that are open to reinterpretation [and context].”

 He particularly objects to measures that break down the outcome into a series of sub-questions and that assume their sum fully captures the overall outcome. McKenna raises a similar concern in his study, while emphasizing the importance of a hard science approach to measure development.

Only one of the outcomes that these patients thought were im&shy;portant (ie, to get rapid and complete relief from pain and other symptoms) would be partially measurable by the PROMIS mea&shy;surement sets established by the National Institutes of Health to measure patient outcomes.

 PROMIS has developed question&shy;naires to ask about pain, fatigue, gastrointestinal symptoms, arm and leg function, sexual function, sleep, physical activity, affect, cognition, self-efficacy, substance use, social support, peer and family relationships, and social roles and activities. There are also global health status questions.

 The individual question sets (short forms) for each of these domains varies from 4 to 10 questions selected from banked sets that contain as many as 121 questions. Their psychometric qualities are mostly well stud&shy;ied, and they are a wonderful resource for researchers, but there has been no patient involvement in selecting or reviewing these measures.

The other main source for patient outcome measures is the National Quality Forum. It has endorsed nearly 700 measures created by other organizations, 208 of which are identified as outcome measures.

 Although most of the latter are clinical out&shy;comes related to specific conditions, some do address patient function, usually in relation to a specific procedure—few are as generic as the outcomes considered important by our patient subjects. A similar resource in Britain, the Health and Social Care Information Centre, began work in 2009 on PROMs and current&shy;ly has measurement sets in place for hip and knee replacement, hernia repair, and varicose vein treatment&mdash;again, mostly clinical and specific for a procedure and condition.

 There is no indica&shy;tion on its website that patients had any role in identifying these measures.

PCORI has prioritized developing patient-centered outcome measures and is becoming a resource for PROMs. It is devel&shy;oping standards for such measures, but so far has not created or endorsed any specific measures. Its Methodology Committee has recommended that patients be included in the peer review process for grant proposals, and it has a standard requiring that researchers “engage people representing the population of inter&shy;est and other relevant stakeholders in ways that are appropriate and necessary in a given research context.”

Developing outcome measures that are meaningful to patients is a new endeavor for healthcare, and accordingly, it will take some time to get it right. Once we know what patients want mea&shy;sured, learning where to most efficiently and accurately obtain that information is another challenge. Although it might seem obvious that the best source of data is directly from patients, this may not always be feasible because of the cost and difficulty tracking and obtaining responses. Thus, we need to learn whether some of those outcomes can be obtained from medical records or insurance claims data. Yet another challenge will be learn&shy;ing how to incorporate those measures into the daily practice of medicine. Black suggests in his study that outcome measures could be transformative for healthcare, but admits that their use in routine practice is still uncommon, and identifies a series of challenges.

 As if to demonstrate that, his group recently pub&shy;lished a study using PROMs to compare surgeons and conclud&shy;ed that the choice of outcome measure can substantially alter a surgeon’s rating.

 Valderas et al performed a systematic review of the literature on the impact of measuring PROMs in clinical practice.

 They confirmed that “contexts and interventions that will yield important benefits remain to be clearly defined.”

Our findings in this study are limited to patients with 2 types of clinical situations: back pain that tends to be chronic or recur&shy;rent and abdominal pain that tends to be acute. The underlying disease processes and patient impacts of these 2 situations are both heterogeneous and different, which highlights how import&shy;ant it is to survey patients across a broad range of contexts if the goal is to fully understand patient preferences. The advantage of surveying patients as we did was that having their own recent experience with these potentially serious and disabling problems should have made their responses less abstract. All of the sample sizes in this study were small, and the subjects were not especially diverse or representative nationally, so these findings require con&shy;firmation in larger more representative, diverse samples. It is also likely that we have not identified all of the important outcomes.

Nevertheless, the findings are especially important in suggest&shy;ing that there are a wide variety of outcomes that are important to patients and their families, outcomes that are not necessarily limited to relief of particular symptoms or recovery of certain functions. Thus far, most of the attention to PROMs has focused on the psychometric and feasibility aspects of measures that are limited to specific symptoms and functions. It might be wise to first obtain more input from patients before once again learning that the outcomes we measure so well have little relevance to what patients want to know before they make decisions about their care.

The authors are grateful to the skilled interviewers who conduct&shy;ed all of these interviews so well, and to Cally Vinz, vice president at Institute for Clinical Systems Improvement, who has been an active participant in our meetings and helped identify and recruit physicians for interviews. The authors are also extremely grate&shy;ful to the patients, family members, and physicians who have provided them with so much helpful input; this includes Aimee Vang, Carol Dexter, Katharine Pittman, Jamie Gudknecht, Jen&shy;ny Zakoski, David Freedland, and Gail Soens from the Patient/ Family Advisory Board, who have been provided with a modest honorarium for their time. As advisers, they have each provided very helpful feedback on our survey protocol and pilot tested and provided suggestions to improve the survey described in this paper. In addition, John Wilkinson, MD; Ian Kenning, MD; and Deepti Pandita, MD, from the Physician Advisory Board have reviewed the authors’ plans and findings and provided practicing physician perspectives that have helped us to remain focused on that part of their targeted audience, even though we were unable to provide them with any recompense.

The funding agency, PCORI, had no role in the design and conduct of the study; collection, management, analysis, and in&shy;terpretation of the data; and preparation, review, or approval of the manuscript project.

: HealthPartners Institute for Education and Research (LIS, SEA, JOT, RRW, JYZ), Minneapolis, MN; Health&shy;Partners Medical Group (JB), Minneapolis, MN; Group Health Research Institute (DC), Seattle, WA; University of Washington (JGJ), Seattle, WA; University of California (RS-B), San Francis&shy;co, CA; Saint Paul, MN (CN).

: This work was supported through a Patient-Cen&shy;tered Outcomes Research Institute (PCORI) Pilot Project Pro&shy;gram Award. All statements in this report, including its findings and conclusions, are solely those of the authors and do not nec&shy;essarily represent the views of PCORI, its Board of Governors, or Methodology Committee.

: Dr Solberg is a member of the ICSI regional board (Q1 collaborative), an employee of HealthPartners health system, and has past grants received from CMS, National Insti&shy;tute of Health, AHRQ, and PCORI. Dr Jarvik is a member of the General Electric-Association of University Radiologists Re&shy;search Academy Fellowship Advisory Board, has received a grant from PhysioSonics, an ultrasound-based diagnostic company, and also receives royalties from PhysioSonics. Dr Smith-Bindman has previously received a grant from PCORI for an unrelated study. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

: Concept and design (LIS, CKN, RS-B, JB, RRW); acquisition of data (LIS, DSC, JB, JOT, RRW, JYZ); analysis and interpretation of data (LIS, SEA, DSC, CKN, JGJ, RS-B, JB, JOT, RRW, JYZ); drafting of the manuscript (LIS, SEA, RS-B, JB); critical revision of the manuscript for important intel&shy;lectual content (LIS, SEA, DSC, CKN, JGJ, RS-B, JB, JOT, RRW, JYZ); statistical analysis (SEA, RRW); obtaining funding (LIS, RS-B); supervision (JOT); and providing content expertise on back pain (JGJ).

: Leif I. Solberg, MD, HealthPartners In&shy;stitute for Education and Research, PO Box 1524, MS#21111R, Minneapolis, MN 55440-1524. E-mail leif.i.solberg@healthpart&shy;ners.com.

1. Wu AW, Kharrazi H, Boulware LE, Snyder CF. Measure once, cut twice—adding patient-reported outcome measures to the electronic health record for comparative effectiveness research. 

2. Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Insti&shy;tute (PCORI) national priorities for research and initial research agenda. 

3. Cella D, Riley W, Stone A, et al; PROMIS Cooperative Group. The Patient-Re&shy;ported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. 

4. McClimans L. A theoretical framework for patient-reported outcome measures. 

5. McKenna SP. Measuring patient-reported outcomes: moving beyond misplaced common sense to hard science. 

6. Instruments available for use in Assessment Center. Assessment Center website. https://www.assessmentcenter.net/documents/InstrumentLibrary.pdf. Updated November 10, 2015. Accessed November 2015.

7. Hays RD, Bjorner JB, Revicki DA, Spritzer KL, Cella D. Development of physical and mental health summary scores from the patient-reported outcomes measure&shy;ment information system (PROMIS) global items. 

8. Revicki DA, Kawata AK, Harnam N, Chen WH, Hays RD, Cella D. Predicting EuroQol (EQ-5D) scores from the patient-reported outcomes measurement infor&shy;mation system (PROMIS) global items and domain item banks in a United States sample. 

9. Measures, reports & tools. National Quality Forum website. http://www.quality&shy;forum.org/Measures_Reports_Tools.aspx. Accessed November 17, 2015.

Articles

Patients with abdominal or back pain identified 21 outcomes important to them, but the reported outcomes are quite different from the symptom and function outcomes studied by researchers.

It Is Time to Ask Patients What Outcomes Are Important to Them

To evaluate the effects of providing appropriateness criteria through guideline-based electronic health record (EHR) decision support for high-tech diagnostic imaging (HTDI) procedures.

Chart audits were performed on a random sample of adult primary care orders for 3 HTDI procedures (computed tomography [CT] and magnetic resonance imaging [MRI] of the head, and MRI of the lumbar spine) before and after implementation of an EHR decision support system. 

Level of appropriateness, abnormal findings, and apparent effects on patient care.

A total of 299 eligible audits were performed. Decision support was associated with a 20% to 36% drop in spine MRI and head CT orders, but head MRI order volume was unaffected. Combined results for the 3 procedures showed that a larger proportion of studies ordered after implementing decision support (89.2% vs 79.5%, 

 = .02) fit appropriateness criteria, and more postimplementation studies had A ratings (highest utility rating) (81.8% vs 70%, 

 = .04). However, there were no differences in the proportion of tests with positive findings (42/132 vs 28/120, 

criteria) or the proportion with a likely impact on patients (6.6% vs 10.8%,

 These data support the feasibility of using chart audits to assess the relationship between appropriateness criteria and HTDI orders. Although introduction of EHR clinical decision support for diagnostic imaging orders was associated with reduced volume and increased appropriateness of orders, there was little apparent impact on either findings or patients.

Implementation of decision support in the electronic medical record system of a large medical group reduced the volume and increased the appropriateness of high-tech diagnostic imaging orders for head and spine scans, although the effects varied by procedure.

There was no increase in the proportion of scans with positive findings.

Few scans before or after implementation appeared likely to have affected patient outcomes.

Scans were as likely to affect the patient care process (usually by adding services) as to improve patient outcomes.

Many policymakers suggest that conversion to electronic health records (EHRs) will improve the quality and cost of medical care.

 Others argue that such benefits will require that EHRs include at least electronic decision support.

 However, there still is not much evidence about how much value electronic decision support adds. The systematic review of controlled trials of computerized decision support systems by Garg et al concluded that two-thirds of 97 trials showed some improvement in care processes, but only 7 of 52 showed improvement in patient outcomes.

One area in which electronic decision support is likely to be helpful is in high-tech diagnostic imaging (HTDI) test orders. These very costly tests have been increasing at a rapid rate nationally over the past 15 years.

 Published studies suggest that this increase is occurring among tests ordered by both radiologists and nonradiologists and that it is associated with wide geographic variation in rates. As a result, there is increasing pressure to encourage adherence to appropriateness criteria, a situation for which electronic decision support seems well suited.

Rosenthal et al reported preliminary results from the implementation of an electronic decision support system for HTDI procedures at Massachusetts General Hospital, showing that it was acceptable to physicians and appeared to change ordering habits.

 However, this conclusion was based on changes in concordance with appropriateness criteria in the decision support system over time, with no data comparing such concordance before and after the system began. Moreover, there is little direct evidence on which to base appropriateness criteria. Thus, we know little about the effects on patient care processes or outcomes of applying these criteria. To better understand these issues, we studied the impact of implementing a decision support system for HTDI in the EHR of a large multispecialty medical group. The following questions were addressed: 

1. Was there sufficient information in the medical chart to permit measurement of appropriateness criteria for HTDI test orders?

2. Did compliance with appropriateness criteria change?

3. Did positive findings or effects on patients change?

This study was conducted within a 600-physician multispecialty medical group in the Twin Cities region of Minnesota that serves an active patient population of 400,000 patients. Since 2004, its primary care clinics have used the Epic paperless EHR that includes computerized orders for all tests and incorporates either electronic or scanned reports of those test results.

In February 2007, the medical group rapidly implemented a decision support system within Epic for HTDI tests. The system used appropriateness criteria from the American College of Radiology that have ratings of A (high utility), B (intermediate utility), and C (low utility). Because the EHR ordering system required that a reason be entered for all orders, and because these criteria could not be expected to cover all circumstances, an “other” alternative also was available, which collected unstructured information for about 15% (8%-19%) of orders for the procedures we studied. Only completed orders were recorded in the EHR.

The system did not prevent orders, regardless of criteria fit. Physicians received little feedback on their order results (although system use identified the appropriateness of each order). There were no financial incentives or disincentives associated with ordering patterns.

To avoid the preparation or acclimatization periods before or after implementation for this study, as well as to avoid any seasonal effects on procedure orders, we selected 2-month windows (6 months before and 6 months after the decision support system was implemented). We studied 3 of the more commonly ordered HTDI procedures: computed tomography (CT) and magnetic resonance imaging (MRI) of the head, and MRI of the lumbar spine, because these procedures have relatively fewer and less complex ordering reasons than many others.

 Our study was limited to original orders (ie, not follow-up of a previous test) by a primary care clinician for adults (age 19 years and over) in our primary care clinics to further simplify and standardize the analysis.

The implemented system contained 72 criteria for CT of the head and 68 for MRI of the head, but only 27 for MRI of the lumbar spine. “A” ratings were associated with 65% of these criteria for CT of the head, whereas 85% of the criteria for the other 2 procedures had A ratings. If the apparent reason in the chart for ordering a test did not resemble any of the existing criteria, we recorded this test as lacking a criterion fit.

Fifty cases of each completed procedure were drawn randomly during the selected months before and after implementation of decision support. They were excluded if the order had actually come indirectly from a specialty consultant, or was a follow-up to an earlier test.

Audits were performed by 2 experienced nurse auditors using predetermined definitions. The 2 physicians in our group (LIS, JB) separately reviewed every case with positive findings to confirm the audit results and decide on the patient impact, followed by discussion between them until agreement was reached. During the training period for the audit, all auditors reviewed the same 15 cases, and we as a group discussed the issues raised by the cases to attain a common approach to data collection. Blinding either nurse or physician reviewers to the dates of the procedure was impossible, because they had to search the EHR for information needed.

Results of a test were called positive when an abnormality either seemed to explain the patient’s problem or required a medical decision. The effect on care was categorized as 1 of the following:

• Probable impact on patient health or function 

• Impact on the care process (usually additional consultation or testing)

The analysis was limited to descriptive statistics, although there were enough cases with adequate information in the record so that there was sufficient power to detect large differences, especially when we combined the cases for all 3 procedures. To evaluate the association between the pre- and postimplementation periods for the fit with appropriateness criteria, compliance rating, tests with positive findings, test results within the normal range, and test results that led to changes in care decisions, we performed 2-sided Fisher exact tests.

During the 2-month study period in 2007 compared with 2006, the volume of completed test orders for these 3 procedures dropped for head CT and spine MRI by 36.5% and 20%, respectively. However, completed test orders for head MRI increased by 3.3%. A total of 38 cases were excluded, leaving 151 cases before and 148 cases after implementation. Sixty-two percent of the cases involved women and the mean age was 57.8 years, with no pre/post differences. Only 6% of the procedures were ordered by nurse practitioners.

 shows that the great majority of orders could be definitely rated by the criteria in both the pre—decision support and post–decision support periods, but only MRI of the head and spine showed an increase in definitely meeting appropriateness criteria after decision support was present. Only MRI of the head showed an increase in the proportion of orders rated as A in the postperiod. More than 90% of the head CTs were ordered for headache, trauma, or a neurologic deficit, and nearly all the spine MRIs were ordered for back pain or radiculopathy. Head MRIs also were ordered for headache or neurologic deficits, but many were ordered for evaluation of vertigo or other conditions.

 shows that neither of the head procedures had many positive findings, although MRI of the head did have an increase in positive findings in the postperiod, as well as an increase in the proportion clearly related to the specific indication for doing the test. Implementation of decision support did not change the very low frequency with which either type of head HTDI test impacted patients (2.5% of the time), but the greater impact of spine MRI increased even more in the postperiod (from 14% to 30%, 

 = .18). Except for MRI of the spine in the postperiod, all audits revealed a high proportion of incidental findings. Spine MRI orders that did not meet appropriateness criteria greatly increased the proportion of spine MRI results that had an impact on patients (from 7% to 62%) in the postperiod. Head HTDI orders that did not meet appropriateness criteria had no impact on patients in either the pre- or postperiod (data not shown in the table).

When we combined all 3 procedures, 10% more cases had a definite rating of appropriateness after implementation and 12% more cases with a definite rating of appropriateness had an A rating. No other differences were significant between the 2 time periods, and there was no evidence of an effect on positive findings or patients.

This study showed that it is possible to use chart audits to assess the fit between recorded case information and appropriateness criteria for most cases involving these 3 procedures. It also suggests that there may have been a small increase in definite fit with the criteria and an increase in the proportion with an A rating, at least for some procedures. However, none of these increases were large or homogeneous across procedures, and for only 1 procedure (MRI of the head) was this better fit associated with an increase in the proportion of positive findings. Impact on patients was very limited for head studies, but larger for spine MRI, especially in the postperiod, suggesting there may be problems with the validity of the appropriateness criteria.

The number of HTDI studies dropped substantially for CT of the head and MRI of the spine in association with the introduction of this decision support system, whereas the volume of completed head MRIs rose slightly. This finding may reflect ordering redirection from CT to MRI. In comparison, the total number of HTDI of all types declined in Minnesota by 1.5% during this time period (based on claims data from all Minnesota payers) after increasing by 8.3% per year over the 3 years before that.

The American College of Radiology first published its criteria in 1995, and Martin et al have shown that they could be applied to 76% of the imaging procedure requests made from a general internal medicine clinic.

 The problem with these criteria is that there are a very large number of tests and possible indications and there is relatively little evidence, so the criteria are based on the opinions of expert panels.

 The variable results of this study for different procedures and the lack of increased positive findings or effects on patients suggest the need for more valid criteria based on evidence from more research.

 at Massachusetts General Hospital and Sanders and Aronsky

 at Vanderbilt have shown that a computerized order entry with decision support can be widely accepted by clinicians and can affect ordering behavior. On the other hand, Freeborn et al found that simply distributing guidelines for lumbar spine imaging or providing feedback about ordering patterns had no effect on use rates.

 Hadley et al reviewed charts of trauma patients with imaging studies and found that (1) it was possible to apply American College of Radiology criteria to the chart information, and (2) if the criteria had been applied to these patients, there would have been a 39% reduction in imaging costs and a 44% reduction in radiation dose.

 Bradley et al found that preevaluation MRI of chronic shoulder pain patients did not appear to have any effect on the treatment or outcome.

 Kaups et al studied repeat head CT studies on patients with head injury and found that in the absence of clinical deterioration, these studies did not alter management.

It remains unclear whether a new requirement for fit with criteria that results in fewer imaging studies differentially reduces unnecessary tests. Although we found that relatively few of these orders had positive findings or impact on patients, further studies on larger and more varied orders with a control or comparison group are needed to verify these results and to assess the overall impact on patient care and costs.

This quasi-experimental study clearly has limitations. These include relatively small sample sizes from a single medical group and audits based on relatively subjective judgments. We also have no information about possible changes in the volume of patients requiring testing or changes in indications. However, our study suggests that similar studies can be based on chart audits and that there may be considerable variability among procedures in the extent to which decision support affects either fit with appropriateness criteria or impact on ordering volume or patient outcomes. We also clearly need a better evidence base behind the development of appropriateness criteria, with more evidence for the specific value of various combinations of imaging tests, ordering criteria, and medical conditions.

From the HealthPartners Research Foundation (LIS, FW, JCB, KJP, CAV, MAM), HealthPartners Medical Group, Minneapolis, MN.

This study was supported by Partnership grant 07-070 from HealthPartners Research Foundation. It originated with the HTDI Steering Committee of the Institute for Clinical Systems Improvement and its constituent health plans and medical groups, which wanted to learn the effects of their new approach to improving the quality and cost of high-tech diagnostic imaging. The Steering Committee was responsible for coordinating the implementation

of electronic health record decision support among 5 large medical groups in the state, and it provided helpful input during our frequent updates on this study and its findings.

Ms Vinz reports receiving honoraria to speak and funds to attend conferences/meetings related to decision support initiatives. The other authors (LIS, FW, JCB, KJP, MAM) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Concept and design (LIS, FW, JCB, KJP, CAV, MAM); acquisition of data (LIS, JCB, KJP); analysis and interpretation of data (LIS, FW, JCB, KJP, CAV); drafting of the manuscript (LIS, FW, CAV); critical revision of the manuscript for important intellectual content (LIS, FW, CAV, MAM); statistical analysis (FW); obtaining funding (LIS); and administrative, technical, or logistic support (KJP, MAM).

Leif I. Solberg, MD, HealthPartners Research Foundation, HealthPartners Medical Group, PO Box 1524, MS #21111R, Minneapolis, MN 55440-1524. E-mail: leif.i.solberg@healthpartners.org.

1. Corrigan JM, Eden J, Smith BM, eds, for the Institute of Medicine. 

Leadership by Example: Coordinating Government Roles in Improving Health Care Quality. 

Washington, DC: National Academies Press; 2003.

2. Hillestad R, Bigelow J, Bower A, et al.

 Can electronic medical record systems transform health care? Potential health benefits, savings, and costs. 

3. Miller RH, West C, Brown TM, Sim I, Ganchoff C.

 The value of electronic health records in solo or small group practices. 

Electronic decision support for high-tech diagnostic imaging was associated with reduced volume and increased appropriateness, but had little impact on findings or patients.

John C. Butler, MD

Articles

It Is Time to Ask Patients What Outcomes Are Important to Them

Effects of Electronic Decision Support on High-Tech Diagnostic Imaging Orders and Patients