Leif I. Solberg, MD

To identify the outcomes desired by patients (and their family members) with abdominal or back pain and to compare patient and physician opinions regarding the importance of each outcome.

After identifying 21 potentially important outcomes from the lit&shy;erature and telephone interviews with patients and family members, we asked 40 patients, 11 family members, and 11 primary care physicians in telephone interviews to rate the importance of each outcome to patients on a scale of 1 to 5 scale (5 = most important), stratified by pain location.

Mean patient ratings of the 21 outcomes ranged from 3.3 to 5, with the average rating across all items higher for patients with back pain than those with abdominal pain (4.50 vs 4.09; 

= .049). Physicians rated the importance of these outcomes to patients significantly lower than the patients did for both abdominal pain (4.1 vs 3.5; 

= .0003). Family member ratings were similar to those of the patients (4.3 vs 4.2; 

= .8), where&shy;as physicians rated the importance to patients to be an average of 0.6 points lower than the ratings of patients for abdominal pain and 0.8 points lower for back pain.

Many outcomes are important to patients and their family mem&shy;bers, but they mostly represent quality-of-life events rather than the symptom and function measures heretofore focused on by researchers. Physicians appear to rate most of these outcomes somewhat lower in importance.

Patient outcomes and their measurement for comparative effectiveness research, performance measurement, and patient care are increasingly important. Early measures of patient outcomes were usually selected from the perspective of a clinician, but there was a shift to a more patient-centric perspec&shy;tive 15 years ago as measures of health status and quality of life.

 By 2010, patient outcomes had attained such priority nationwide that the Patient-Centered Outcomes Research Institute (PCORI) was established to emphasize incorporating patient viewpoints and outcomes in every aspect of healthcare research.

This large shift has required the development of patient-re&shy;ported outcomes measures (PROMs). Subsequently, the National Institutes of Health established the Patient-Reported Outcomes Measurement Information System (PROMIS) in 2004 with the goal of “providing clinicians and researchers access to efficient, precise, valid, and responsive…measures of health and well-be&shy;ing.”

 Although this system was primarily created for clinical re&shy;search, it is rapidly becoming the standard source of PROMs for all purposes. The PROMIS Assessment Center currently contains measure sets that are publicly available in 19 domains for assessing the physical, mental, and social health of both adults and children.

However, the PROMIS tools largely measure specific patient functions and symptoms and seem to assume that such specific measures are the only outcomes that matter to patients and their families. The PROMIS website does not describe any involve&shy;ment of patients in the selection of its outcome measures, and no publications describe attempts to learn from patients what outcomes they care most about. Although patients probably do care about relief from symptoms and improvement in function, there may also be other outcomes that are equal or more import&shy;ant to them.

Therefore, we developed this study to discover what outcomes patients and their family members cared about among those who had experienced back or abdominal pain problems that were se&shy;rious enough to require advanced diagnostic imaging studies. We focused on patients with actual experiences, thinking they should be most aware of outcomes important to them. After identifying potentially important outcomes through preliminary open-ended interviews, we surveyed patients, their family members, and pri&shy;mary care physicians to learn their rating of the importance of each outcome and then compared these ratings among patients, family members, and physicians.

We conducted this study among the patients receiving care from an 800-physician multi-specialty medical group in the Minneap&shy;olis-St. Paul metropolitan area. In order to facilitate access to health plan claims data for these patients for a later phase of the study, patient recruitment was limited to the 60% with insurance from the affiliated health plan; this population includes patients on prepaid medical assistance and a racial and ethnic profile sim&shy;ilar to that of the Twin Cities metropolitan area.

In order to create a list of potentially important patient outcomes, we randomly identified adults who had undergone either a mag&shy;netic resonance imaging (MRI) or computed tomography (CT) scan for abdominal or back pain about 1 year prior. We first sent these individuals a letter explaining the study and providing an opportunity to opt out of further recruitment contacts. If they did not opt out, trained interviewers called them to assess their willingness to participate in a telephone interview, during which they were asked about all of the outcomes they had wanted from the care of their pain problem. Interviewers next read them a list of other possible outcomes that had been generated by the re&shy;search team (which included a patient co-investigator) and asked them to rate the importance of each outcome on a scale of 1 to 5 (5 = extremely important, 1 = not at all important). Finally, they were asked again about any other outcomes they might have thought of. We gave completed interviewees a $40 gift card—this incentive was mentioned in both the pre-notification letter and introductory script. Out of 7 patients contacted within 3 tele&shy;phone attempts, 6 agreed to participate and completed the inter&shy;view. Two of those 6 said there was a family member who was very familiar with their pain problem and would be willing to be contacted, so we also recruited them and completed interviews. The responses from these 8 people were so consistent that no further pilot interviews seemed needed.

Following the pilot, in hopes of completing 40 patient telephone interviews, we identified an additional random pool of 83 adult patients who had experienced a first CT or MRI scan of the back or abdomen for pain about 1 year prior. We followed the same protocol used in the pilot; the interview script—revised after the pilot calls&mdash;confirmed eligibility, obtained demographic infor&shy;mation, and asked about the most important outcomes desired from the medical care of their problem. Then we asked subjects to rate a revised list of 21 outcomes (see 

for the list in the order in which items were asked) on an importance scale of 1 to 5 (again, 5 = extremely important, 1 = not at all important). Finally, we asked them whether there was a family member who was familiar with their problem who might be willing to complete a similar interview. If so, we made contact arrangements and the family member interview followed the same process. Interviews lasted an average of 15 minutes and all respondents were sent a $40 gift card.

We identified and recruited practicing adult primary care physi&shy;cians through the help of a co-investigator (CV) at the Institute for Clinical Systems Improvement for interviews by our practic&shy;ing primary care physician co-investigator (JB). Because the phy&shy;sicians had not had a specific personal experience to reference, we provided them with common case scenarios for patients with back or abdominal pain as context for their answers. We asked what outcomes they thought would be most important to the scenario patients; then they were asked to rate each of the 21 out&shy;comes (see Table 1) identified from patients on the same 1-to-5 scale from the perspective of their patients. They were also asked about the most important outcomes from their perspective as a clinician.

Associations between type of pain and patient attributes were tested using contingency tables and Pearson’s χ

 and Fisher’s ex&shy;act tests. Independent samples 

tests were used to test differenc&shy;es in outcome importance ratings between abdominal and back pain patients, as well as differences in outcome importance rat&shy;ings between patients and physicians. The study has 80% power (α = 0.05, 2-sided test) to detect a difference of 0.9 standard deviations (SDs) in the mean outcome rating from abdominal and back pain patients. An α of 0.05 was considered statistically significant, 2-tailed tests were used, and there was no correction for multiple testing. Analyses were conducted using SAS version 9.1 (SAS Institute Inc, Cary, North Carolina).

Of 83 patients with call attempts, we could not reach 33, and 1 of those contacted was ineligible as they reported not having the scan (see 

). Of the 49 eligible patients who could be contacted, 41 completed interviews—but the data from 1 was inadvertently lost&mdash;for a participation rate of 84% of eligible pa&shy;tients contacted, or 49% of all those attempted. Twelve patients nominated a family member who had been actively involved in the care of their pain problem for interviews, and 11 of those 12 completed them. Responding patients were similar to nonre&shy;sponders by age, race, Medicare insurance, pain type, and scan type. The major difference was that responders were much more likely to be females (92.5% vs 46.2%; 

<.001) and to be on Med&shy;icaid (7.5% vs 0%; 

The characteristics of participants are listed by pain type in 

. The majority were female and white. Aside from the type of scan, none of the other characteristics differed signifi&shy;cantly by pain type. Ten of the family members were spouses and 1 was a daughter. Family member characteristics were similar to those of the patients except that family members were more likely to be male (7 of 11 [63%]) and married (10 of 11 [91%]) than patients as a whole.

Seventeen primary care physicians were identified as potential participants in the second-stage interview, but 4 of them referred us to colleagues who agreed to participate and completed inter&shy;views along with 7 others. Each physician was from a different clinic: 4 were from the Twin Cities metro area and 7 were from the rest of the state; 2 were internists and 9 were family physi&shy;cians; 3 were female and 8 male; and 6 had been in practice for 26 to 43 years, 2 for 12 to 15 years, and 3 for 4 to 6 years.

In the initial open-ended phase of the pilot patient inter&shy;views, the main outcome identified by most patients and family members was to learn the cause of the pain. Seven of the other outcomes on the list were also mentioned spontaneously before respondents were given the list to rate. Only 1 (“minimize dis&shy;comfort from the tests used to assess the pain”) was raised that was not on the list.

contains the mean importance ratings by patients for the 21 outcomes and the difference between those ratings by pa&shy;tients with abdominal pain versus those with back pain. There was considerable overlap between the 2 groups, although patients with abdominal pain gave lower importance ratings for nearly all outcomes than did patients with back pain; the overall mean rat&shy;ing was 0.4 points lower than those with back pain (4.09 vs 4.50; 

= .049). Higher ratings were given by patients with abdomi&shy;nal pain for only 2 outcomes: to find the cause and to get rapid and complete relief. The overall mean rating of the 21 outcomes from family and patients was similar (4.32 for family, 4.28 for pa&shy;tients; 

= .86). Most (8 of 11) family members were connected with patients who had experienced abdominal pain. Importance ratings by family members of abdominal pain patients were the same or directionally higher than those of abdominal pain pa&shy;tients for 16 of 21 outcomes, and there were no statistically sig&shy;nificant differences.

provide graphic comparisons between patient and physician importance ratings for each of the outcomes, or&shy;dered by the magnitude of the difference between them. For ab&shy;dominal pain, physicians rated the importance to patients to be an average of 0.6 points lower than the ratings of patients (3.53 vs 4.09; 

= .04); for back pain, that difference was 0.85 points lower (3.65 vs 4.50; 

= .0003). Each of the 4 individual out&shy;comes with statistically significantly lower physician ratings for abdominal pain was also significantly lower for back pain. There were also another 9 of the outcomes for back pain with statis&shy;tically significant physician-patient differences. Moreover, all of the items with 

<.01 or less were in the back pain list. The out&shy;comes with greatest agreement between patients and physicians were finding the cause, getting rapid and complete relief, and re&shy;turning to work and productivity as soon as possible. For most outcomes, the 3 younger physicians provided lower ratings than the 8 with longer practice experience. Patient ratings of abdomi&shy;nal pain outcomes tended to show more variability than physician ratings, with 15 of the 21 SDs for abdominal pain outcomes be&shy;ing higher for patients than physicians. However, for back pain, the pattern was reversed, with 7 of the 21 SDs for back pain outcomes being higher for patients than physicians.

When the physicians were asked about the most important outcomes from their perspective, 5 highlighted making a diagno&shy;sis, 6 thought it was needed to guide treatment, and 4 wanted to rule out something serious. Four physicians also noted a reduc&shy;tion in symptoms and 3 identified restoration of normal function as soon as possible. Single physicians mentioned humane care, prognosis, and good access to care.

These patients and their families rated a wide variety of out&shy;comes from their care as important. Although there were differ&shy;ences between the average importance ratings by type of pain, the differences were mostly small and all of the outcomes we asked about had relatively high ratings that were similar between patients and family members. Finally, the primary care physician interviewees tended to rate each of these outcomes as having lower importance than did patients or their families, even though the physicians were asked to rate the importance of each out&shy;come from the patients’ perspective rather than from their own. The similar shape of the importance difference curves in Figures 2 and 3 across all of these outcomes suggests that these dif&shy;ferences are real. Such differences highlight the importance of asking patients—both individually and collectively&mdash;about what outcomes are important to them.

The results of this study raise important questions about what approach should be used to identify PROMs. The current fo&shy;cus on using measures that have been identified by researchers and those limited to specific functions and symptoms will likely not capture the same outcomes that patients themselves might identify as important. McClimans, in his study, has objected to the whole idea of standardized questions, instead proposing a theoretical framework for thinking about PROMs that under&shy;stands them “as posing genuine questions to patients—questions that are open to reinterpretation [and context].”

 He particularly objects to measures that break down the outcome into a series of sub-questions and that assume their sum fully captures the overall outcome. McKenna raises a similar concern in his study, while emphasizing the importance of a hard science approach to measure development.

Only one of the outcomes that these patients thought were im&shy;portant (ie, to get rapid and complete relief from pain and other symptoms) would be partially measurable by the PROMIS mea&shy;surement sets established by the National Institutes of Health to measure patient outcomes.

 PROMIS has developed question&shy;naires to ask about pain, fatigue, gastrointestinal symptoms, arm and leg function, sexual function, sleep, physical activity, affect, cognition, self-efficacy, substance use, social support, peer and family relationships, and social roles and activities. There are also global health status questions.

 The individual question sets (short forms) for each of these domains varies from 4 to 10 questions selected from banked sets that contain as many as 121 questions. Their psychometric qualities are mostly well stud&shy;ied, and they are a wonderful resource for researchers, but there has been no patient involvement in selecting or reviewing these measures.

The other main source for patient outcome measures is the National Quality Forum. It has endorsed nearly 700 measures created by other organizations, 208 of which are identified as outcome measures.

 Although most of the latter are clinical out&shy;comes related to specific conditions, some do address patient function, usually in relation to a specific procedure—few are as generic as the outcomes considered important by our patient subjects. A similar resource in Britain, the Health and Social Care Information Centre, began work in 2009 on PROMs and current&shy;ly has measurement sets in place for hip and knee replacement, hernia repair, and varicose vein treatment&mdash;again, mostly clinical and specific for a procedure and condition.

 There is no indica&shy;tion on its website that patients had any role in identifying these measures.

PCORI has prioritized developing patient-centered outcome measures and is becoming a resource for PROMs. It is devel&shy;oping standards for such measures, but so far has not created or endorsed any specific measures. Its Methodology Committee has recommended that patients be included in the peer review process for grant proposals, and it has a standard requiring that researchers “engage people representing the population of inter&shy;est and other relevant stakeholders in ways that are appropriate and necessary in a given research context.”

Developing outcome measures that are meaningful to patients is a new endeavor for healthcare, and accordingly, it will take some time to get it right. Once we know what patients want mea&shy;sured, learning where to most efficiently and accurately obtain that information is another challenge. Although it might seem obvious that the best source of data is directly from patients, this may not always be feasible because of the cost and difficulty tracking and obtaining responses. Thus, we need to learn whether some of those outcomes can be obtained from medical records or insurance claims data. Yet another challenge will be learn&shy;ing how to incorporate those measures into the daily practice of medicine. Black suggests in his study that outcome measures could be transformative for healthcare, but admits that their use in routine practice is still uncommon, and identifies a series of challenges.

 As if to demonstrate that, his group recently pub&shy;lished a study using PROMs to compare surgeons and conclud&shy;ed that the choice of outcome measure can substantially alter a surgeon’s rating.

 Valderas et al performed a systematic review of the literature on the impact of measuring PROMs in clinical practice.

 They confirmed that “contexts and interventions that will yield important benefits remain to be clearly defined.”

Our findings in this study are limited to patients with 2 types of clinical situations: back pain that tends to be chronic or recur&shy;rent and abdominal pain that tends to be acute. The underlying disease processes and patient impacts of these 2 situations are both heterogeneous and different, which highlights how import&shy;ant it is to survey patients across a broad range of contexts if the goal is to fully understand patient preferences. The advantage of surveying patients as we did was that having their own recent experience with these potentially serious and disabling problems should have made their responses less abstract. All of the sample sizes in this study were small, and the subjects were not especially diverse or representative nationally, so these findings require con&shy;firmation in larger more representative, diverse samples. It is also likely that we have not identified all of the important outcomes.

Nevertheless, the findings are especially important in suggest&shy;ing that there are a wide variety of outcomes that are important to patients and their families, outcomes that are not necessarily limited to relief of particular symptoms or recovery of certain functions. Thus far, most of the attention to PROMs has focused on the psychometric and feasibility aspects of measures that are limited to specific symptoms and functions. It might be wise to first obtain more input from patients before once again learning that the outcomes we measure so well have little relevance to what patients want to know before they make decisions about their care.

The authors are grateful to the skilled interviewers who conduct&shy;ed all of these interviews so well, and to Cally Vinz, vice president at Institute for Clinical Systems Improvement, who has been an active participant in our meetings and helped identify and recruit physicians for interviews. The authors are also extremely grate&shy;ful to the patients, family members, and physicians who have provided them with so much helpful input; this includes Aimee Vang, Carol Dexter, Katharine Pittman, Jamie Gudknecht, Jen&shy;ny Zakoski, David Freedland, and Gail Soens from the Patient/ Family Advisory Board, who have been provided with a modest honorarium for their time. As advisers, they have each provided very helpful feedback on our survey protocol and pilot tested and provided suggestions to improve the survey described in this paper. In addition, John Wilkinson, MD; Ian Kenning, MD; and Deepti Pandita, MD, from the Physician Advisory Board have reviewed the authors’ plans and findings and provided practicing physician perspectives that have helped us to remain focused on that part of their targeted audience, even though we were unable to provide them with any recompense.

The funding agency, PCORI, had no role in the design and conduct of the study; collection, management, analysis, and in&shy;terpretation of the data; and preparation, review, or approval of the manuscript project.

: HealthPartners Institute for Education and Research (LIS, SEA, JOT, RRW, JYZ), Minneapolis, MN; Health&shy;Partners Medical Group (JB), Minneapolis, MN; Group Health Research Institute (DC), Seattle, WA; University of Washington (JGJ), Seattle, WA; University of California (RS-B), San Francis&shy;co, CA; Saint Paul, MN (CN).

: This work was supported through a Patient-Cen&shy;tered Outcomes Research Institute (PCORI) Pilot Project Pro&shy;gram Award. All statements in this report, including its findings and conclusions, are solely those of the authors and do not nec&shy;essarily represent the views of PCORI, its Board of Governors, or Methodology Committee.

: Dr Solberg is a member of the ICSI regional board (Q1 collaborative), an employee of HealthPartners health system, and has past grants received from CMS, National Insti&shy;tute of Health, AHRQ, and PCORI. Dr Jarvik is a member of the General Electric-Association of University Radiologists Re&shy;search Academy Fellowship Advisory Board, has received a grant from PhysioSonics, an ultrasound-based diagnostic company, and also receives royalties from PhysioSonics. Dr Smith-Bindman has previously received a grant from PCORI for an unrelated study. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

: Concept and design (LIS, CKN, RS-B, JB, RRW); acquisition of data (LIS, DSC, JB, JOT, RRW, JYZ); analysis and interpretation of data (LIS, SEA, DSC, CKN, JGJ, RS-B, JB, JOT, RRW, JYZ); drafting of the manuscript (LIS, SEA, RS-B, JB); critical revision of the manuscript for important intel&shy;lectual content (LIS, SEA, DSC, CKN, JGJ, RS-B, JB, JOT, RRW, JYZ); statistical analysis (SEA, RRW); obtaining funding (LIS, RS-B); supervision (JOT); and providing content expertise on back pain (JGJ).

: Leif I. Solberg, MD, HealthPartners In&shy;stitute for Education and Research, PO Box 1524, MS#21111R, Minneapolis, MN 55440-1524. E-mail leif.i.solberg@healthpart&shy;ners.com.

1. Wu AW, Kharrazi H, Boulware LE, Snyder CF. Measure once, cut twice—adding patient-reported outcome measures to the electronic health record for comparative effectiveness research. 

2. Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Insti&shy;tute (PCORI) national priorities for research and initial research agenda. 

3. Cella D, Riley W, Stone A, et al; PROMIS Cooperative Group. The Patient-Re&shy;ported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. 

4. McClimans L. A theoretical framework for patient-reported outcome measures. 

5. McKenna SP. Measuring patient-reported outcomes: moving beyond misplaced common sense to hard science. 

6. Instruments available for use in Assessment Center. Assessment Center website. https://www.assessmentcenter.net/documents/InstrumentLibrary.pdf. Updated November 10, 2015. Accessed November 2015.

7. Hays RD, Bjorner JB, Revicki DA, Spritzer KL, Cella D. Development of physical and mental health summary scores from the patient-reported outcomes measure&shy;ment information system (PROMIS) global items. 

8. Revicki DA, Kawata AK, Harnam N, Chen WH, Hays RD, Cella D. Predicting EuroQol (EQ-5D) scores from the patient-reported outcomes measurement infor&shy;mation system (PROMIS) global items and domain item banks in a United States sample. 

9. Measures, reports & tools. National Quality Forum website. http://www.quality&shy;forum.org/Measures_Reports_Tools.aspx. Accessed November 17, 2015.

Articles

Patients with abdominal or back pain identified 21 outcomes important to them, but the reported outcomes are quite different from the symptom and function outcomes studied by researchers.

It Is Time to Ask Patients What Outcomes Are Important to Them

 To assess the association between degree of change in medical home transformation and the satisfaction of patients, physicians, and other staff with the experience.

Cross-sectional surveys of lead physicians and patients.

 Lead physicians in the first 108 primary care clinics in Minnesota certified as patient-centered medical homes (PCMHs) were asked about the presence and change over the past 3 years of medical home-related practice systems, as well as the job satisfaction of their physicians and staff. Patients in 54 of these clinics were surveyed using the CG-CAHPS (Clinician-Group Consumer Assessment of Healthcare Providers and Systems) questions about their experience.

 The extent of change in systems over time was significantly correlated with reported positive changes in physician (0.29, P = .002) and staff (0.27, P = .005) job satisfaction, but not with the number of systems. System change was negatively correlated with patient reports of access to care (—0.43, P = .001) but unrelated to their experience with physicians and staff.

 These results add to a minimal literature on these important topics by suggesting improved physician and staff satisfaction, while highlighting the importance of the amount of change on both their satisfaction and that of patients. There may be a need to be particularly careful that medical home changes do not cause deterioration in patient access.While there is currently a rush to encourage and facilitate the transformation of primary care practices into patient-centered medical homes (PCMHs), surprisingly little is known about the impact of such a change on the experiences and satisfaction of either patients or healthcare personnel. In 2010, Hoff described the foundation for the PCMH as shaky, primarily because so little was known about whether either patients or physicians find it attractive.

 In 2012, he published a systematic review of studies through 2010, finding only a few studies of patient or clinician satisfaction; their results presented a mixed picture.

 The main positive results were from studies in Group Health Puget Sound, atypical because of salaried physicians and a common budget between health plan and medical group.

 Evaluation of the 36 diverse family practices in the National Demonstration Project was especially troubling, since after 2 years of work on transformation, patients in these practices reported a worse experience with care than at baseline.

 Since then, studies of Canadian or safety-net clinics have found positive results for patient or clinician satisfaction, but not since 2010 has this topic been addressed regarding typical United States practices that are becoming PCMHs.

This topic is very important, both because the primary care physicians who are increasingly seen as central to healthcare reform are generally discouraged with their work life and because a major goal of that reform is to improve the experience of patients. Shanafelt et al recently reported that burnout is more common among physicians than among other US workers, and physicians in specialties at the front lines seem to be at greatest risk.

 Linzer et al surveyed physicians in 119 New York and Upper Midwest primary care practices and found that half said their work pace was chaotic, three-fourths felt they had a low degree of control over their work, and a quarter reported being burned out.

 International comparisons of patient satisfaction are complex, but a Commonwealth Fund survey of patients in 6 developed nations found the United States to be in fifth place for the kinds of patient-centered care that, when successful, are associated with satisfaction.

As part of our larger evaluation of 132 primary care practices in Minnesota that had achieved state certification as “healthcare homes,” we had an opportunity to survey clinic leaders and to obtain patient satisfaction data from many of these clinics.

 We used those data to address the question of whether the satisfaction of patients, physicians, or clinic staff was associated with the degree of transformation in their clinics. Our hypothesis was that the satisfaction of each of these groups would be improved by the transformation.

In 2010, Minnesota established standards and a process for certification of primary care practices as medical homes (here called healthcare homes) as well as an associated payment system change for patients covered by state programs, all intended to encourage primary care transformation. As of May 2013, 242 of the 728 primary care practices in Minnesota (33%) had been certified as healthcare homes.

In order for clinics to become certified, they had to demonstrate compliance with standards in 5 areas: access, a patient registry, care coordination, care plans, and quality improvement methods. The certification process involved an application and self-assessment; an on-site visit that included interviews with both staff and patients; and review by a multi-stakeholder advisory committee. The leaders of all 132 clinics that had been certified as of October 2011 agreed to participate in this study. However, this report is limited to those 108 certified clinics that provide care to adults, returned the survey of practice systems noted below, and either completed an additional survey that included satisfaction questions or had standardized patient experience data available.

We collected descriptive information about these clinics from a survey completed by the manager of each clinic; it was constructed to identify their size, the number of clinics in their medical group, and patient population. We also administered a survey (and have reported the results previously) that measured the presence of practice systems for medical homes.

 This is the research version of the original PPC-PCMH (Physician-Practice Connection — Patient-Centered Medical Homes) survey used by the National Committee for Quality Assurance to recognize practices as medical homes.

 Known as the PPC-Research Survey (or PPC-RS), this instrument is completed by the medical leader of each clinic to document the presence and function of medical home—related systems present at the time of completion as well as those that were present 3 years earlier. It is scored on a scale of 0% to 100%, where 100% means that all measured systems are present and working well. Scores were calculated for each clinic for what was then the present time (2010-2011) and the point 3 years prior, and for the change between those points in time. The results from this instrument have been reported previously for the subset of clinics that serve adults and we have used them to assess the degree of transformation and the change in the past 3 years.

	1. Job satisfaction among physicians in our clinic has increased as a result of changes we made to become a medical home

	2. Job satisfaction among nonphysician staff in our clinic has increased as a result of changes we made to become a medical home

Our measurement of physician and staff job satisfaction was dependent on separate assessments by the physician leader and clinic manager on a survey on which 108 of the 120 adult clinics had responses from both leaders (90.0%). The specific questions asked with a 5-point answer scale, from Strongly Agree (1) to Strongly Disagree (5), were:

Answers to these questions (on a 1-5 scale) were averaged between the 2 leader respondents for each clinic.

Measurement of patient experience depended on standardized use of the CG-CAHPS (Clinician and Groups version of the Consumer Assessment of Healthcare Providers and Systems) survey developed by MN Community Measurement, a nonprofit organization established and supported by all the health plans in Minnesota to develop and report publicly on standardized measures of quality. CG-CAHPS is a well-established public domain survey created and maintained under contract to the Agency for Healthcare Research and Quality; it is increasingly used nationally as the standard way to assess patient experience.15 The data about patient experience used in this study were collected from those clinics that volunteered to participate in this survey in 2010, knowing that in the next year it would be required for all clinics in Minnesota.

Four domains of satisfaction were reported by MN Community

Measurement. The individual questions within sections 1, 2, and 3

	1. Access to care—a composite of 5 questions about the frequency of getting urgent and routine appointments as soon as needed, of having phone calls returned during office hours and after hours as soon as needed, and of being seen by the provider within 15 minutes of appointment time.

	2. Provider communication—a composite of 6 questions about the frequency of the provider listening carefully, giving understandable information, being aware of one’s medical history, showing respect, spending enough time, and providing test results.

	3. Office staff—a composite of 2 questions about the frequency of office staff being helpful and showing courtesy and respect.

	4. Provider rating—a single question rating the provider on a 1-to-10 scale from worst to best.

had response choices of Never, Sometimes, Usually, and Always:

Each question is scored as the percentage of respondents indicating the highest satisfaction choice, and the composites represent an average of these percentages across the constituent items. The overall provider rating represents the percentage of respondents choosing either a 9 or 10. An average of 327 responses were obtained per clinic, representing 35% of the surveys distributed.

	- Physician and staff satisfaction data were available for 108 clinics

	- Patient experience data were available for 54 clinics

The analysis used Spearman correlations to test the relationship between each satisfaction measure and the PPC score for systems in the present (at the time of the survey) as well as the absolute amount of change in the prior 3 years. Spearman rank correlations are more powerful than parametric statistics (eg, Pearson correlations, linear regression) when the data are ordinal and non-normal. Due to varying sample sizes, the satisfaction analyses were conducted with 2 subsets of the original 132 clinics depending on the availability of data for each question:

 provides comparative descriptive information about the clinics included in these analyses, both the 108 with data on staff and physician job satisfaction and the subset of 54 with patient experience data. Those with patient satisfaction data are clearly different, with nearly all of these clinics being part of large medical groups in the Minneapolis-St. Paul metropolitan area that have smaller proportion of patients who have Medicare or Medicaid insurance.

The frequencies and variance for the 8 variables of interest for this analysis are shown in 

. On average, leader ratings of staff or physician satisfaction related to medical home changes were neutral but with wide variance, while patient experience with access was much lower than their experience with staff or providers. Practice systems were reported to have more than doubled over the previous 3 years, but with wide variation among the clinics in both current level and extent of change during that time.

, each measure of satisfaction is tested for correlation with the current PPC score (ie, degree of correlation to the ideal medical home) or the change in PPC score (ie, transformation) over the previous 3 years. These data show that there is no relationship between the current PPC score and experience or satisfaction of either patients or clinic personnel. However, the extent of change over the past 3 years is strongly associated with higher job satisfaction by both physician and physician personnel (as reported by clinic leaders). Change is also strongly negatively associated with patient experience with access.

We also tested the associations between clinic characteristics and satisfaction using Spearman-rank correlations, and demonstrated that metro area practices had higher patient access satisfaction scores (r = .31, P = .02) and practices largely limited to primary care physicians had higher physician and staff satisfaction (r = .20 and .23, P = .04 and .02). A regression analysis confirmed that these associations with clinic characteristics did not change the satisfaction relationships with PPC. Controlling for location (urban vs suburban) and physician composition differences, a standard deviation increase in PPC change (6 percentage points for job satisfaction measures and 18 percentage points for patient satisfaction measures) is associated with an increase of 0.15 in physician job satisfaction (P = .029) and an increase of 0.12 in staff job satisfaction (P = .028), but a 1.8 decrease in percent patient satisfaction with access (P = .005).

These data suggest that our hypothesis about an inverse relationship between transformation and patient experience is only true for access problems in clinics where there has been more change in clinic systems. This may represent access deterioration as an unintended consequence of change. Patients of clinics that have undergone little change, regardless of the current level of transformation, do not state this dissatisfaction. On the other hand, neither the extent to which clinics have medical home systems nor the degree to which they have changed to create them appears to be related to patient satisfaction with personal interactions with either staff or physicians. In contrast, the more change that has occurred, the more staff and physicians in these clinics are reported by their leaders to be satisfied.

When compared with previous reports of patient experience in relation to medical home transformation, this study and the clinics analyzed demonstrate differences. Most important may be that the conflicting studies reported from Group Health and from the National Demonstration Project were able to compare change in measured satisfaction over time in medical home versus comparison clinics, while we measured satisfaction at a single point in time for clinics achieving certification as medical homes. However, we were able to relate that single measure to the degree to which clinics reported making change in their systems.

The literature is otherwise mixed about PCMH relationships to satisfaction. At Group Health, after adjusting for differences between a PCMH clinic and 2 comparison clinics at baseline, there were statistically significant increases at 12 months that had mostly decreased at 24 months in patient satisfaction with the quality of doctor-patient interactions, shared decision making, coordination of care, and access, but no change for helpfulness of office staff.

 The National Demonstration Project compared patient satisfaction in facilitated versus self-directed medical practices, and had a similar measure to ours of the number of model components in place as a way to measure change.

 Over 26 months, there were no significant improvements in any of 8 domains of patient-rated outcomes. There were borderline significant decreases in coordination, comprehensiveness, and access in both groups of clinics. Those practices increased their proportion of model components by an average of 24%, with a borderline significant relationship (P = .11) to patient- reported outcomes.

Two studies in safety-net clinics have shown an association between

PCMH characteristics or perceptions and higher staff and

provider/resident morale and satisfaction.

safety-net clinic also showed an improvement in patient satisfaction

In the recent systematic review that included a few more observational studies, Jackson et al concluded that there was moderately strong evidence that PCMH interventions are associated with small improvements in patient reports of satisfaction and care coordination.

 They also reported a low level of evidence that primary care staff may be more satisfied in PCMH practices. In their systematic review covering roughly the same time period, Hoff et al found inconclusive evidence of change in patient/family experience. Finally, in a study of 393 practices published after these systematic reviews, Martsolf et al concluded that there was “no association between PCMH processes and patient experience.”

Even fewer studies have measured the impact of the PCMH on clinician and staff job satisfaction. Reid’s evaluation of physician burn-out among the clinical staff at a Group Health PCMH clinic compared with 2 control clinics showed no differences at baseline and a significant decrease in only the PCMH clinic after making PCMH changes over 24 months.

 Hoff ’s review did not find any other adequate studies of provider experience.

Our study’s limitations include its cross-sectional design that was limited to clinics achieving medical home certification, its dependence on reports by clinic leaders about the job satisfaction of physicians and other staff, and its limited and self-selected subgroup of clinics with patient satisfaction data. Because the data for clinic characteristics, job satisfaction, and patient satisfaction were not normally distributed, we used Spearman-rank correlations to test the association between clinic characteristics and satisfaction. Subsequently, we conducted a linear regression that suggests that clinic characteristics do not confound the associations between PPC and satisfaction. However, due to the smaller sample size in the MNCM satisfaction measures, there is potential for instability in the regression models.

Also, as Table 1 makes clear, Minnesota primary care clinics are unusual in being larger than many other comparable United States clinics and often aggregated into larger medical groups. Thus it is possible that those differences, or external factors other than transformation, might have affected these findings. However, Minnesota Community Measurement data were also available for patient experience in 44 self-selected Minnesota clinics not certified as healthcare homes, although we have no other data about those clinics. Those clinics had experience scores similar to those of the certified clinics on access, physician communication, and staff helpfulness, but their physician ratings were actually 2.6% higher than the healthcare homes’ ratings. Finally, patient experience data were not available for all of the certified clinics in our study. However, our sample size, data about change over time, and the availability of other published evidence about the transformation process and its other impacts on these clinics provide a unique comprehensive examination of them.

Overall, these findings add evidence to the important topic of the impact of medical home changes on clinicians, staff, and patients. Taken together with the minimal other published work on these topics, we suggest that impact on patient experience is not great, but that rapid and large changes risk deteriorating patient access. On the other hand, those same rapid changes may contribute to improved physician and staff job satisfaction. Since morale among primary care professionals is generally not good, this is an important finding.

 Sinsky et al report from case studies of 23 high-performing primary care practices that “a shift from a physician-centric model of work distribution and responsibility to a shared-care model, with a higher level of clinical support staff per physician and frequent forums for communication, can result in high-functioning teams, improved professional satisfaction, and greater joy in practice.”

The changes reported in this study suggest that clinics that function like medical homes may indeed be happier places to work, both for physicians and other staff.

The authors would like to thank Eva Kline-Rogers, MS, RN, NP (University of Michigan, MCORRP), and Cydni A. Smith, BA (University of Michigan School of Public Health); Redah Mahmood, MD; Daniel Montgomery, MS; Rachel Sylvester, BS (MCORRP); and the student data abstractors who helped with this study. We are very grateful for the cooperation of the Minnesota Department of Health throughout this study, and for the way in which the leaders of the certified healthcare homes in Minnesota cooperated with the data collection. We are also grateful to Minnesota Community Measurement for providing us with the patient satisfaction data used in this analysis.

 HealthPartners Institute for Education and Research, Minneapolis, MN (LIS, LS, ALC, RRW, PLF, JOT); MN Community Measurement, Minneapolis, MN (AMcGS).

 This project was supported by grant number R18HS019161 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.

 The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Leif I. Solberg, MD, HealthPartners Research Foundation, PO Box 1524, MS#21111R, Minneapolis, MN 55440-1524. E-mail: 

.1. Hoff T. The shaky foundation of the patient-centered medical home. 

2. Hoff T, Weller W, DePuccio M. The patient-centered medical home: a review of recent research. 

3. Reid RJ, Coleman K, Johnson EA, et al. The Group Health medical home at year two: cost savings, higher patient satisfaction, and less burnout for providers. 

4. Reid RJ, Fishman PA, Yu O, et al. Patient-centered medical home demonstration: a prospective, quasi-experimenetal, before and after evaluation. 

5. Jaen CR, Ferrer RL, Miller WL, et al. Patient outcomes at 26 months in the patient-centered medical home National Demonstration Project. 

6. Nutting PA, Crabtree BF, Stewart EE, et al. Effect of facilitation on practice outcomes in the National Demonstration Project model of the patient-centered medical home. 

7. Rosser WW, Colwill JM, Kasperski J, Wilson L. Progress of Ontario’s Family Health Team model: a patient-centered medical home. 

Becoming a medical home appears to increase physician and staff job satisfaction, but it also risks decreasing patient satisfaction with access to care.

Patient Experience and Physician/Staff Satisfaction in Transforming Medical Homes

To identify the care model factors that were key for successful implementation of collaborative depression care in a statewide Minnesota primary care initiative.

We used a mixed-methods design incorporating both qualitative data from clinic site visits and quantitative measures of patient activation and 6-month remission rates.

Care model factors identified from the site visits were tested for association with rates of activation into the program and remission rates.

Nine factors were identified as important for successful implementation of collaborative care by the consultants who had trained and interviewed participating clinic teams, and rated according to a Likert Scale. Factors correlated with higher patient activation rates were: strong leadership support (0.63), well-defined and -implemented care manager roles (0.62), a strong primary care physician champion (0.60), and an on-site and accessible care manager (0.59). However, remission rates at 6 months were correlated with: an engaged psychiatrist (0.62), not seeing operating costs as a barrier to participation (0.56), and face-to-face communication (warm handoffs) between the care manager and primary care physician for new patients (0.54).

Care model factors most important for successful program implementation differ for patient activation into the program versus remission at 6 months. Knowing which implementation factors are most important for successful activation will be useful for those interested in adopting this evidence-based approach to improving primary care for patients with depression.

Nine implementation factors were most important for the success of the collaborative care model for depression and differed for patient activation into the program versus achieving remission at 6 months.

• Strong leadership support and a strong physician champion are essential for patient activation into the program.

• The more well defined and implemented the care manager role, the higher the rate of patient activation.

• The more engaged a psychiatrist was and the more often in-person communication occurred, the more frequently patients experienced remission from their depression.

• The less likely a group experienced operating costs as a barrier, the more likely their patients were to experience remission.

There is extensive evidence from randomized controlled trials that collaborative care for depressed adults in primary care improves patient outcomes.

 Key elements in evidence-based collaborative care programs include consistent measurement and monitoring of depression severity, close proactive follow-up by a clinic-based care manager, and regular psychiatric consultation focused on treatment changes for patients who are not improving with initial treatment. Based on these studies, the US Preventive Services Task Force recommends that routine screening of adults for depression is justified only when systems for collaborative depression care are in place.

 Not only can collaborative care produce better patient outcomes (with rates of remission and response that are approximately twice those of usual care), but it can also produce net cost savings over 4 years.

Despite these findings, however, little is known about which implementation factors are most important for achieving these outcomes. For example, prior studies of collaborative care have employed care managers with wide varieties of education and experience without providing information about comparative benefits on outcomes.

 It is also unclear what supports a care manager needs to function most effectively or whether it is important for the psychiatrist to be onsite to provide consultation and supervision. Similarly, it is unknown whether an effective local primary care champion or face-to-face communication between the primary care provider (PCP) and care manager are important.

Between 2008 and 2012, an initiative led by a regional quality improvement collaborative, the Institute for Clinical Systems Improvement (ICSI), systematically provided standardized training in implementing collaborative depression care and consultative support for primary care clinics throughout Minnesota and western Wisconsin. The initiative, Depression Improvement Across Minnesota — Offering a New Direction (DIAMOND), included payment redesign through a partnership with nearly all commercial health plans in the state.

 While maintaining fidelity to the core aspects of the model was required, local tailoring was considered important, so there were significant variations in implementation. The initiative also collected standardized process and outcomes data as part of the quality improvement support system, as well as information about each clinic’s approach to the care model. This quantitative information was supplemented with a round of site visits to all participating groups, providing a unique opportunity to document differences in care processes and implementation strategies. This information allowed examination of which approaches to implementation might be important for high levels of enrollment and good patient outcomes.

The DIAMOND initiative was created in 2006 by a diverse stakeholder group convened by ICSI that included health plans, medical clinics, patients, and employers, with the goal of planning a new approach to depression care. After extensive reviews and discussions, it became clear that both the collaborative care model and payment redesign were needed. The group recommended that payers provide a monthly fee to DIAMOND-certified sites for eligible patient-members enrolled in the care model.

The structure of the initiative was based largely on the collaborative care model as it was tested in the Improving Mood-Promoting Access to Collaborative Treatment (IMPACT) study.

 It focused on 6 components: 1) use of the Patient Health Questionnaire-9 (PHQ-9)

 for assessment and ongoing monitoring; 2) use of a registry for systematic tracking of patients; 3) use of evidence-based guidelines to provide stepped care treatment modification/ intensification; 4) relapse prevention education; 5) a care manager located in the clinic to provide education, care coordination, behavioral activation, and support of medication management; and 6) a consulting psychiatrist to meet with the care manager for weekly case review and treatment change recommendations.

ICSI conducted training for 5 sequences of clinics participating in the new model over the course of 2 years; every 6 months a new sequence started the 6-month training and implementation program, beginning in September 2007 and continuing until the final sequence started implementation in March 2010. Each sequence consisted of 10 to 26 clinics. In Minnesota nearly all PCPs are organized into single or multispecialty organizations termed “medical groups” that include a number of clinics or practice sites; small, independent practices are rare. A total of 99 clinics representing 21 different medical groups implemented the program.

Each clinic provided standardized monthly data reports through a common Internet portal about the number of patients seen by the care coordinator, the number enrolled in DIAMOND (activation rate), and the PHQ-9 scores (needed to calculate response [change in PHQ-9] and remission [PHQ-9 <5] rates at 6 and 12 months). These quantitative data were supplemented with interview data from a round of site visits in 2009-2010 to all medical groups. For this analysis, we focused on medical groups who had completed all site visits and had at least 50 patients in their DIAMOND program (7 had <50) for a total of 42 clinics from 14 medical groups. The local Institutional Review Board reviewed and approved this study.

Activation rate was defined as the number of eligible patients (PHQ-9 >10) who entered DIAMOND per PCP full-time equivalent per month (PCP FTE/M). Remission rates (defined as PHQ-9 <5) were calculated at 6 months post activation. To calculate the overall activation and remission rates for each medical group, the monthly rates were averaged for the period of March 2008 to September 2010.

At least 2 ICSI staff attended each site visit, and all clinics were provided with materials prior to the site visit meeting. Materials included sequence-specific outcomes data; an overall DIAMOND data report strategies each group used in implementation; and a discussion guide focused on barriers and facilitators. The latter included questions about practice culture; team approach; care manager role and duties; medical/psychiatric complexities of patients; psychiatry consults; care coordination; registry use; and approach to financial issues (see 

). Site visit meetings included the core team participating in training and implementation, which included the project lead, care manager, and PCP champion. Other staff encouraged to attend were other physicians, the consulting psychiatrist, and the quality improvement lead. Following each site visit, ICSI staff completed a structured qualitative narrative to document their assessment of factors affecting implementation. This narrative focused on their perceptions of the implementation strategies, barriers, and facilitators, noting information about team dynamics, staff concerns, clinic staff response to the program, and their overall impression of program implementation at the site. Summaries were then prepared by the ICSI site-visit teams and were reviewed by the entire study team.

Twenty-three factors were initially identified in the structured qualitative narratives. The analysis team and ICSI staff (n = 8) then used a modified Delphi method to identify, multi-vote, and rank factors believed to be most related to successful implementation of DIAMOND (see 

Following identification of these factors, a Likert scale rating system was used to determine the extent to which each factor was present in each medical group, from 0 (absent implementation) to 4 (full implementation). ICSI staff rated each medical group on each of the 9 top implementation factors.

To assess the association between implementation factors and activation and remission rates, we calculated Pearson correlation coefficients between each implementation factor and activation and remission rates. Scatter plots were used to understand the form of the relationship for all associations. Simple linear regression was used to estimate the effect of each 1-point increase (on a scale of 0-4) in implementation on activation and remission rates at 6 months. All reported P values are 2-sided and considered significant at P <.05.

This analysis focuses on the 14 medical groups implementing DIAMOND that had 50 or more patients in their program. The majority were multispecialty medical groups (79%) located in the Twin Cities metropolitan area (57%). The number of clinics implementing the program in each group, the PCP FTE/M count of each, and activation and remission rates are shown in

. On average, about 1 patient was activated per PCP FTE/M, and 23% of patients activated into the program were in remission at 6 months. In keeping with the approach of allowing local tailoring, features of the care manager role varied across program sites. Of the 32 care managers in these medical groups, there were registered nurses (n = 15, 47%), licensed practical nurses/certified medical assistants (n = 11, 34%), and licensed social workers/bachelor’s- level psychologists (n = 6, 19%). The majority (72%) had their DIAMOND care manager role as their primary duty, while 28% had other shared clinical duties. Most care managers (59%) worked with patients from a single clinic, with the remaining (41%) working with patients from several clinics.

Implementation Factors and Patient Activation and Remission.

 Correlation analysis showed statistically significant and moderately strong positive correlations for 5 of the implementation factors with patient activation into the program: strong leadership support, strong care manager, care manager role well defined and implemented, care manager on-site and accessible, and strong PCP champion (see 

). We conducted simple linear regression of significant correlations to estimate the effect of increases in scale rating (rating scale 0-4) of implementation factors. Each of these factors was associated with about a 0.4 increase in activation rate.

Correlation analysis also showed statistically significant and moderately strong positive correlations between 3 implementation factors and patient remission rates at 6 months: engaged psychiatrist, warm handoffs (meaning referrals from clinicians to the care manager are usually conducted face-to-face rather than though indirect means), and operating costs not seen as a barrier (see

). Simple linear regression to estimate the effect of an additional increase in scale rating (rating scale 0-4) on remission showed that the less often a medical group experienced operating costs as a barrier, the more likely their patients were to experience remission. Similarly, the more engaged a psychiatrist was and the more often warm handoffs occurred, the more likely patients experienced remission from their depression.

Nine factors were considered important for implementation of the DIAMOND collaborative care model, broadly including areas of leadership, care management, physician engagement, and financial issues. Our findings show that the implementation factors significantly correlated with patient activation were different from the factors correlated with 6-month remission. Having strong leadership support, a strong PCP champion, a strong care manager whose role is both well defined and implemented, and a care manager who is on-site and accessible were significantly correlated with activating patients into the program. On the other hand, having an engaged psychiatrist, warm handoffs, and not seeing operating costs as a barrier were significantly correlated with depression remission at 6 months.

Program implementation is a vital component of building effective collaborative care for depression, although much of the research to date has focused on outcomes, sustainability, and cost-effectiveness.

 A review of multisite studies outlined specific implementation steps and decisions needed to tailor collaborative care for local needs.

 A study of collaborative care in the Netherlands identified factors that facilitated implementation, including continuous supervision of care managers, a supportive Web-based tracking system, and a reimbursement system allowing payment for mental health practitioners.

A qualitative analysis of implementation activities in 42 organizations found sites averaged 30 different implementation efforts with modest intensity.

There are no previous studies identifying specific factors in implementation and how they relate to patient activation and remission outcomes.

Remission of depressive symptoms has long been the primary focus for successful programs receiving significant attention, but activation (enrollment) of patients into the program is equally important. Across the entire DIAMOND initiative, enrollment of eligible patients with depression averaged about 15%, or fewer than 1 out of 6 eligible patients, with activation rates varying among medical groups and only a few exceeding 20%.

 Thus, while improved remission rates were good, the overall impact of the program was limited by relatively low activation rates of eligible patients. A focus on both activation and remission could provide a stronger foundation for success and ongoing program support.

Our data show that strong organizational leadership was the most important factor in patient activation; it has long been identified with program success. Providing ongoing institutional support and direction helps lay the foundation on which programs can build.

 Organizational structure and leadership support are the most common facilitators of success for improving the treatment of depression in primary care.

 Expert team leadership and support from local management also strongly influence the success of programs for improving depression care.

 We found that a strong PCP champion for depression care is important for overcoming barriers in the clinic setting and can encourage PCPs to refer their depressed patients to the care manager.

 Resistance by individual physicians to sharing the care of their patients with a care manager can be a significant barrier to patient activation.

The care manager is a critical element in any collaborative care program.

 Our data show that important components for patient activation are having a care manager whose role is clearly defined and well implemented and who is onsite. Recognizing that collaborative care is a specific model of care with systematic processes is essential, as it sets a clear process for how the care manager role is implemented in the clinic setting.

 Clearly defining the care manager role supports the care manager and clarifies how he or she will be interacting with both patients and other clinic personnel. This provides an important foundation for communication and interaction and the basis for a successful team.

While having the care manager on-site and available was associated with activation, in-person communication with the PCP was more highly correlated with remission. Theoretically, in-person communication might have been expected to be more likely to affect patient activation than remission, since it develops the relationship between the PCP and care manager and facilitates patient acceptance of the care manager. However, it may be that it has a greater effect on remission instead because it provides a foundation for stronger working relationships between providers regarding ongoing patient care. Previous studies have similarly found that collaborative mental healthcare is most successful when clinicians are colocated in the clinic setting.

Having an engaged psychiatrist was also strongly associated with patient remission. Engagement by a psychiatrist can provide important and ongoing support for the patient, care manager, and treating PCP. Engaged psychiatric consultants can help address concerns about psychiatric diagnosis or treatment as they arise, make recommendations about adjusting treatments if patients are not improving, and help problem-solve challenging care situations. This consultation builds the expertise of the care manager and PCP, and it can evolve into effective long-term working relationships. Collaborative care managers working with high-risk mothers with depression found they highly valued psychiatric expertise and needed increased psychiatric support with patients of higher complexity to improve care and outcomes.

Not seeing operating costs as a barrier was also highly correlated with remission rates. As with face-to-face communication, this factor may initially appear more directly related to the activation of patients, given that reimbursement for DIAMOND care was contingent on whether a patient’s health plan provided this coverage. Many clinics, however, provided DIAMOND care for patients even without payment coverage, potentially decreasing the association of cost considerations with activation. Costs for ongoing care provided by care managers were absorbed into overall program costs in these clinics and covered by the clinic or medical group. It may be that clinics which took this approach and provided ongoing services to all patients, regardless of coverage or ability to pay, demonstrated higher remission rates because of that commitment to better outcomes.

There are limitations to this study that warrant caution in interpretation of these results. The identification of implementation factors was based on subjective ratings, albeit by the people most familiar with the operational issues of this model. The sample size for the analysis is small and focused on medical groups composed of individual heterogeneous clinics. Finally, we have little information about other factors that may have contributed to activation or remission rates, such as patient characteristics or

other organizational factors. While we acknowledge these limits, we also note that there are strengths to mixedmethod approaches like this, such as the unique opportunity to study real-world settings in all their complexity and to take advantage of the expertise and skill of both the program and the clinic staff.

Factors most important for successful implementation of collaborative care for depression differ for patient activation versus achieving remission; both are critical to program success.

Leif I. Solberg, MD

Articles

It Is Time to Ask Patients What Outcomes Are Important to Them

Patient Experience and Physician/Staff Satisfaction in Transforming Medical Homes

Effective Implementation of Collaborative Care for Depression: What Is Needed?

The Effect of Depression Treatment on Work Productivity