James M. Naessens, ScD

Advance care planning (ACP) consults have been credited with increasing the likelihood that patients will receive desired care at the end of life, including reduced hospitalization and spending. We aimed to investigate whether implementation of ACP consults leads to improved advance directive (AD) completion rates.

Chart review was conducted among patients in hematology/oncology and primary care to compare AD completion rates of patients in each clinical area offered ACP consults between July 1, 2017, and June 30, 2018, and those not offered ACP consults prior to July 1, 2017.

Completed ADs increased from historic rates of 1% to 65.9% of primary care patient records. Among hematology/oncology patients offered consults, completed ADs were present in 28.1% of patient records compared with historic rates of 3%.

 Patients offered ACP consults had significantly higher AD completion rates than those without consults in both primary care and hematology/oncology practices. The difference in ACP consult attendance between the primary care and hematology/oncology groups clearly suggests that further improvements can be made to assist patients in accessing AD education and completion support.

Patients who are offered advance care planning consults have significantly higher advance directive documentation rates than those without consults in both primary and specialty care practices.

Advance directives were completed by 82% of primary care and 71% of hematology/oncology patients completing advance care planning consults.

Consult acceptance rates appear to be influenced by mode of invitation, as well as patient age and underlying condition.

In addition to making end-of-life care more likely to match patient wishes, advance directives have been shown to lower end-of-life costs.

The Patient Self-Determination Act was established in 1990 to ensure that US patients are informed of their right to make medical care decisions.

 Medical ethicist Bud Hammes, PhD, led a community effort in La Crosse, Wisconsin, in 1993 to develop an advance care planning (ACP) and decision-making model known as Respecting Choices.

 ACP entails 4 primary actions: (1) thinking through one’s relevant values and preferences, (2) talking about one’s values and preferences, (3) documenting one’s stated values and preferences, and (4) reviewing and updating documentation specific to one’s stated values and preferences. When performed in a comprehensive manner, these actions serve to (1) minimize decision-making burden on family members; (2) reduce the likelihood of conflicts among a patient’s spokesperson, family members, and health care providers; (3) minimize the likelihood of over- or undertreatment; and (4) maximize the likelihood that medical care is consistent with patient wishes.

 Within 2 years of implementing ACP in La Crosse, completion of advance directives (ADs) in the region rose from 2% to 45%.

 This improvement was attributed to widespread standardization, documentation, and utilization of ACP processes.

In addition to better aligning care with patient wishes,

 adoption of ACP has been credited with reducing end-of-life cost of care. In 2015, Dartmouth Atlas identified the La Crosse region as ranking among the areas with the lowest hospital utilization and total health care cost for end-of-life care.

 The national rate for hospitalizations in the last 6 months of life was 125 hospitalizations per 100 older decedents, whereas inpatient spending for that time frame was $15,366 per decedent. Similar rates for the La Crosse region were 102 hospitalizations per 100 older decedents and $12,194 per decedent.

Clinicians at Mayo Clinic Florida (MCF) sought to determine whether ACP consultations could be implemented in an academic, multispecialty health care organization located in the Southeast United States with comparable results.

Beginning July 2014, MCF implemented a pilot to investigate the impact of offering ACP consultations on AD completion rates. An ACP consult refers to an in-depth conversation between a patient and facilitator (typically a social worker). During the 1-hour consult, situations and examples are used to help patients understand options regarding future health care choices. ACP consults are performed using the Honoring Choices model.

 An AD describes 2 types of legal documents, a Living Will and a Healthcare Power of Attorney/Designation of Healthcare Surrogate, that enable patients to plan for and communicate end-of-life wishes and designate health care decision makers to act on their behalf if necessary. Initial facilitator preparation includes 15 total hours of online and classroom training. All trained facilitators receive annual continuing education. The MCF ACP facilitator team includes social workers, volunteers, and support staff.

The pilot aimed to achieve 100 patient referrals yielding 50 ACP consults and 50 completed ADs documented in the electronic health record by October 2014. Over the next 2 years, the program was expanded to include primary care, hematology/oncology, dialysis, neuroscience, and palliative care patients. Patient self-referrals were also encouraged.

Our retrospective study compared the AD completion rate for patients offered ACP consults with that for a randomly selected group of patients seen in the same specialties in the 90 days prior to consult implementation. The study entailed manual chart reviews of patients from 2 specialties that had the largest number of ACP consults: primary care (Community Internal Medicine) and hematology/oncology (Cancer Center). The referral pathway in primary care stemmed from provider education by a physician champion for ACP. In hematology/oncology, the referral pathway was generated primarily from the psychosocial distress screen and social work consults. Data on completed ADs were collected for patients who were offered ACP consults during the time range of July 1, 2017, to June 30, 2018. For each of the control groups, a random sample was reviewed from the list of patients who had an appointment scheduled in primary care or hematology/oncology in the 90 days prior to implementation after excluding patients who subsequently had ACP consults. Data on completed ADs were collected from the samples of primary care patients from March 1, 2015, to May 31, 2015, and hematology/oncology patients from February 1, 2016, to April 30, 2016. Patient demographics, ACP consult attendance rates, and AD documentation rates were compared between the intervention and control groups with χ

 tests, and logistic regression adjusting for differences in patient age.

, primary care patients who were offered ACP consults were significantly older than hematology/oncology patients offered consults. In addition, the hematology/oncology patients offered consults were significantly older than their control group. There were no significant differences in gender between the 2 study groups.

The ACP consult attendance and AD completion rates for each group are displayed in the 

. A total of 320 patients in the primary care group were offered ACP consults during our time frame, and 257 (80.3%) attended an ACP consult. The ACP consult led to a completed AD document for 211 (82.1%) patients. Of the 63 patients in the primary care “offered and declined” group, 30 patients opened a patient portal message offering an ACP consult but did not respond and 33 patients had a scheduled ACP consult but did not attend. None of these patients had a completed AD in their record. Overall, 65.9% of primary care patients who were directly offered a consult completed ADs. The historic rate of completed ADs among a random sample of primary care patients seen in the prior 3 months was 1%.

A total of 690 patients in the hematology/oncology group were offered ACP consults. A total of 222 (32.2%) patients completed ACP consults, leading to 157 (70.7%) completed ADs. There were 468 patients in the hematology/oncology “offered and declined” group; 453 patients received a digital distress screening tool that offered an ACP consult, 11 patients had a scheduled ACP consult but did not attend, and 4 patients had a separate appointment in which they were offered an ACP consult but declined. We reviewed 100 randomly selected patients from this group, of whom 8 (8%) had a completed AD in their record. Overall, extrapolating from our sample, 28.1% of hematology/oncology patients who were directly offered a consult completed an AD. Meanwhile, the historic rate of completed ADs among a random sample of hematology/oncology patients seen in the prior 3 months was 3%. Among patients attending ACP consults, AD completion rates were higher among primary care patients than hematology/oncology patients (primary care: 82.1%; hematology/oncology: 70.7%; age-adjusted 

ADs have been shown to increase the likelihood that patients receive the care that they want at the end of life.

 We discovered that providing ACP consults was associated with a significant increase in AD completion rates in both primary care, where completion rates among those offered consults was 65.9% compared with 1% among the historical control group, and in the hematology/oncology patient group, where completion rates increased from the historical rate of 3% to 28.1% among those offered consults. Although we hypothesized that hematology/oncology patients might complete ADs at higher rates due to malignancy diagnoses, among patients who attended ACP consults, AD completion rates were significantly higher in primary care than in hematology/oncology. Primary care patients also attended ACP consults at higher rates than hematology/oncology patients. One explanation for these findings lies in how the opportunity for ACP consult was presented to patients. All the primary care patients offered consultations were on the primary care panel of one of the authors (C.M.). They were invited to participate through a letter from their primary care provider. Meanwhile, most of the Cancer Center patients who were offered ACP consults were identified at the time of psychosocial distress screening. Based on answers to an iPad-based tool about ACP, patients were informed about AD, asked about their current AD status, and offered an ACP consult. These findings could also be age-related, as the primary care patients were a mean of almost 4 years older than the hematology/oncology patients. Alternatively, the hematology/oncology patients may be interested in ACP but may want to complete ADs with their primary care providers.

A review of our program noted several key strengths and critical success factors. First, some providers regularly discussed ADs with their patients, leading to increased referrals for ACP consults. This finding points to the criticality and impact of the referral source.

 Second, utilizing an electronic psychosocial distress screening tool to identify candidates for ACP consults served to increase patients’ overall awareness of ACP, including the availability of consults. Third, the option for patients to self-refer presented an additional path for ACP consults. Fourth, the support of dedicated resources as a consistent point of contact for information, education, general inquiries, scheduling, and problem-solving proved critical to the success of the study. Fifth, the enthusiasm and continued support from trained facilitators and committed leadership proved invaluable to the success of the pilot and subsequent expansion of ACP consults organization wide.

Our findings were consistent with published literature. In a systematic review of ACP in the population of patients with end-stage renal disease, the implementation of ACP consults was associated with statistically significant improvements in the completion of ADs.

 The review also stressed the importance of the standardized ACP process as a “core business” competency at the organizational level, which resonated with our study. Specifically, provisions to ensure optimal patient access to ACP resources, including properly trained health care personnel, were deemed crucial. Literature focusing on the acute care environment highlighted the importance of continued organizational and administrative support in terms of “provider education, patient information, and quality review” in the ACP process.

Establishing ACP appointments has some financial implications. Is the effort worth the costs? Our costs were not excessive, as they were primarily for 1-hour appointments for social workers and administrative support. Although several studies report the cost savings of completed ADs near the time of death, we could find no studies reporting the cost benefit of potentially improving AD completion through offering ACP consultations. A systematic review reported in 2016 found cost savings per patient for having documented ADs ranging from $1041 to $64,827 in 6 studies and 1 study reporting no savings.

 An additional study published in 2018 reported costs in the year prior to death of $9500 less in the ACP group than among matched controls.

One important caveat to consider is that the costs of ACP consults are currently borne by health care providers, whereas the AD completion benefits are experienced by health care payers or society as a whole. Our analysis suggests that to maximize the benefits of AD completion, health care providers offering ACP consults should be reimbursed for their efforts. Since 2016, Medicare has included additional payments for ACP as an optional add-on to the annual wellness visit.

 We are encouraged that our efforts appear to have replicated the La Crosse, Wisconsin, experience. Further efforts are needed to determine how effective this approach could be in other settings.

In addition to the need for more study of the impact of AD completion, we feel that it is important to assess whether ACP consults can be modified to improve their benefit. The experience seen in differences in ACP consult acceptance clearly suggests that improvements can be made in getting patients to attend these consults once there is a broader understanding of their importance.

Several limitations specific to our study were also noted: (1) inconsistencies among providers resulted in less-than-optimal ACP consult referrals; (2) patients exhibited varying levels of comfort in their ability to utilize technology (ie, the iPads used for psychosocial distress screening); and (3) the lack of specifically allocated and dedicated, trained facilitators, as well as adequate space, affected availability of and accessibility to ACP consults. Because of limited manual review resources, we focused our analysis on primary care and hematology/oncology, which had the largest number of ACP consults.

Patients offered ACP consults have significantly higher AD documentation rates than those without consults in both primary care and hematology/oncology practices. The benefit of lower end-of-life costs (based on the literature) far exceeds the cost of offering ACP consults. Further research is needed to improve patient attendance at ACP consults.

 Administration (MAL), Section of Social Work (ERG, NPF, NP, GD), Department of Health Sciences Research (LJW), and Community Internal Medicine (CM), Mayo Clinic, Jacksonville, FL; Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery (JR, JMN), Mayo Clinic, Rochester, MN.

 This research was made possible by funding from the Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery.

The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (MAL, ERG, GD, JMN); acquisition of data (ERG, NPF, LJW, NP, GD); analysis and interpretation of data (MAL, LJW, JR, NP, GD, JMN, CM); drafting of the manuscript (MAL, ERG, NPF, JR, NP, JMN, CM); critical revision of the manuscript for important intellectual content (MAL, ERG, NPF, JR, JMN, CM); statistical analysis (LJW, JR, JMN); provision of patients or study materials (NPF); obtaining funding (JMN); administrative, technical, or logistic support (MAL, ERG, NPF, GD, JMN); and supervision (MAL, ERG, NPF, JMN, CM).

 James M. Naessens, ScD, Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, Mayo Clinic, 200 First St SW, Rochester, MN 55902. Email: naessens@mayo.edu.

1. Annas GJ. Nancy Cruzan and the right to die. 

. 1990;323(10):670-673. doi:10.1056/NEJM199009063231010

2. Hammes BJ, Rooney BL. Death and end-of-life planning in one midwestern community. 

. 1998;158(4):383-390. doi:10.1001/archinte.158.4.383

3. Pearlman RA, Tonelli M, Braddock C III, Edwards K. Advance care planning & advance directives. University of Washington Medicine Department of Bioethics and Humanities. Accessed July 29, 2020. http://depts.washington.edu/bhdept/ethics-medicine/bioethics-topics/detail/54

4. Hatkoff C, Kula R, Levine Z. How to die in America: welcome to La Crosse, Wisconsin. 

. September 23, 2014. Accessed August 26, 2020. https://www.forbes.com/sites/offwhitepapers/2014/09/23/how-to-die-in-america-welcome-to-la-crosse/#1643eb06e8c6

5. Brinkman-Stoppelenburg A, Rietjens JAC, van der Heide A. The effects of advance care planning on end-of-life care: a systematic review. 

. 2014;28(8):1000-1025. doi:10.1177/0269216314526272

6. End of life care. Dartmouth Atlas Project. Accessed February 14, 2020. https://www.dartmouthatlas.org/interactive-apps/end-of-life-care/

7. Wilson KS, Kottke TE, Schettle S. Honoring Choices Minnesota: preliminary data from a community-wide advance care planning model. 

. 2014;62(12):2420-2425. doi:10.1111/jgs.13136

8. Silveira MJ, Kim SYH, Langa KM. Advance directives and outcomes of surrogate decision making before death. 

. 2010;362(13):1211-1218. doi:10.1056/NEJMsa0907901

9. Detering KM, Hancock AD, Reade MC, Silvester W. The impact of advance care planning on end of life care in elderly patients: randomised controlled trial. 

10. Sudore RL, Fried TR. Redefining the “planning” in advance care planning: preparing for end-of-life decision making. 

. 2010;153(4):256-261. doi:10.7326/0003-4819-153-4-201008170-00008

11. Schellinger S, Sidebottom A, Briggs L. Disease specific advance care planning for heart failure patients: implementation in a large health system. 

. 2011;14(11):1224-1230. doi:10.1089/jpm.2011.0105

12. O’Halloran P, Noble H, Norwood K, et al. Advance care planning with patients who have end-stage kidney disease: a systematic realist review. 

. 2018;56(5):795-807.e18. doi:10.1016/j.jpainsymman.2018.07.008

13. Seal M. Patient advocacy and advance care planning in the acute hospital setting. 

14. Klingler C, in der Schmitten J, Marckmann G. Does facilitated advance care planning reduce the costs of care near the end of life? systematic review and ethical considerations. 

. 2016;30(5):423-433. doi:10.1177/0269216315601346

15. Bond WF, Kim M, Franciskovich CM, et al. Advance care planning in an accountable care organization is associated with increased advanced directive documentation and decreased costs. 

. 2018;21(4):489-502. doi:10.1089/jpm.2017.0566

16. Frequently asked questions about billing the Physician Fee Schedule for advance care planning services. CMS. July 14, 2016. Accessed August 10, 2020. https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/PhysicianFeeSched/Downloads/FAQ-Advance-Care-Planning.pdf

Articles

Implementing advance care planning consults can increase advance directive completion rates. The authors demonstrate the impact of consults on completed advance directives in the medical record.

Impact of Advance Care Planning Consults on Advance Directives Completion

With the advent of healthcare payment reform, identifying high-risk populations has become more important to providers. Existing risk-prediction models often focus on chronic conditions. This study sought to better understand other factors to improve identification of the highest risk population. 

A retrospective cohort study of a paneled primary care population utilizing 2010 data to calibrate a risk prediction model of hospital and emergency department (ED) use in 2011. 

Data were randomly split into development and validation data sets. We compared the enhanced model containing the additional risk predictors with the Minnesota medical tiering model. The study was conducted in the primary care practice of an integrated delivery system at an academic medical center in Rochester, Minnesota. The study focus was primary care medical home patients in 2010 and 2011 (n = 84,752), with the primary outcome of subsequent hospitalization or ED visit. A total of 42,384 individuals derived the enhanced risk-prediction model and 42,368 individuals validated the model. Predictors included Adjusted Clinical Groups—based Minnesota medical tiering, patient demographics, insurance status, and prior year healthcare utilization. Additional variables included specific mental and medical conditions, use of high-risk medications, and body mass index. 

The area under the curve in the enhanced model was 0.705 (95% CI, 0.698-0.712) compared with 0.662 (95% CI, 0.656-0.669) in the Minnesota medical tiering-only model. New high-risk patients in the enhanced model were more likely to have lack of health insurance, presence of Medicaid, diagnosed depression, and prior ED utilization. 

An enhanced model including additional healthcare-related factors improved the prediction of risk of hospitalization or ED visit.

		Many risk models that predict hospitalization or emergency department (ED) use are based on comorbid health burden. 

		We investigated other potential predictors, including previous utilization, and found an association of higher hospitalization and ED use with the following: Minnesota medical tiering, 2 prior ED visits, warfarin use, no insurance or Medicaid, being aged over 70 years, and obesity stage 3.

		These findings help to develop new models for determining patients at risk for adverse health outcomes. They focus on potential future areas of clinical practice and research on social determinants of health.

Changing the payment incentives to improve the organization of care delivery, including accountable care organizations, has led to a reorganization of delivery of care focusing on patient-centered medical homes.

 Many patient-centered medical homes stratify their populations and try to align the level of care with the needs of their patients. Providers manage the most complex patients with increased resources to avoid adverse outcomes requiring high-cost care in acute settings.

Two key questions need to be answered: Who is likely to have utilization, and what can be done for these patients? These medically complex patients could be better managed through care coordination programs that are integrated within the delivery system, such as Guided Care, which incorporates specially trained nurses to supplement primary care in a comprehensive care model that provides coordinated, patient-centered primary care for patients with complex needs.

 Different methods for identifying these patients have been developed, including provider identification and referral.

 A hybrid system that utilizes provider selection and an automated risk stratification model may be ideal. Most of the currently available public and proprietary risk models and instruments to identify the population for inclusion in more intensive management programs are based on number and combination of comorbidities. These instruments include self-administered tools such as the Predicting Risk of Admission (PRA) survey, which incorporates self-rated health and comorbid conditions as well as prior healthcare utilization.

 Another model utilizing medical record data is the Elder Risk Assessment (ERA) index, which determines risk of hospitalization based upon increasing age and previous hospitalization as well as comorbid health conditions.

A third risk model, Adjusted Clinical Groups (ACG), uses medical care claims data and stratifies patients based on age and existing chronic conditions.

 The state of Minnesota pays medical homes a care management fee based on Minnesota medical tiering. Minnesota medical tiering is a system of classifying and counting comorbidities originally based on collapsing chronic diseases from ACGs into a body system count.

 This system can also be manually applied without the proprietary software. Our clinical practice uses Minnesota medical tiering to identify appropriate patients for care management. Our prior work comparing these instruments, including Minnesota medical tiering, indicates that these instruments are similar in terms of predicting the risk of hospitalization or emergency department (ED) visit; however, the ACG performed slightly better while the Charlson Comorbidity Index performed worse.

 Although the administrative data-based risk identification methods have been validated and can predict hospitalization with C statistics of 0.67-0.73,

 other potential predictors may improve current risk prediction models and could provide information useful in developing tailored interventions to manage a patient’s condition.

Our clinical experience indicates that coordinating the management of multiple chronic conditions is important in reducing rehospitalizations among geriatric patients with multimorbidity, but other factors may also be important in predicting and managing an individual’s risk.

 In our practice, many patients predicted to be high risk are excluded from care management interventions because they have concurrent mental health or social conditions that care coordinators are unprepared to address. Body mass index (BMI), medication use,

 living environment, English as a second language, previous healthcare use including specialty visits, insurance, and mental health status

 might all be important factors. Patients with those issues may need a different approach, including more support from social services, pharmacy, and psychology.

Obesity is known to influence utilization, but it is widely underdiagnosed and so may be missed in many comorbidity counts.

 The use of anticoagulation therapy, pain, and diabetic medications can lead to an increased need for unplanned acute care and may be amenable to pharmacy consultation.

 Patients living on their own or in dysfunctional families may not be able to cope with increased treatment burdens and may benefit from social services.

 Similarly, those who need language interpretation services may need extra assistance. Patients with a high number of specialty visits may have uncoordinated care and may need help in coordinating their care. Lack of insurance and lower socioeconomic status are associated with increased rates of hospitalization.

 Patients experiencing mental health issues such as depression can also have higher readmission rates.

 Providers and systems with established comorbid health models should understand the potential roles of these additional predictors for hospitalization. 

It is unclear how well these additional factors would predict hospitalization when added to an already established clinical model based on administrative data. To answer this question, we performed a retrospective cohort study comparing a standard risk stratification model of Minnesota medical tiering with an enhanced model containing additional predictors of demographics, selected use of medications, mental health conditions, other specific chronic medical conditions, prior healthcare utilization, and insurance status. We sought to understand how well the enhanced model predicts hospitalization and ED use in a subsequent year.

We constructed a retrospective cohort study to develop and validate a clinical prediction model for hospitalization. The study was approved by the Mayo Clinic Institutional Review Board.

The study was conducted at an academic medical center in Rochester, Minnesota. The population came from the primary care medical practice, Employee and Community Health (ECH), which includes 4 primary care clinics in Dodge and Olmsted Counties, Minnesota. Primary care is provided by family medicine, primary care internal medicine, and community pediatric and adolescent medicine.

Patients 18 years or older who were assigned to a primary care provider within any of the primary care clinics for all 12 months in 2010 (base year) and throughout 12 months or until death in 2011 (assessment year) were included in the analysis. Subjects were excluded if they refused consent for medical record review in accordance with Minnesota state law. Of the 90,411 patients within the ECH program, as of December 31, 2010, 84,752 (93.7%) gave consent for medical record review.

All information for enrolled ECH patients was electronically abstracted from the electronic medical record (EMR) and administrative databases for 2010 and 2011.

We modeled a combined binary outcome of hospitalization and/or ED visit (hereafter: hosp/ED) at St. Mary’s Hospital or Rochester Methodist Hospital in the assessment year. Hospitalization was defined as any elective or emergent overnight. To avoid double counting, ED visits exclude those resulting in a hospitalization.

Demographic variables—age, gender, marital status, and socioeconomic status&mdash;were collected at the end of the base year. Age was grouped into 6 categories: 18 to 29 years, 30 to 39 years, 40 to 49 years, 50 to 59 years, 60 to 69 years, and 70 years or older. Socioeconomic status was determined by insurance type (Medicaid or no insurance versus other insurance type).

As Minnesota medical tiering is the basis for care coordination payments for patient-centered medical homes in Minnesota, we used it as the basis for our risk model development. Medical conditions are based on diagnosis codes billed by clinical providers.

 Minnesota medical tiering has 5 categories from 0 with no conditions to tier 4 with more than 10 conditions. In our care management program, high-risk patients possess Minnesota medical-tiering scores of 3 or 4. 

Candidate predictors for identifying high-risk patients included primary speaking language not being English; high-risk medications (warfarin, insulin, narcotics); and previous healthcare utilization in specific settings (ED visits, hospitalization, specialty provider visits). Medications were based on the presence of outpatient pharmacy orders in 2010 written by any healthcare providers within the system. Medication use was determined from the EMR, not from filled prescriptions. The medications included warfarin, narcotics and insulin. Non—English speaking status was assigned to those patients needing an interpreter, which was documented within the electronic health record. We included all diagnosis codes (

International Classification of Diseases, Ninth Revision, Clinical Modification

]) from hospitalizations, ED visits, and primary and specialty care visits to identify comorbid mental health and medical conditions, based on a published method.

BMI—recorded as weight (kilograms) / height (meters)

—was also included as a potential predictor. BMI was assessed as being the most recent value in the record within 2 years prior to January 1, 2011. Specific BMI categories followed standard World Health Organization criteria: underweight (BMI <18.5); normal weight (BMI 18.5-24.9); overweight (BMI 25-29.9); stage 1 obesity (BMI 30-34.9); stage 2 obesity (BMI 35-39.9); and stage 3 obesity (BMI >40).

We developed the enhanced model to identify those patients at high risk of hosp/ED within the next year. The 2 models compared in the analysis consisted of Minnesota medical tiering and the enhanced model, which incorporated additional candidate factors. Data were randomly divided into development (50%) and validation cohorts.

We derived the final enhanced model from a stepwise logistic regression followed by a sensitivity analysis seeking stability in the predictors determined by the severity of the outcome variable. Predictors in the saturated model consisted of 72 main factors as well as all possible 1-way and 2-way interaction terms. We determined stability of the variables by changing the number of ED visits or inpatient (IP) visits in the outcome variable. We performed the saturated stepwise model while incrementally increasing the number of ED visits or IP visits. The final set of variables for each change in outcome variable was captured. We then considered a count of how many times a variable had a 

 value ≤.05 in the model. Finally, we included any variable that remained significant in 4 or more of the 11 models considered.

The stepwise regression was repeated 100 times and each repetition considered the variables in random order to determine if order affected the results of the stepwise algorithm. Each iteration produced the same model. Effects were entered into and removed from the model so that each forward selection step could be followed by a backward elimination step. The algorithm stopped when no further effects could be added to the model or if the current model was identical to a previous model.

Comparisons between prediction models focused on predictability and precision. We used C statistics (area under the receiver operating characteristic [ROC] curve [AUC]) as our measure of predictability. For assessment of precision, we compared the sets of high-risk patients found within the tier 3 and 4 groups in Minnesota medical tiering with an equal number of high-scoring patients from the enhanced model. Statistical analysis was performed using SAS version 9.3 (SAS Institute, Cary, North Carolina).

A total of 42,384 and 42,368 patients were included in the development and validation samples, respectively. The development and validation data sets were similar. There were no differences in sex, age decile, BMI, marital status, insurance status, prior healthcare utilization, comorbid health status, or Minnesota medical tiering (

). In addition, the 2 samples had similar rates of hospitalization and ED visits with 18.2% of the development sample and 17.6% of the validation sample having a hospitalization or ED visit in the subsequent year.

Those patients who belonged to the highest category of Minnesota medical tiering (4) had the greatest odds of incurring any hospitalization or ED visit with an odds ratio of 18.2 (95% CI, 15.6-21.24) compared with the lowest tier category (0). Next most important was prior utilization of 2 or more ED visits with an odds ratio of 7.13 (95% CI, 6.34-8.01) for predicting hosp/ED. Similarly, 2 or more previous hospital stays had an odds ratio of 4.28 (95% CI, 3.90-4.69) for predicting hosp/ED. Mental health conditions, including substance abuse, personality disorder or dementia, each had odds near 3 for predicting hosp/ED. Specifically, depression had an odds ratio of 2.92 (95% CI, 2.71-3.15). Among other medical conditions, cardiac disease with heart failure and pulmonary heart disease had odds ratios over 5. All specific conditions with odds ratios over 2 are reported in Table 1. All specific conditions are listed in 

. Patients in each of the medication categories had odds ratios greater than 2, as did those utilizing 3 or more specialist visits in the prior year. Patients with a language interpreter did not have elevated risk of hosp/ED (Table 2).

The multivariable enhanced tiering model is presented in 

. The final model included 30 main effects (representing 16 factors) and 4 2-way interaction terms (more detailed results are available from the authors).

Of notable mention, being in the highest medical tier remains the strongest predictor for hosp/ED after the addition of other factors with an odds ratio of 3.50 (95% CI, 2.78-4.42). More than 1 previous ED visit also resulted in an increased odds ratio of 3.17 (95% CI, 2.78-3.60). Several significant interaction terms are noted in Table 3. In the validation cohort, the C statistic for the enhanced multivariable model was 0.711 (95% CI, 0.704-0.717) compared with a C statistic for the Minnesota medical tiering alone model of 0.667 (95% CI 0.660-0.673). These C statistics were actually slightly higher in the validation cohort than in the development cohort (enhanced model: 0.705 [95% CI, 0.698-0.712]; Minnesota medical tiering: 0.662 [95% CI, 0.656-0.669]). The differences between the ability of the enhanced model to predict hospitalization and ED use and Minnesota medical tiering are displayed in the ROC curves found in the 

Many different patients were identified as being high-risk in the enhanced model as compared with those in the original Minnesota medical tiering model (tier 3 or 4). 

 presents the characteristics of high-risk patients identified with both models as well as those unique to the 2 approaches. We found that patients identified as high-risk in the enhanced model were much more likely to experience hospital utilization in 2011 than those in the Minnesota medical tiering-only model (

 <.001). Patients identified as high-risk based only on the enhanced model, compared with those identified only through Minnesota medical tiering, were of younger age, had higher previous healthcare utilization, and had more frequent mental health conditions. Many of the high-risk enhanced model patients were aged less than 40 years compared with only 11% of patients being aged less than 40 years in the Minnesota medical tiering model.

In this retrospective cohort of patients receiving care within the ECH, several variables were important in predicting hosp/ED in addition to the number of chronic comorbidities currently used in Minnesota medical tiering and our care coordination identification. We found that age, BMI, previous healthcare utilization, mental health conditions, anemia, heart failure, epilepsy, hyperlipidemia, warfarin use, and narcotic use all simultaneously contributed to the prediction of hospitalization and ED visits in the next year. This enhanced model had an AUC of 0.711 (95% CI, 0.704-0.717) for predicting hosp/ED, as compared with 0.667 (95% CI, 0.660-0.673) found in the model utilizing only Minnesota medical tiering. Previous work comparing different risk stratification methods for hospitalization determined slight superiority of ACG compared with the ERA, Hierarchical Condition Categories, Minnesota medical tiering, and chronic care counting instruments.

 The enhanced model takes into consideration previous hospitalization, specific high-risk illnesses, mental health conditions, and high-risk medication use (eg, warfarin, narcotics) that are not universally accounted for in other models. In comparing high-risk groups of the same size from both models and excluding patients identified in both models, 47% of the patients in the enhanced model experienced hosp/ED compared with 32.5% of patients from the Minnesota medical tiering model. The enhanced model accounts for factors in a younger population (eg, mental health and previous utilization) that are not detected in the Minnesota medical tiering model.

Previous hospital and ED utilization served as strong predictors for hosp/ED in the following year. Specifically, 2 or more prior ED visits presented an odds ratio (OR) close to 3 while 1 prior hospital stay resulted in an OR of 1.79 (95% CI, 1.66-1.93). Together, hospitalization and ED visits often indicate a medically complex individual or an individual with severe illness. Many risk stratification models have utilized previous healthcare usage as a predictor for future healthcare use. The PRA incorporates previous hospitalization within the past year as a factor predictive of future hospitalization.

 Similarly, the ERA also includes previous hospital days to calculate a risk score for future hospitalization.

 Previous ED visits as well as previous hospitalization have been associated with increased hospital admission in a multi-centered VA study of 1378 patients designed to identify factors of hospitalization.

An enhanced risk model incorporating medication use, prior healthcare utilization, and mental health with comorbid health conditions predicts healthcare utilization better than health conditions alone.  

Enhanced Risk Prediction Model for Emergency Department Use and Hospitalizations in Patients in a Primary Care Medical Home

Published literature should be more critically evaluated when the chances of safety, outcome, or cost implications are high

		Abstracts can use overstated results or terms to drive readership and citation rates.

		Since the abstract may be the only part of a paper that administrators read, errors in abstracts can have significant real-world impacts on decision making.

		Errors in interpretation of results can also have significant real-world impacts on decision making that can ripple across the entire spectrum of healthcare management.

 that compares different risk stratification methods is enlightening, although the authors have overstated their results and a discrepancy between the abstract and their results both contribute to the possibility that their study could be misinterpreted in costly ways.

It is good practice to include CIs on all point estimates, and CIs are at least available for the main results presented in 

 (on following page). But the authors fail to interpret those values, and as a result conclude—incorrectly&mdash; that the Adjusted Clinical Group (ACG) model is the best. Given the information available in the paper, it is correct to say that the results are consistent with there being no difference between ACG and Minnesota Tiering (MN). In fact, the paper’s figures (which lack CIs, unfortunately) and C statistic values suggest that any differences in outcomes for all models might be statistically distinct at times but are practically trivial. It is also questionable whether the C statistic is even appropriate for comparing models with different drivers, and whether it accurately portrays the receiver operating characteristic curve upon which it rests (it can’t).

The specific error in the abstract states that the C statistic for the ACG model for predicting the highest 10% of cost users is 0.81, but Table 2 shows that the correct value is 0.76 (95% CI, 0.75-0.76). Since this overlaps with the results for the MN model (0.74; 95% CI, 0.74-0.75), the ACG model is not clearly “superior to the others.” If the authors meant to refer to readmissions, again the overlap in CIs between ACG and MN show that they are essentially equivalent models in this outcome as well. The term “superior” is a value-laden word that can be easily overinterpreted without a close read of the results; its use here presents both clerical and conceptual errors.

It may well be that the ACG model is superior to these other models, but this paper did not demonstrate that—in fact, this study’s results actually support the idea that all of these models are practically equivalent. Academics might call that conclusion a “negative result” and decide not to publish because of the excess difficulty of getting it through review. But as an industry professional, I would consider “negative results” like these&mdash;when properly interpreted&mdash; “fiscally important results.”

 is read by thousands of on-the-ground healthcare industry workers, and time-constrained administrators often take abstracts at face value, having no time to evaluate the results themselves. In fact, this paper made its rounds among our administrators when it was published, and may (or may not) have contributed to our subsequent decision to purchase one of these models. Purchasing risk models is expensive, and internal switching, implementation, and training costs multiply that well beyond the cost of the model. I would ask that editors and reviewers make absolutely sure that authors’ conclusions are warranted by their results before publication—improper interpretation can be as damaging and/or costly as clerical errors when subjected to the decision-making hurricane that is today’s healthcare industry.

The letter “Simple Errors in Interpretation and Publication Can Be Costly,” a response to the article “Risk-Stratification Methods for Identifying Patients for Care Coordination,” is primarily based on 1 line in the abstract. The author of the letter did identify a typographical error in the abstract which included the C statistic for readmissions instead of that for high-cost users (we have submitted an erratam to correct the error). Although we agree that the word “superior” could have been softened, the ACG did perform better than other models in predicting healthcare utilization. For example, as can be seen in Table 2 (below) of the original paper, for predicting the top 10% high-cost users, the ACG had non-overlapping CIs with all models except for MN Tiering, which is based on the ACG.

We believe the abstract was clear. Unfortunately, as abstracts are limited with regards to space, we would hope the reader would draw their impressions from the entire paper.â€‹ The conclusions of the paper stated that although the ACG was generally better at predicting utilization, all models had good concordance, suggesting that choosing any model would be more beneficial than none. All models, excluding ACG and MN Tiering, are free and publically available.

We agree with the correspondent that receiver operating characteristic curves should not be the only basis of comparison between models. In fact, the analysis did not rely strictly on C statistics, but also focused on calibration, particularly regarding the accuracy of identifying patients at the high end of each of the outcome distributions as displayed in Table 3 and Figure 2 (on following page).

 Dwight Barry, PhD, Group Health Cooperative, 320 Westlake Ave N, Ste 100, Seattle, WA 98109. E-mail: barry.d@ ghc.org.

The authors (LRS, PYT, NDS, RJS, MEB, DMF, JMN) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of the article.

: Concept and design (LRS, PYT, NDS, RJS, MEB, DMF, JMN); acquisition of data (LRS, JMN); analysis and interpretation of data (LRS, NDS, RJS, MEB, DMF, JMN); drafting of the manuscript (LRS, PYT, MEB, JMN); critical revision of the manuscript for important intellectual content (LRS, PYT, NDS, RJS, MEB, JMN); statistical analysis (LRS, JMN); obtaining funding (JMN); and administrative, technical, or logistic support (DMF).

Lindsey R. Haas, MPH, Health Care Policy and Research, Mayo Clinic, 200 First St Southwest, Rochester, MN 55905. E-mail: haas.lindsey@mayo.edu. 1. Haas LR, Takahashi PY, Shah ND, et al. Risk-stratification methods for identifying patients for care coordination. Am J Manag Care. 2013;19(9):725-732.

2. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928-935.

3. Hand DJ. Measuring classifier performance: a coherent alternative to the area under the ROC curve. Mach Learn. 2009;77:103-123.

4. Trouble at the lab. The Economist. 2013;409(8858):26-30. 

A recent AJMC study contained overstatements and small but importantly placed errors that have the potential to cause unwarranted on-the-ground cost problems.

Simple Errors in Interpretation and Publication Can Be Costly

Care coordination is a key component of the patient-centered medical home. However, the mechanism for identifying primary care patients who may benefit the most from this model of care is unclear.

To evaluate the performance of several risk-adjustment/stratification instruments in predicting healthcare utilization.

 All adults empaneled in 2009 and 2010 (n = 83,187) in a primary care practice were studied. We evaluated 6 models: Adjusted Clinical Groups (ACGs), Hierarchical Condition Categories (HCCs), Elder Risk Assessment, Chronic Comorbidity Count, Charlson Comorbidity Index, and Minnesota Health Care Home Tiering. A seventh model combining Minnesota Tiering with ERA score was also assessed. Logistic regression models using demographic characteristics and diagnoses from 2009 were used to predict healthcare utilization and costs for 2010 with binary outcomes (emergency department [ED] visits, hospitalizations, 30-day readmissions, and highcost users in the top 10%), using the C statistic and goodness of fit among the top decile.

The ACG model outperformed the others in predicting hospitalizations with a C statistic range of 0.67 (CMS-HCC) to 0.73. In predicting ED visits, the C statistic ranged from 0.58 (CMSHCC) to 0.67 (ACG). When predicting the top 10% highest cost users, the performance of the ACG model was good (area under the curve = 0.76) and superior to the others.

 Although ACG models generally performed better in predicting utilization, use of any of these models will help practices implement care coordination more efficiently.

For care coordination to be used effectively, it is important to identify which patients are likely to benefit from it. Little evidence exists as to which risk-stratification method performs better in predicting hospitalizations, emergency department visits, 30-day readmissions, and high-cost users.

		There appeared to be good concordance among 6 different risk screening instruments in predicting hospitalization.

		Adjusted Clinical Group classification was generally better than other models in predicting healthcare utilization.n

		Focusing care coordination efforts within the medical home on patients likely to benefit most requires appropriate identification of the highest risk, highest utilizing patients.

With rising prevalence of chronic disease, patient-centered medical homes (PCMHs) were created to deliver higherquality, more cost-effective primary care in the United States.

 A key component of PCMH is the implementation of care coordination, especially if targeted to the right people.

 While some states have developed risk-adjusted care coordination payments,  methods for effective identification of individuals likely to benefit from care coordination are not clear.

Patients with multiple chronic conditions may see many providers, often in an uncoordinated manner, adding to their increased risk of high healthcare utilization through test duplication, edication management conflicts, and medical errors.

These patients are also at increased risk for hospital admissions and emergency department (ED) visits. Primary care doctors may not routinely have the time or resources to properly manage these complex patients who have multiple chronic health problems. Thus, care  coordination for the right patients could decrease unnecessary care and prevent adverse outcomes.

Most existing PCMH programs identify care coordination patients through physician referrals.

 However, this method may not identify all potential beneficiaries. Care coordination should be focused on patients who will benefit most, maximizing the impact on both quality and costs. Risk-stratification models can be efficient tools for both providers and health plans to screen populations and select individuals for care management programs who are at risk of future hospitalization and functional decline.

 Several methods using administrative data have been described to help with the identification of high-risk  patients.

 However, it remains unclear which method will perform best in identifying patients in need of care coordination. To answer this important and fundamental question, we compared the performance of 6 common risk-adjustment methods in predicting hospitalizations, ED visits, 30-day readmissions, and high expenditures. This study extends the existing literature on patient selection for care coordination with results from direct comparison of the following models: Adjusted Clinical Groups (ACGs), Hierarchical Condition Categories (HCCs), Elder Risk Assessment (ERA), Chronic Comorbidity Count (CCC), Charlson Comorbidity Index, and Minnesota Health Care Home Tiering (MN Tiering). The identification of an optimal method may guide implementation of care coordination programs within PCMHs. Furthermore, identifying these predictable  high-utilization cases may lead to other effective interventions.

A retrospective cohort analysis compared multiple methods to identify high-risk patients. These methods were applied to all primary care patients 18 years or older empaneled to the Employee and Community Health (ECH) practice (family medicine, primary care internal medicine, and community pediatric and adolescent medicine) at Mayo Clinic, Rochester, Minnesota, the community-focused primary care arm of a large integrated multispecialty group practice.

To be a patient of ECH, one must be a Mayo Clinic employee or live in the community, identify an ECH primary care provider, and be enrolled within his/her panel. Participants assigned to a primary care provider for all 12 months in 2009 (base year) and throughout 12 months or until death in 2010 (assessment year) based on the electronic medical record were included in the analysis. The organized development of care coordination within ECH occurred during 2011; hence, usual care (encounter driven) was provided to all patients during the study. Subjects were excluded if they refused consent for medical record review in accordance with Minnesota state law.

All information was electronically abstracted from the electronic medical record and administrative databases within Mayo Clinic’s health records system. Mayo Clinic maintains all electronic medical record information within 1 system, including inpatient and outpatient visit information.

Demographic variables collected in the base year included age, sex, marital status, and insurance status. Age was grouped into 4 categories of 18 to 44 years, 45 to 64 years, 65 to 84 years, and more than 85 years. Insurance status was also grouped into 4 categories: public (Medicare or Medicaid), private, no insurance, and unknown. Diagnosis codes (

International Classification of Diseases, Ninth Revision, Clinical Modification 

) for each patient encounter, as well as utilization and cost information, were extracted from institutional billing data for year 2009. Number of hospital days was also retrieved from 2008 to calculate ERA scores. We included all diagnosis codes from hospitalizations, ED visits, and primary and specialty care evaluation and management visits.

The ACG is a commonly used classification system based on administrative diagnosis data and developed by Johns Hopkins to measure morbidity.

 The ACG methodology was developed to predict the utilization of medical resources using the presence or absence of specific diagnoses from both inpatient and outpatient services for a specified period of time, along with age and sex, to classify each person into 1 of 93 discrete ACG categories with similar expected utilization patterns. Adjusted Clinical Groups can be used to improve accuracy and fairness in forecasting healthcare utilization

 and have been found to predict inpatient hospitalizations as well as or better than other case-mix tools in many health systems.

 The MN Tiering model is based on a product of ACGs, Major Extended Diagnostic Groups.

 MN Tiering is currently being used by the State of Minnesota to determine management fees for care coordination among medical home plans. The purpose of MN Tiering is to group patients into “complexity tiers” based on the number of major condition categories from which they suffer.3 The total sum of conditions is grouped into the following 5 patient complexity levels: low (tier 0): 0  conditions; basic (tier 1): 1 to 3 conditions; intermediate (tier 2): 4 to 6 conditions; extended (tier 3): 7 to 9 conditions; and complex (tier 4): 10 or more conditions.

In 2004, the Centers for Medicare & Medicaid Services (CMS) implemented HCCs to adjust Medicare capitation payments for health expenditure risk of Medicare Advantage Plan (health maintenance organization) enrollees.

 diagnosis codes and demographic data for each patient are aggregated into 70 condition categories that contribute to a single risk score.17 Several studies among Medicare patients have provided evidence that HCC scores for risk adjustment can be effective at predicting hospitalizations.

 Elder Risk Assessment Index. The ERA Index was developed to identify patients at risk for hospitalization and ED visits in adults 60 years or older.

The ERA Index incorporates a weighted score of age, sex, number of hospital days in the prior 2 years, and marital status, as well as selected medical conditions (diabetes, coronary artery disease, congestive heart failure, stroke, chronic obstructive pulmonary disease,and dementia).

The minimum score on the index is —1 and the maximum score possible is 34.

The CCC method was derived by Naessens and colleagues

 from a modification of the method of Hwang and colleagues,

 based on the publicly available Agency for Healthcare Research and Quality’s Clinical Classification Software. The CCC method is more comprehensive than most comorbidity counts. The total sum of chronic conditions was grouped into 6 categories: 0, 1, 2, 3, 4, and 5 or more. Comorbidity counts have been shown to be associated with high annual costs as well as persistence in high costs.

 The Charlson Comorbidity Index, originally derived to classify comorbidities affecting 1-year mortality in cancer patients, sums weights for 17 specific conditions.23 In our study, the sum of Charlson Comorbidity Index counts was used to predict future outcomes. The performance of the Charlson Comorbidity Index in predicting poor outcomes has been assessed in various large populations, and its validity as a prognostic measure of outcomes has been consistently demonstrated.

 A care coordination patient enrollment process was developed in our primary care practice based on combining MN Tiering and the ERA score as a hybrid model. Patients eligible for enrollment in care coordination included all patients in MN tier 4 (individuals with more than 10 comorbid conditions) and those in MN tier 3 with ERA scores greater than 10. Although all of the previously mentioned risk-stratification instruments have been validated, it remains unclear how the hybrid method of combining MN Tiering and ERA score will compare with them.

Identifying which patients are likely to benefit from care coordination is important. We evaluated the performance of 6 risk-screening instruments in predicting healthcare utilization.

James M. Naessens, ScD

Articles

Impact of Advance Care Planning Consults on Advance Directives Completion

Enhanced Risk Prediction Model for Emergency Department Use and Hospitalizations in Patients in a Primary Care Medical Home

Simple Errors in Interpretation and Publication Can Be Costly

Risk-Stratification Methods for Identifying Patients for Care Coordination

Effect of Multiple Chronic Conditions Among Working-Age Adults

Flunking Asthma? When HEDIS Takes the ACT