John F. Steiner, MD, MPH

 To compare missed appointment rates for patients receiving a single reminder either 3 days prior to a primary care visit, 1 day prior to the visit, or both 3 days and 1 day prior to the visit.

 Three-armed randomized controlled trial.

 Text messages or interactive voice response calls were sent to patients with appointments at 25 primary care clinics in an integrated delivery system. A multivariable prediction model was developed to identify patients at high risk of missing appointments, based on prior appointment-keeping history and other variables from electronic health records.

 Among 54,066 randomized patients, those who received reminders both 3 days and 1 day prior to the visit were less likely to miss their appointment than those who received only a 3-day or 1-day reminder (4.4% vs 5.8% vs 5.3%, respectively; 

<.001). In patients at high risk, 20.5% of those who received 2 reminders missed their visit, compared with 25.0% and 24.2% of those with only 3-day or 1-day reminders, respectively (

 <.001). Visit satisfaction was unaffected by providing an additional reminder.

 Two automated reminders were more effective than 1 in reducing missed appointments and did not reduce visit satisfaction. A predictive model based on clinical characteristics and prior appointment history can accurately identify patients who are at highest risk of missing appointments. These individuals may benefit more from multiple reminders, but additional strategies are necessary to further reduce their rates of missed appointments.

Text message or phone reminders 3 days and 1 day before a primary care appointment were more effective in reducing missed appointments than a single reminder, particularly in patients whom a clinical prediction rule identified as at high risk of missing appointments.

Decision makers posed the question to fine-tune existing reminder protocols through collaboration with researchers in a learning health system.

Because prior randomized trials had not directly compared either the timing or number of reminder calls, the findings may apply to other organizations.

Healthcare delivery systems use many strategies to remind patients of upcoming clinic appointments. A 1992 meta-analysis found that mailed reminders or telephone calls from clinic staff consistently reduced missed appointments.

 Newer tools, such as interactive voice response (IVR) telephone calls and text messages, have also been effective while improving the efficiency and reducing the cost of appointment reminders.

Although the effectiveness of visit reminders is well established, little is known about the optimal number or timing of reminders. A reminder several days in advance provides patients an opportunity to cancel their appointment with sufficient lead time for a clinic to schedule other patients into vacated slots. On the other hand, a reminder the day prior to an appointment may allow the patient to better assess whether the visit remains necessary or conflicts with other obligations.

 Although multiple reminders may be more effective than a single reminder, patients may perceive them as unnecessary or intrusive.

Learning health systems combine clinical data, quality improvement methods, and research skills to address operational questions rapidly and at low cost.

 When operational interventions are randomized, the resulting information is both directly actionable and potentially generalizable to other delivery systems. We previously reported the results of a collaborative trial between operational leaders and researchers in an integrated healthcare system, which found that a single reminder delivered by text or IVR message 1 day prior to a primary care appointment reduced the rate of missed appointments compared with no reminder.

 To refine the approach to delivering appointment reminders, we conducted a second randomized controlled trial to evaluate 3 strategies for providing primary care visit reminders: (1) a single reminder 3 days prior to the visit, (2) a single reminder 1 day prior to the visit (the standard approach based on the initial trial), or (3) reminders both 3 days and 1 day prior to the visit. We also assessed the effect of these 3 approaches on visit satisfaction and the impact of the intervention in the subgroup of patients at highest risk of missing their appointments.

This project took place at Kaiser Permanente Colorado (KPCO), an integrated healthcare delivery system with more than 650,000 members in the Rocky Mountain West. Patients received care at 27 primary care clinics that were staffed by family physicians, internists, pediatricians, nurse practitioners, and physician assistants. Patients could make appointments for primary care visits in person, by telephone, or through a web portal. Since 2005, KPCO's research department has operated a visit reminder system that initiates telephone calls to landlines or cellular phones and text messages to cellular phones. Text messages are the primary communication modality unless the recipient has requested voice messages. We designate this system as IVR-T (interactive voice response or text). This system has been used in numerous interventions, several of which have been deployed operationally after completion of the research.

At the time of this study, the delivery system sent a single IVR-T reminder to each primary care patient 1 day before their scheduled visit, based on the results of our prior randomized trial.

 To fine-tune the approach to visit reminders, operational leaders then asked whether a 3-day reminder might be preferable to a 1-day reminder, because it provided greater opportunity to fill slots vacated through cancellations. Although leaders were also interested in determining whether 2 reminders would further reduce missed appointments, they raised concerns that multiple calls might be burdensome to patients and encouraged us to use existing visit satisfaction data to explore this issue.

In response to these questions, we designed a pragmatic intervention trial. Stratifying by clinic site, we randomly assigned all patients with primary care appointments between October 15, 2016, and December 15, 2016, to 1 of 3 groups: a single IVR-T reminder 3 calendar days prior to the appointment; a single IVR-T reminder 1 calendar day prior to the appointment; or 2 IVR-T reminders, with one made 3 days and the other 1 day prior to the appointment. Calls were made on weekends, as well as business days. All patients were eligible if they had a primary care appointment and had made their appointment 3 or more days in advance of the scheduled visit. Patients who previously requested not to be contacted by phone or text message were excluded. One clinic was excluded because its staff made personal calls rather than using IVR-T reminders, whereas a second site was excluded because it opted to continue using the 1-day reminder protocol. We excluded scheduled telephone visits, visits for office procedures, and visits to non—primary care departments, such as radiology and ophthalmology. Patients who made appointments online through the KPCO patient portal and received email reminders were included in the intervention. Although information about email reminders was unavailable, the randomized design is likely to have distributed individuals who received these reminders equally among the 3 groups.

All text messages or calls were delivered in English. Those receiving a text message were asked to respond with their medical record number if they wished to cancel the appointment. Patients who answered the IVR call confirmed their identity by providing their medical record number, then received a short message confirming the date and time of the appointment and were offered the opportunity to cancel their appointment through the appointment line. If an answering machine responded, a recorded message provided the same information.

We used a randomization algorithm in the program that managed the IVR database to assign each visit to 1 of the 3 groups. Because randomization took place at the visit level, patients with multiple visits during the study period could receive the IVR-T intervention for some visits but not others. We limited the analysis to the first appointment randomized.

Study Measures: Electronic Health Records and Visit Satisfaction Survey

We used appointment records to determine the primary outcome: whether the appointment was missed, cancelled, or completed. We selected patient characteristics as covariates and potential predictors of missed appointments based on the findings of prior studies.

 These variables included age, sex, race/ethnicity, marital status, employment, day and time of the appointment, number of comorbid conditions in the 2 years prior to the visit,

 health insurance type, duration of enrollment in KPCO (a proxy for the level of familiarity with the system), lead time to appointment (defined as the number of days between scheduling the appointment and the date of that appointment), the sum of the number of emergency department (ED) visits and hospitalizations within KPCO or other institutions reimbursed by KPCO health insurance within 6 months of the index appointment, and the number of missed primary care appointments within 6 months prior to the index appointment.

KPCO developed and conducted ongoing surveys to assess member satisfaction after completed primary care visits. These surveys were distinct from standard surveys of patient experience, such as the Hospital Consumer Assessment of Healthcare Providers and Systems survey.

 Members who had missed or cancelled their appointments were not surveyed. The 58-item survey included a global item on visit satisfaction: “Thinking just about your visit on [date] with [clinician name], and everything that happened from the time you scheduled the appointment until the time you left the medical office, how would you rate it?” Response options were excellent, good, adequate, and fair to poor. We added 4 items to surveys administered between October 1, 2016, and December 31, 2016. These items (with response options in parentheses) were: (1) Did you receive a reminder message before your visit? (yes/no/don’t remember); (2) Approximately how many reminder messages did you receive (1/2/3/4 or more/don’t remember); (3) How satisfied were you with the timing of reminders we sent you before your visit? (extremely satisfied/very satisfied/somewhat satisfied/not very satisfied/not at all satisfied/don’t know/not applicable); and (4) How satisfied were you with the number of reminders we sent you before your visit? (same response options as for question 3).

The survey vendor attempted to complete 50 surveys from each primary care department (internal medicine, family medicine, or pediatrics) at each site. Surveys were initially administered by email. If the member did not respond, they received a reminder email. Patients without email addresses received a phone survey with up to 5 call attempts. Surveys were typically completed within a week of the visit, with a historical response rate of 35%.

We defined a 1.0% absolute reduction in the rate of missed appointments (eg, from 6.0% to 5.0%) as an operationally significant effect. We calculated that 100,000 primary care visits, equally apportioned among study arms, would detect this difference, and projected that 2 months would enable us to reach this sample size. To estimate the power to detect differences in visit satisfaction, we used pretrial data that 84% of patients rated their visit as excellent or good to determine that 726 surveys per study arm would detect a 5% change in satisfaction.

To compare patient demographics, clinical characteristics, and intervention outcomes among the 3 intervention groups, we used Wilcoxon rank sum tests for nonnormally distributed continuous variables and χ

 tests for discrete variables. We conducted a prespecified subgroup analysis to compare the effectiveness of the reminder interventions in the quartile of patients at highest risk of missing appointments with that in the 3 quartiles at lower risk. We developed a predictive model for the risk of missing an appointment based on previously described variables from the electronic health record. We excluded cancelled visits from this model. The clinical department (designated as an integrated primary care department, family medicine, internal medicine, or pediatrics) and the 25 clinic sites were entered into the model as fixed effects to address clustering of patients within departments and departments within clinic site. Linear and quadratic terms for continuous predictor variables were assessed, and both terms were included in the final model if statistically significant. Missing values for each predictor were included in the model as a separate category. Backward selection with Wald χ

 tests guided the selection of predictors. The final prediction model included an indicator variable for treatment group and all covariates with 

 values <.05. The discrimination of the model was assessed with the C statistic, defined as the probability that in any 2 individuals randomly drawn from the sample, the predicted risk of a missed appointment is higher among the individual who actually missed the appointment.

This project was approved by the KPCO Institutional Review Board with a waiver of individual informed consent. Because the project was conducted to address an operational issue, it was not registered on clinicaltrials.gov.

A total of 65,038 appointments were scheduled for 55,829 patients at 25 primary care clinics over the 2-month study period. After exclusions shown in the 

, we analyzed 54,066 visits for 54,066 patients. Text messages were sent to 41,339 individuals (76.4%), IVR calls were made to 8038 (14.9%), and 4689 (8.7%) could not be reached by either modality. Of these 54,066 patients, 18,135 were randomized to receive a 3-day reminder call, 18,029 to receive a 1-day reminder call, and 17,902 to receive both 3-day and 1-day reminders. Patient characteristics were balanced across intervention arms (

Overall, 5.2% of appointments were missed, 15.2% were cancelled, and 79.7% were completed. As shown in 

, the group receiving both 3-day and 1-day reminders had the lowest rate of missed visits (

 <.0001). The number needed to treat (NNT) was 76; in other words, 1 fewer appointment was missed for every 76 patients who received both reminders compared with those who received a single reminder. In separate comparisons between missed and completed appointments and between cancelled and completed appointments, the intervention reduced missed appointments (

<.0001) but had no effect on cancellations (

 = .53). The 3 groups did not differ in satisfaction with the number or timing of reminders or in overall visit satisfaction (Table 2). Individuals who received 2 IVR-T reminders did report receiving more reminder messages (a mean of 1.8 reminders vs 1.5 reminders in the other 2 groups; 

Text messages and telephone calls were equally effective in reducing the rate of missed appointments. Among those who received text messages, rates of missed appointments were 5.6%, 4.9%, and 4.2% for the 3-day, 1-day, and both-days intervention groups, respectively, whereas among those who received calls, rates of missed appointments were 4.8%, 4.4%, and 3.8% (

The statistical model for missed appointments included 45,873 patients after excluding 8193 individuals who cancelled appointments. Predictors of missing an appointment are shown in 

. The strongest individual predictor was the number of missed outpatient appointments within the prior 6 months. The multivariable model showed excellent discrimination (C statistic = 0.93). A model limited to linear and quadratic terms for the number of prior missed visits also showed excellent discrimination (C statistic = 0.92).

The missed appointment rate in the highest-risk quartile was 23.3%, compared with 0.4% in the 3 lower-risk quartiles. In the highest-risk quartile, 2 reminders reduced the rate of missed appointments by 4.5% compared with the 3-day reminder and by 3.7% compared with the 1-day reminder (

). The 2-reminder intervention had an NNT of 25 compared with a single reminder. In the 3 lower-risk quartiles, 2 reminders remained significantly more effective than a single reminder, but the NNT was 1328. The interaction among subgroups was statistically significant (

In this 3-armed randomized trial, we found that 2 reminders, delivered by text message or telephone 3 days and 1 day prior to a primary care visit, were more effective than a single reminder delivered either 3 days or 1 day prior to a visit. The intervention was effective despite a low baseline rate of missed appointments.

 Member satisfaction with the reminder process and the primary care visit was similar in the 3 study arms. The effectiveness of the intervention was substantially greater in patients at highest risk of missing their appointments, as defined by a multivariable prediction model.

The effectiveness of a single visit reminder in medical settings is well established.

 Less is known about the effect of varying the timing or number of reminders. One randomized trial in an endoscopy clinic found that IVR reminders 3 days versus 7 days prior to the appointment were equally effective.

 Systematic reviews have found that the effectiveness of a single reminder was comparable when delivered between 1 and 7 days prior to a visit, although they did not report any trials that directly compared the timing or number of reminders.

 Thus, our finding that 2 automated reminders were more effective than a single reminder in reducing missed appointments suggests a simple approach to increase reminder effectiveness.

 The patient-level predictors of missed appointments in Table 3 are similar to those found in prior studies.

 The C statistic of 0.93 indicates that the model was able to accurately discriminate between individuals who completed their appointments and those who missed them.

 Other organizations have also reported high predictive accuracy for such models.

 Although several sociodemographic and clinical variables contributed to the model, a model including just the number of prior missed appointments had a C statistic of 0.92, which suggests that attendance at prior appointments alone can accurately identify individuals at risk of subsequent missed visits.

Our analysis of high-risk patients was particularly informative. We developed a multivariable risk score rather than assessing intervention impact on multiple subgroups as defined by individual risk factors.

 This analysis showed that among 75% of patients, the risk of a missed appointment was less than 1% (Table 4). Although 2 reminders were slightly more effective than a single reminder in this group, the NNT of 1328 demonstrated that reminders for primary care visits were rarely beneficial. In contrast, the model identified a high-risk quartile of patients whose rate of missed appointments was 23%, comparable with rates reported in other settings.

 In this subgroup, 2 reminders were substantially more effective than 1 (NNT of 25), but they only reduced the rate of missed appointments to 20.5%. Automated text or phone messages will need to be augmented by more resource-intensive interventions, such as staff outreach or patient navigation,

 to further reduce missed appointments in these individuals.

The study had several strengths. It demonstrated that randomization can be used to rapidly address questions of operational importance using data from administrative and clinical sources. The marginal costs of the intervention were low because operational data, the IVR-T system, and the survey protocol were already in place. The results are consistent with those of other recent studies and systematic reviews

 and should be replicable in other healthcare systems that provide automated reminders by telephone or text message.

The study also had limitations, many of which illustrate the constraints of conducting randomized trials in an operational environment. Our intervention took place within a single integrated delivery system with a low baseline rate of missed appointments. Thus, our findings may not be generalizable to other settings. Fewer individuals were randomized into the trial than projected. Although the sample size could easily have been increased by extending the trial, we honored our original commitment to organizational leaders to complete the intervention within 2 months. Consequently, the number of completed surveys was also smaller than predicted, limiting statistical power to identify differences in satisfaction. Because only individuals who kept their appointments were included in the visit satisfaction survey, we could not assess whether those who missed or cancelled their visits had different attitudes about reminders. Additionally, although the response rate was not calculated in our study, about 35% of members typically respond to visit satisfaction surveys, raising concerns about response bias. The study was conducted over 2 months, and we did not assess the sustained effectiveness of the intervention. Some variables in our prediction model, such as the number of prior outpatient appointments, ED visits, and hospitalizations, may be more difficult to assess outside of integrated healthcare delivery systems. Although patients who made appointments online received email reminders in addition to IVR-T reminders, these individuals were likely distributed equally among the 3 study arms and did not affect the relative differences we observed. Finally, we did not determine whether appointments cancelled after the 3-day reminder were more often filled by other patients than those cancelled after the 1-day reminder.

If other studies confirm that multiple reminders are more effective than a single reminder in reducing missed appointments, system leaders will face the decision about whether to send multiple reminders to all patients with upcoming appointments or restrict this approach to patients at highest risk. An argument in favor of the first approach is that multiple reminders did not appear to reduce visit satisfaction. However, if local satisfaction surveys suggest that patients view repeated reminders as unnecessary or intrusive or if an external IVR-T vendor bases their charges on the number of reminders sent, restriction of multiple reminders to high-risk patients might be justified despite the additional analytic effort necessary to implement the risk algorithm in real time.

In conclusion, we found that multiple text message or telephone reminders were more effective than a single reminder in reducing missed primary care visits, particularly in patients at highest risk. This study also demonstrates the value of a learning health system collaboration between operational leaders and researchers to address pragmatic questions of immediate concern.

 Institute for Health Research, Kaiser Permanente Colorado (JFS, MRS, SX), Denver, CO; Kaiser Foundation Health Plan of Colorado (JZD, AK), Denver, CO.

 This project was funded by the Kaiser Permanente Health Plan of Colorado.

 Dr Steiner received internal grant support from Kaiser Permanente to conduct this study. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (JFS, MRS, JZD, AK, SX); acquisition of data (JFS, MRS, JZD, SX); analysis and interpretation of data (JFS, MRS, SX); drafting of the manuscript (JFS, AK, SX); critical revision of the manuscript for important intellectual content (JFS, MRS, JZD); statistical analysis (SX); provision of patients or study materials (JFS, MRS); obtaining funding (JFS, AK); administrative, technical, or logistic support (AK); and supervision (JFS, SX).

 John F. Steiner, MD, MPH, Institute for Health Research, 10065 E Harvard Ave, Ste 300, Denver, CO 80231. Email: john.f.steiner@kp.org.

1. Macharia WM, Leon G, Rowe BH, Stephenson BJ, Haynes RB. An overview of interventions to improve compliance with appointment keeping for medical services. 

. 1992;267(13):1813-1817. doi: 10.1001/jama.1992.03480130129038.

2. Free C, Phillips G, Watson L, et al. The effectiveness of mobile-health technologies to improve health care service delivery processes: a systematic review and meta-analysis. 

. 2013;10(1):e1001363. doi: 10.1371/journal.pmed.1001363.

Articles

Two text message or phone reminders were more effective in reducing missed primary care appointments than a single reminder, particularly in patients at high risk of missing appointments.

Optimizing Number and Timing of Appointment Reminders: A Randomized Trial

 To illustrate the use of cluster analysis for identifying sub-populations of complex patients who may benefit from targeted care management strategies.

 We identified a cohort of adult members of an integrated health maintenance organization who had 2 or more of 17 common chronic medical conditions and were categorized in the top 20% of total cost of care for 2 consecutive years (n = 15,480). We used agglomerative hierarchical clustering methods to identify clinically relevant subgroups based on groupings of coexisting conditions. Ward’s minimum variance algorithm provided the most parsimonious solution.

 Ward’s algorithm identified 10 clinically relevant clusters grouped around single or multiple “anchoring conditions.” The clusters revealed distinct groups of patients including: coexisting chronic pain and mental illness, obesity and mental illness, frail elderly, cancer, specific surgical procedures, cardiac disease, chronic lung disease, gastrointestinal bleeding, diabetes, and renal disease. These conditions co-occurred with multiple other chronic conditions. Mental health diagnoses were prevalent (range 28% to 100%) in all clusters.

 Data mining procedures such as cluster analysis can be used to identify discrete groups of patients with specific combinations of comorbid conditions. These clusters suggest the need for a range of care management strategies. Although several of our clusters lend themselves to existing care and disease management protocols, care management for other subgroups is less well-defined. Cluster analysis methods can be leveraged to develop targeted care management interventions designed to improve health outcomes.

	This study illustrates the use of cluster analysis to identify sub-populations of complex patients for potential targeted care management within an integrated health maintenance organization.

		Among a cohort of adults with multimorbidity and high healthcare utilization, we identified 10 clinically relevant clusters of complex patients.

		While care management protocols may already exist in many healthcare settings for some common clusters, other clusters identified present opportunities for new or enhanced care management.

		Data mining methods such as cluster analysis can be applied in other settings where electronic diagnosis data are readily available.

	By 2020, over 81 million persons in the United States will have 2 or more chronic conditions.

 Multimorbidity results in adverse health outcomes and higher healthcare costs, and challenges current models of care delivery.

 Care management has the potential to improve health outcomes for persons with multimorbidities. However, most disease and care management strategies have been developed to improve specific health outcomes for populations defined by single diseases or specific circumstances (such as hospital discharge).

 There is a need for strategies that can identify sub-populations with multiple, interacting diseases, in order to provide them with appropriate and relevant care management support.

	Investigations to identify these populations of complex patients have traditionally relied upon multivariable regression analyses to identify patient-level characteristics (such as demographics and diseases) that predict the outcome of interest (such as hospitalization).

 As compared with investigations that use multivariable regression analyses to identify individual disease predictors of specific outcomes, data mining techniques provide an opportunity to empirically identify groups of patients with similar patterns of multimorbidities. One such technique, cluster analysis, refers to classification methods that are used for discovering groups or “clusters” of “highly similar entities” within data sets.

 Cluster analyses are common in psychology, sociology, and marketing research, and the methods have been used to a limited extent in health services research.

 While cluster analyses previously have been used to discover patterns of multimorbidities,

 in this study we demonstrate the application of such methods for identifying clusters of patients with high utilization that may suggest opportunities for enhanced care management in a managed care setting.

	We used cluster analysis to explore a large, 2-year cohort of health maintenance organization members with 2 or more chronic conditions. We hypothesized that within a large, complex patient population, cluster analysis would reveal groups of patients with distinct patterns of comorbid conditions. Although some of these subgroups would be characterized by well-known patterns of co-occurring medical conditions with established care management strategies, other subgroups would reveal combi-nations of comorbidities that might benefit from new, proactive, and targeted care management.

	Kaiser Permanente Colorado (KPCO) is an integrated, not-for-profit health maintenance organization. During the years studied (2006 and 2007), KPCO had approximately 430,000 members. This study was approved by KPCO’s Institutional Review Board.

	The study population consisted of KPCO members 21 years or older on January 1, 2006, categorized in the top 20% of total cost of care in both 2006 and 2007, each with 2 or more of 17 common chronic medical conditions. Annual cost estimates combined general ledger costs with direct and indirect utilization-related costs to provide cost-of-care estimates for KPCO members.

 We excluded members with a long-term care facility stay, chronic kidney dialysis, or an inpatient visit of greater than 30 days during the 2 years based on the premise that their unique and significant care management needs are likely to already be well defined. Six extremely high cost outliers were also removed from the cohort.

	We compiled a list of 17 chronic medical conditions based on prevalence in the general population, prevalence in our specific cohort, and a literature search of conditions likely to predict hospitalization or adverse health outcomes in complex patients.

 The selected conditions were diabetes, chronic obstructive pulmonary disease (COPD), chronic kidney disease, stroke, obesity, dementia, fall, hip fracture, chronic pain, skin ulcer, orthopedic surgery, back surgery, abdominal surgery, gastrointestinal bleeding, cancer (excluding non-melanoma skin cancer), cardiac disease (which included coronary artery disease and congestive heart failure), and mental health conditions—primarily depression, but also including generalized anxiety and bipolar disorders. Determinations of whether cohort members had a given condition were based on inpatient and outpatient 

International Classification of Diseases, Ninth Revision (ICD-9)

 diagnosis and procedure codes in 2006 and 2007. In addition, we used KPCO’s cancer registry to determine cancer diagnoses in 2005, 2006, and 2007. We considered a cohort member to have obesity if they had an 

 diagnosis code for obesity or a median body mass index (BMI) greater than or equal to 30 in 2006 and 2007. BMI data were available for 98.6% of cohort members; if a cohort member did not have a BMI value or an obesity diagnosis in 2006 or 2007 then we did not consider them to be obese.

	SAS version 9.2 (SAS Institute, Cary, North Carolina) was used for all analyses. We described demographic attributes, healthcare utilization, comorbidity score (using the Quan adaptation of the Elixhauser comorbidity index),

 and prevalence of clinical conditions within the cohort using frequencies and medians with 25th and 75th percentiles.

 We used agglomerative hierarchical clustering to identify clinically relevant groups of cohort members with similar multimorbid conditions. With this method of cluster analysis, each cohort member starts as its own cluster. The 2 most similar clusters are merged and this new cluster replaces the 2 former clusters. The process continues until there is only 1 cluster containing all observations.

 After the clustering algorithm is run, the user must select the appropriate cutoff point for the number of clusters desired based on clinical importance or other pre-specified criteria.

 Various algorithms are available for cluster analysis. For this study, we used Ward’s minimum variance method as the primary algorithm. With this algorithm, every possible cluster combination is considered at each step of agglomerative hierarchical clustering, and the combination thatresults in the smallest addition to the error sum of squares is selected.

 Ward’s method is a widely used algorithm which minimizes the variance within clusters, and is also known to produce clusters of similar sizes.

 We compared results from Ward’s method to results using the flexible beta algorithm, where the user sets different levels of beta, and beta values less than zero optimize the dissimilarity between clusters.

 In the analytic data set, the presence or absence of each of the 17 conditions was represented with a 1 or 0 for each cohort member. We first randomly split the full analytic data set into 2 equally sized data sets. We then converted each split data set into a dissimilarity matrix using Jaccard’s coefficient. This is an appropriate distance measure for clinical conditions, as it considers the number of conditions that 2 people have in common and ignores conditions that neither person has.

	We ran our primary algorithm, Ward’s minimum variance method, on both split data sets. The pseudo F, pseudo T, and 

 statistics were examined for different numbers of clusters to identify possible clustering solutions.

 These statistics pointed to several desirable numbers of clusters, and membership in these clusters was described by examining the prevalence of each condition in the cluster. Cluster membership was compared between the 2 split data sets to assess the consistency of the clustering process. Since cluster membership was similar between the 2 split data sets, thus reinforcing the stability of the algorithm in this population, Ward’s algorithm was run on the entire data set. We subjectively determined that a 10-cluster solution produced the most clinically relevant clusters. For comparison, we then produced 10 cluster solutions using the flexible beta method, with beta set at -0.25 and -0.5, and compared these results with Ward’s method. This confirmed that Ward’s algorithm resulting in 10 clusters appeared to be the most parsimonious solution and provided the most clinically relevant groups. We then described the 10 clusters by the number of cohort members in the cluster, median age of cluster members, and percentage of cluster members with the most prevalent conditions in that cluster. We also described relative cost of care ratios for each cluster.

 provides descriptive demographic and disease characteristics of the study cohort (n = 15,480). The median age of cohort members was 65 years, and 59.1% of cohort members were women. Cohort members had a median of 5 chronic medical conditions (including, but not limited to, the 17 conditions included in the cluster analysis).

	In this study we tried 3 clustering algorithms—Ward’s, flexible beta with beta set to -0.25, and flexible beta with beta set to -0.5. Full results from the 3 methods are in the 

. All 3 methods identified some of the same distinct clusters, including clusters of patients with chronic pain and mental health conditions and gastrointestinal bleeding and mental health conditions. However, we preferred Ward’s method because the flexible beta methods tended to produce many small homogeneous clusters and several large heterogeneous clusters, while Ward’s method produced more similarly sized and more homogeneous clusters. Eight of the 10 clusters produced by Ward’s method had what we referred to as “anchoring” conditions, or chronic conditions shared by almost all (>98%) of cluster members. We considered such anchoring conditions potentially useful given the ultimate goal of proactively identifying characteristics of persons who may benefit from targeted care management. With the flexible beta results, only 5 and 7 of the clusters from the method with beta set at —0.25 and –0.5, respectively, had anchoring conditions.

 provides a description of cluster membership using Ward’s algorithm. Of the 10 clusters, several described groups of patients with well-known and highly prevalent comorbidities such as diabetes and obesity, cardiac disease and obesity, renal disease and diabetes, and the multiple diseases and conditions common in the frail elderly. Other clusters were defined by combinations of comorbidities that are less frequently described including abdominal and orthopedic surgeries with obesity. Two clusters included individuals who were substantially younger than the others: mental health conditions and chronic pain, and mental health conditions and obesity. Two single conditions were highly prevalent across all groups: mental health conditions (primarily depression) and obesity. These conditions were present in all clusters with prevalence rates that ranged from 28% to 100% (mental health conditions) and from 38% to 100% (obesity). Prevalence of all 17 conditions in each cluster is available in the Appendix, as are relative cost ratios.

	In this investigation, we demonstrated the use of cluster analysis to identify distinct subgroups of patients with specific combinations of co-occurring conditions in a managed care population. Exploratory by nature, cluster analysis provided a unique method for investigating the co-occurrence of multiple conditions. When working with large data sets, simple tabulations of the number of chronic conditions can be difficult to interpret. In our population in which cohort members had at least 2 of 17 conditions of interest, there are theoretically over 100,000 different possible combinations of coexisting conditions. In actuality, our cohort only had 1507 different combinations of the 17 conditions of interest. Even this number of naturally occurring clusters is much too large to allow identification of targeted care management opportunities.

	Compared with people with only 1 chronic condition, persons with multiple chronic conditions experience worse health outcomes, including, but not limited to, lower quality of life, poorer functional status, and excess morbidity and mortality.

 Chronic conditions do not occur in isolation, yet many existing care management strategies are directed toward single conditions. Cluster analysis provided a data-driven approach to identifying 10 clinically relevant groups of patients with patterns of comorbidities that could be targeted with new, enhanced care management strategies.

	In our analysis, the largest cluster was characterized by older age coupled with conditions associated with frailty, such as stroke and dementia, along with chronic conditions including diabetes, obesity, and cardiac disease. Along with this group, our analysis produced several other clusters of wellknown, high-cost co-occurring conditions for which chronic disease management programs are widespread, such as cancer with obesity and/or mental health conditions (primarily depression) and cardiac disease co-occurring with obesity and diabetes. Identification of these expected groups in our analysis provided additional assurance of the validity of our data mining method.

	Our analysis produced several clusters that suggest potential care management opportunities in our managed care setting, especially in relatively younger adults. Two clusters from this cohort of individuals had median ages of 50 years (cluster 4) and 55 years (cluster 1). All 1676 members of cluster 4 had both mental health conditions and obesity, but none of the other 15 conditions of interest. This highly homogeneous cluster represented 10.8% of the entire study cohort. Individuals in cluster 1 were characterized by chronic pain co-occurring with mental health conditions and/or obesity. Chronic pain is associated with depression and anxiety, and has been linked to increased utilization of health resources.

 Coordinated care strategies to relieve depression and improve chronic pain outcomes have been evaluated to a limited extent.52,53 Cohort members in cluster 1 also had the third highest median total cost of care of all clusters, after the clusters of cancer and surgical patients. Identification of these 2 clusters supports the notion that multimorbidity and patient complexity are not limited to the elderly.

 It is likely that the relatively younger adults in these 2 groups would benefit from care management that emphasizes integrated depression care.

 Finally, we noted a high prevalence of mental health conditions and obesity across all clusters, suggesting that these 2 conditions should be addressed in most, if not all, comprehensive care management programs.

	Our study has several limitations. Cluster analysis is an exploratory classification method that is supported by a relatively small body of statistical evidence, and different clustering algorithms produce different results.

 Knowing this inherent problem, we tried several algorithms. We chose the ultimate clustering solution based on subjective review and clinical relevance of cluster membership. In addition, agglomerative hierarchical clustering forces everyone into a cluster. There are likely small groups of patients within some of our clusters who have unique combinations of comorbid conditions that differ from the majority of people in that cluster. To assess the consistency of the clustering process within our patient population, we compared results from a randomly split data set of cohort members in 2006 and 2007. To further assess the stability of these clusters over time, analyses should be conducted on cohorts from different years. As with any investigation, the characteristics of our clusters are limited to our data and setting. Replicating these analyses in other settings and other patient populations may potentially reveal different clusters. However, these differences would and should inform management strategies specific to populations in those settings.

	An additional limitation is that we identified the 17 conditions of interest based on electronic diagnosis and procedure data in 2006 and 2007, with the exception of cancer, for which we also included 2005 data and used a cancer registry, and obesity, for which we used BMI data. We anticipate that for most chronic conditions, such as COPD or diabetes, the member would have had at least 1 diagnosis in these years; however, the sensitivity and specificity of diagnostic data are imperfect, leading to potential misclassification of conditions. Lastly, it is likely that individuals in our cohort carried multiple other diagnoses not specifically assessed. Based on the Quan adaptation of the Elixhauser comorbidity index, cohort members had a median of 5 chronic conditions.

 We selected the 17 conditions examined in this study based on prevalence in the general population and in our specific cohort, and on prior knowledge of or association with increased healthcare utilization; however, it is likely that including other conditions in the analysis would result in different clusters or subgroups.

	In this investigation, we demonstrated how cluster analysis can be used to identify homogeneous groups of complex patients from a large heterogeneous population. Such data mining methods can be applied in other settings where electronic diagnosis data are readily available. Alternatively, it is possible to use the conceptual results from this investigation to increase awareness of the need for a diverse array of care management services for individuals with high levels of healthcare utilization. However, further understanding of the care management needs of clusters of patients with similar comorbidities is warranted before designing specific tailored interventions. For example, the specific needs of persons with mental health conditions and obesity should be explored in detail in order to develop relevant care management strategies for that group.

	Cluster analysis methods can be leveraged for targeted care management interventions designed to improve health outcomes and potentially lower healthcare costs. This cluster analysis of a large cohort of individuals with multiple morbidities suggests that complex patients with high healthcare utilization represent a highly diverse group of individuals. While some subgroups may respond well to existing approaches to care management (such as those designed for populations with diabetes, cardiac disease, and frailty), others will likely require new and/or individualized care management strategies to achieve favorable health outcomes.

 From Institute for Health Research (SRN, JFS EAB), Kaiser Permanente Colorado, Denver; Department of Family Medicine (EAB), University of Colorado, Aurora.

Cluster analysis can aid in identifying subgroups of patients with similar patterns of comorbid conditions for targeted care management.

Identifying Subgroups of Complex Patients With Cluster Analysis

: To examine the association between primary care provider turnover in managed care organizations and measures of member satisfaction and preventive care.

: Retrospective cohort study of a national sample of 615 managed care organizations that reported HEDISÃ‚Â® data to the National Committee for Quality Assurance from 1999 through 2001.

: Multivariable hierarchical regression modeling was used to evaluate the association between health plan primary care provider turnover rate and member satisfaction and preventive care measures, including childhood immunization, well-child visits, cholesterol, diabetes management, and breast and cervical cancer screening, adjusting for patient and organizational characteristics, time, and repeated measures.

: The median primary care provider turnover rate was 7.1% (range, 0%-53.3%). After adjustment for plan  characteristics, health plans with higher primary care provider turnover rates had significantly lower measures of member satisfaction, including overall rating of healthcare (P < .01). A 10% higher primary care provider turnover rate was associated with 0.9% fewer members rating high overall satisfaction with healthcare. Health plans with higher provider turnover rates also had lower rates of preventive care, including childhood immunization (P = .045), well-child visits (P = .002), cholesterol screening after cardiac event (P = .042), and cervical cancer screening (P = .024). For example, a 10% higher primary care provider turnover was associated with a 2.7% lower rate of child-members receiving well-child visits in the first 15 months of life.

: Primary care provider turnover is associated with several measures of care quality, including aspects of member satisfaction and preventive care. Future studies should evaluate whether interventions to reduce primary care provider turnover can improve quality of care and patient outcomes.

Analysis of a national sample of managed care organizations found associations between higher primary care provider turnover and lower ratings of member satisfaction and lower ratings of preventive care measures.

Managed care organizations should view primary care provider turnover as an associate of quality-of-care indicators.

Measures implemented at the organizational level to decrease primary care provider turnover may potentially increase quality measures, specifically patient satisfaction, timeliness of care, and effectiveness of care, with the goal of improving patient retention and overall quality of care.

Continuity of care is known as an important element of quality of care.

 It has been associated with higher patient ratings of their healthcare.

Patients who have a continuous relationship with their provider have more trust in their provider, believe that their provider shows respect and listens well, and give higher ratings for coordination of care.

 Continuity  of care has also been associated with higher rates of compliance with preventive care measures, such as higher rates of childhood immunizations.

 Although associations with continuity of care have been studied, little is known about associations with a specific aspect of continuity of care, namely, primary care provider turnover.

From literature, primary care provider turnover is the rate at which primary care providers leave a health plan organization. Known to be costly, healthcare turnover, including physicians, nurses, allied health personnel, and staff, is estimated at 3.4% to 5.8% of a health plan's annual budget.

 The largest cost was due to loss and replacement nurses. Because of the high cost of recruitment and orientation, health plans with higher provider turnover have higher operating costs.

 It is unknown whether primary care provider turnover is associated with quality of care. The few studies on the associations between primary care provider (physician provider and nonphysician provider) and patient satisfaction and preventive care measures have reported mixed results.

Based on the continuity of care literature, we propose that lower provider turnover rates might improve both communication and coordination of patient care.

 Further, low provider turnover rate might be associated with an increase in a physician's knowledge of a patient's needs, patients' trust in their physician, and improved provider teamwork,

 and therefore, should improve the coordination of patient care.

 This in turn would be expected to promote high quality of care, specifically higher patient satisfaction with care and improved effectiveness of care.

The goal of this study was to improve the understanding of associations between primary care provider (both physicians and nonphysician providers) turnover and quality-of-care indicators, including measures of member satisfaction and preventive care. We hypothesized that organizations with higher provider turnover would be associated with lower member satisfaction and lower rates of preventive care measures in the organizations. If found, interventions could potentially be implemented at the organizational level of care to lower primary care provider turnover and thereby improve quality of care.

Data for this study were from the National Committee for Quality Assurance (NCQA). NCQA is an independent, not-for-profit organization created to examine and report on the quality of care the public receives at managed care organizations.

 Managed care organizations voluntarily report aggregate, health plan level data to NCQA. This retrospective cohort study included all commercial managed care organizations that reported data to NCQA on variables measured during the calendar years 1998-2000.

The outcome variables were aggregated health-plan level measures of member satisfaction and preventive care. Data on member satisfaction were obtained from NCQA, which used the Consumer Assessment of Health Performance Survey (CAHPS 2.0HÃ‚Â®).

 CAHPS is a survey of member experiences administered to a random sample of health plan members at the end of each measurement year. The CAHPS measures used in this study comprised 3 domains: satisfaction with provider (overall rating of personal provider, reporting frequency of provider showing respect, reporting of frequency of provider listening carefully, and reporting of frequency of provider explaining well), satisfaction with timeliness of care (how often patient obtained a routine appointment when desired, how often patient obtained an injury/illness appointment when desired, and frequency of clinic wait time longer than 15 minutes), and overall satisfaction (overall rating of healthcare and overall rating of health plan). For this analysis, satisfaction scores were measured on a continuous scale as a percentage of members who answered favorably (ratings of 8, 9, or 10 on a scale of 0 to 10; frequencies of "always" or "usually" vs "sometimes," "rarely," or "never").

On average, 1276 satisfaction surveys were sent out for each plan. The average response rate was 48% and ranged from 15% to 86%. Because the survey response rates varied by outcome and by organization, the analyses were weighted by sample size.

Data on preventive care were obtained from NCQA's Health Plan Employer Data Information Set (HEDIS).

 HEDIS measures are collected by the health plan on specific illnesses and preventive care as a percent of eligible members. The HEDIS data chosen as outcomes in this study comprised 4 domains: well-child visits (percentages of childhood immunizations by age 2 and well-child visits within the first 15 months of life); cardiovascular secondary disease prevention measures (percentages of ÃƒÅ¸-blocker treatment at discharge from hospital after a heart attack, and cholesterol screening performed within 60-365 days after discharge from hospital for acute myocardial infarction, coronary artery bypass graft, or percutaneous transluminal coronary angioplasty); cancer screening measures (percentages of 1 cervical cancer screening among women 

 years of age in the past 3 years and 1 breast cancer screening among women 52-69 years of age in the past 2 years); and diabetes management measures (glycosylate hemoglobin control, eye examination, lipid control, nephropathy). Although other outcome variables were available, we chose these 4 domains because they cover a range of health issues (cardiovascular, cancer, diabetes) and populations (pediatric and adult).

To assure accurate rates from all organizations, NCQA required a 100% sample from organizations with fewer than 411 eligible members. Otherwise, a minimum sample of 411 was required, with some organizations reporting rates from much larger samples. In cases in which the organization was not able to report preventive care outcomes (ie, fewer than 30 members were eligible for the rate), the outcome was left as missing. In cases in which the organization had at least 30 eligible members for the rate, but chose not to report the results, the values were imputed to the lowest 10th percentile, consistent with NCQA methodology.

The independent variable of primary interest was the primary care provider turnover rate. Primary care providers included primary care physicians, nurse practitioners, and physician assistants. The turnover rate was defined as the percentage of primary care providers affiliated with the health plan as of December 31 of the year preceding the measurement year who were not affiliated with the health plan as of December 31 of the measurement year.

 The primary analysis used a combined turnover rate that was weighted by the number of each type of provider (physician/nonphysician) in the organization. A secondary analysis defined turnover using only primary care physician turnover rather than the combined physician and nonphysician turnover rate. Because the turnover rates were based on only one point in time (December 31), it is unknown at what time point during the year the providers left the organization. To help avoid the possibility of measuring the preventive care outcomes prior to turnover, we matched 1 year's turnover with next year's preventive care outcomes. Therefore, the provider turnover rate measured on December 31, 1999, was matched to preventive care outcomes measured in 2000. Consequently, the analysis of preventive care outcomes required a minimum of 2 years of data for cohort follow-up, excluding 259 organizations that reported only 1 year of data. The final study cohort for the preventive care outcome analysis consisted of 356 managed care organizations; 132 organizations with 2 years of data, and 224 organizations with 3 years. The member satisfaction outcomes, however, were measured after the end of the measurement year and therefore, could be matched with the same year's provider turnover rate. The study cohort for the member satisfaction outcomes contained all available data from 615 organizations.

The measures used for risk adjustment are listed in 

. These variables, from NCQA, included health plan characteristics such as size, market penetration, and proportion of male members within the organization. Age and sex were defined by NCQA membership characteristics, whereas race and education were self-reported from the CAHPS survey. Market penetration came from InterStudy of managed care penetration and was defined as the proportion of population enrolled in health maintenance organizations (HMOs).

 Many of the health plan characteristics, such as type of managed care organization and tax status, have been associated with quality of care outcomes and with provider satisfaction.

 Member characteristics, such as age and sex, have been identified as important covariates.

 Less than 10% of the risk adjustment variables were missing. For variables that were missing, imputations were done using the Transcan function in S-plus version 6.2 (Insightful, Seattle, Wash). Transcan uses only the risk variables to model each risk variable as an outcome. The missing values are then imputed to be within the range of the nonimputed values, producing unbiased estimates of variances and covariances.

The goal of these analyses was to assess associations between primary care provider turnover rate and both member satisfaction and preventive care outcomes, adjusting for covariates. The data were repeated measure; that is, organizations report multiple times throughout the study period. Repeated measures are more closely correlated than independent measurements; therefore, the statistical analysis needed to account for this correlation.

 Furthermore, not all organizations reported data for all years, so the analysis had to account for unbalanced data. Mixed-model methodology was chosen for the analyses because it is able to model continuous, normally distributed outcomes with unbalanced data and adjust for repeated measures at the managed care level.

First, univariate mixed models were developed for each satisfaction and preventive care outcome. Next, multivariable models were developed that accounted for year and possible confounders (

) for each satisfaction and preventive care outcome. Although several outcomes were studied, no adjustment for multiple comparisons was performed because each of the hypotheses was developed a priori based on clinical experience. The primary analysis was performed on the imputed data set. As a sensitivity analysis, the models were also analyzed on the nonimputed dataset. The results were similar and are therefore not presented in this report. SAS software version 9.0 (SAS Institute Inc., Cary, NC) was used for all data analyses.

The distribution of primary care provider turnover is shown in the 

. Primary care provider turnover rate was not normally distributed. The median was 7.1% (range, 0%-53.3%). The denominators used to create these rates ranged from 61 to 11 303 providers. Although not outside the range of plausibility, the 53.3% was an outlier, with the next highest value of 36.7%. Elimination of the outlier did not change the results.

The average health plan size was 185 005 members. Most plans were for profit (78%) and most often point-of-service (POS)/HMO combined plans (58%). The plans had been in business for 14 years on average with market penetration averaging 29%. Overall, plan membership averaged 48% male and 74% Caucasian. Associations between covariates and provider turnover are shown in 

. Organizations with higher turnover rates were more often in the East North Central and South Central regions. Organizations with higher turnover were more often HMOs than POS or combination plans, had fewer members with greater than high school education, and had more non-Caucasian members.

Provider Turnover and Member Satisfaction

In multivariable analyses, primary care provider turnover was significantly associated with most of the member satisfaction measures (

). Higher primary care provider turnover was significantly associated with a lower rating of personal doctor (P = .002), lower ratings of provider's communication skills (listening, explanations, show of respect; P < .001 for each), lower rating of getting appointment for injury/illness (P < .001), lower rating of wait time in clinic (P = .019), and lower overall rating of health plan (P < .001) and healthcare (P = .001). A 10% higher primary care provider turnover rate was associated with 0.9% less members rating their overall satisfaction with their healthcare greater than 7 out of 10 (ie, highly satisfied) after controlling for plan characteristics (

). Although, primary care provider turnover was not significantly associated with getting appointment for routine care (P = .08), it was directionally similar. Multivariable analysis using primary care physician-only turnover yielded similar results.

In multivariable analyses, higher primary care provider turnover was significantly associated with lower rates of childhood immunization (P = .045), lower rates of well-child visits in the first 15 months of life (P = .002), lower rates of cholesterol screening (P = .042), and lower rates of cervical cancer screening (P = .024) (

). A 10% higher provider turnover was associated with a 2.7% lower rate of child members receiving well-child visits in the first 15 months of life, a 1.5% lower rate of childhood immunizations, a 1.7% lower rate of cholesterol screening after a heart attack, and a 1.2% lower rate of cervical cancer screening after adjusting for plan characteristics. Primary care provider turnover was not significantly associated with rates of ÃƒÅ¸-blocker treatment or rates of breast cancer screening. Similarly, primary care provider turnover was not significantly associated with any of the diabetes management measures. Multivariable analysis using primary care physicianonly turnover and preventive care outcome had similar results.

The goal of this study was to assess the association between primary care provider turnover and multiple quality-of-care indicators in a large national sample of managed care organizations. We found that higher provider turnover rates were associated with lower member satisfaction ratings of their personal doctor and their provider's ability to listen, to explain, and to show respect for their patients. Higher provider turnover rates were associated with lower satisfaction with getting appointments when desired, wait time in clinic, and overall ratings of healthcare and health plan. Furthermore, organizations with higher provider turnover rates reported lower childhood immunization and lower well-child visit rates, lower cholesterol screening rates, and lower cervical cancer screening rates. In contrast, no associations were found between provider turnover and rates of ÃƒÅ¸-blocker usage, rates of breast cancer screening, or rates of diabetes management. These results remained whether provider turnover rate was defined by physician-only providers or a combination of physician and nonphysician providers.

Our results align with the continuity of care literature, suggesting that providers who remain in an organization are able to build and maintain a relationship with their patients and, consequently, have more satisfied patients. Patients who have a more continuous relationship with their provider rate higher trust in their provider, believe their provider shows respect and listens well, and rate coordination of care and overall quality of care higher.

 Further, our study found an association between high provider turnover and poor satisfaction ratings of clinic wait time. Perhaps this finding is reflective of the more immediate turmoil in the aftermath of a provider leaving an organization. This study expands existing continuity of care literature by demonstrating an association between the specific aspect of continuity of care, provider turnover, and multiple measurements of member satisfaction.

Previous studies have found similar associations between continuity of care and well-child measures, including immunization rates and well-child visits.

 However, the published findings on associations between continuity of care and cardiovascular secondary disease prevention measures and cancer screening measures are not as consistent.

Particularly in the cancer screening literature, results have been mixed as to whether the association was with the site of care or the provider of care.

 Furthermore, much of the cardiac and cancer literature focused on collaboration between specialists and primary care providers, suggesting that less reliance on a primary care provider is necessary.

 Our findings of associations between primary care provider turnover and cholesterol screening and cervical cancer screening rates suggest that a primary care provider, rather than a specialist, monitor these specific preventive care measures. Given the width of the 95% confidence intervals, the findings of no association between primary care provider turnover and ÃƒÅ¸-blockers administered after a myocardial infarction, breast cancer screening rate, or diabetes management are inconclusive. However, no association would indicate the use of disease management programs (such as, postcard/computerized reminders and/or collaborative care between a primary care provider and a specialist, such as, a cardiologist).

Although limited by the study design, our findings may be viewed from a health plan perspective. For instance, because associations have been found between member satisfaction and patient retention in a managed care delivery system,

 it may be hypothesized that higher primary care provider turnover rates, associated with lower member satisfaction, might therefore lead to worse patient retention. Similarly, studies have found associations between increased preventive care usage and decreased morbidity and mortality related to those measures.

 Although the magnitudes of associations among the preventive care measures found in our study are modest when viewed separately, collectively they could potentially have a large impact on preventive care usage within a managed care organization. Overall, the associations found in this study suggest the importance of monitoring primary care provider turnover as a potential measure of quality.

Primary care provider turnover rates are potentially modifiable. Previous literature has reported that giving physicians authority to care for their patients, including the ability to hospitalize, order tests/procedures, and obtain referrals for patients, is associated with a reduction in primary care provider turnover.

John F. Steiner, MD, MPH

Articles

Optimizing Number and Timing of Appointment Reminders: A Randomized Trial

Identifying Subgroups of Complex Patients With Cluster Analysis

Primary Care Provider Turnover and Quality in Managed Care Organizations

Managed Care for Uninsured Adults: The Rise and Fall of a University-Based Program