Christine Vogeli, PhD

 Appropriate selection of patients is key to the success of care management programs (CMPs). Hybrid patient selection approaches, in which large data assets are culled to develop a list of patients for more targeted clinical review, are increasingly common. We sought to describe the patient and practice characteristics associated with high-risk patient identification and selection for a CMP during clinical review, and to explore variation across primary care practices. 

Standardized estimates of Medicare beneficiaries identified as high risk for poor outcomes and high medical expense, and appropriate for a CMP within a large Pioneer Accountable Care Organization, were developed using mixed effects logistic models. Study subjects were 2685 Medicare beneficiaries aged over 18 (includes individuals eligible for Medicare due to a disability) aligned to 35 primary care practices in 2013.

 Independent predictors of patient identification as high risk include older age; higher risk score; recent increases in medical conditions; higher numbers of medical hospitalizations, skilled nursing facility days, and primary care physician visits; and shorter relationships with the primary care physician. Older age, and lower income, but no prior hospice use were independently associated with patient selection for a CMP among the subset of patients identified as being high risk. Adjusted predicted percents of high-risk patients varied significantly across practices overall and for 5 of the 6 patient characteristics that were independently associated with identification as high risk. 

Inconsistency in high-risk patient identification and selection for a CMP may reflect differences in practice resources, but also highlight the need for continual training and feedback in order to protect against unintentional biases.

	Clinician review within hybrid case selection approaches includes decisions about which patients are high risk and, among high-risk patients, identification of the subset of patients most appropriate for a care management program (CMP). 

		Primary care physicians were more likely to recommend high-risk patients who are poorer, older, and not using hospice services for CMP. 

		Significant variations in adjusted proportions of patients selected for care management across practices may reflect differences in practice resources, but may also introduce bias into the process.

		Continual training and feedback to practices on their review decisions may help identify missed patient opportunities.

The observation that health expenditures are concentrated within a relatively small subset of the population is well known by now, with 5% of the population accounting for 50% of medical expense in any given year. One-third of the individuals in this group—frequently those with poor health status and multiple chronic conditions&mdash;remain high-cost the following year,

 making them appropriate for primary care—based care management approaches

 However, although effective care management targeted at the right individuals has been shown to moderate costs and utilization,

 inconsistent results have highlighted the need to appropriately select patients.

There are a host of available case-finding tools and algorithms that can be used to identify individuals appropriate for care management; nevertheless, there is also increasing understanding that these tools have limitations.

 Many case-finding tools are proprietary “black boxes” that create a single score to rank patients according to risk.

 Additionally, these tools can vary considerably in terms of their input criteria, and there is no standardization of definitions across tools

—most importantly, the input criteria cannot be modified to meet specific program characteristics and operational realities.

Data timing and completeness are a challenge for all case-finding algorithms, especially those based on paid claims. Delays in data capture can result in the identification of patients who are no longer enrolled within the health plan or practice, as well as identification of patients who no longer require intervention.

 Incomplete data, such as removal of substance use, HIV/AIDS, and in some states, mental health claims,

 paint incomplete pictures of patient needs. In addition, demographic and psychosocial characteristics are often incomplete or missing from readily available administrative data (eg, poverty, education, living situation). The absence of patient psychosocial data is particularly challenging, as these data could help to identify cohorts of patients who would be most likely helped by care management programs (CMPs)

; for example, addressing cost-related medication underuse among low-income patients may improve diabetes control and reduce inappropriate emergency department utilization.

 Although many healthcare systems do collect psychosocial data on patients, system alignment issues often preclude use of more granular and descriptive data for case finding.

Because of these limitations, case finding based on administrative data is typically not the only method used.

 Patient health-risk assessments and clinician referrals are techniques frequently used to identify patients for case management and can provide more detailed and actionable data, but they require time and resources to implement.

 Often, multiple case-finding methodologies, including predicted risk scores, utilization triggers/markers, medication adherence and gaps in care, and clinician referrals, are used in concert.

Case-finding approaches that rely on multiple methodologies are typically termed “hybrid approaches” and can vary widely, but generally refer to the use of multiple methodologies—often in sequence&mdash;to refine case finding. Within this evaluation, we studied a sequential approach to case finding that uses an algorithm to comb through administrative claims and identify a preliminary set of potentially high-risk patients. This preliminary set of patients is then reviewed by physicians and nurse care managers who make the final decision of whether to refer a patient for care management. This sequential hybrid approach is thought to be reliable

Clinician referral, within the boundaries of a pre-selected list, focuses the referral process and reduces the burden on clinicians, while at the same time ensuring primary care practice and physician (PCP) involvement. This involvement may increase buy-in of the program, which is especially important for programs embedded within physician practices. Clinician input can also help to identify patients with more modifiable health risks, including those with sub-optimal health literacy and health insight, social and home environment challenges, and/or inadequate coping skills and financial resources.

 For example, primary care physicians are more likely to describe patients with poorly controlled diabetes, inadequate insurance, or with mental health or substance use disorders, as complex.

However, clinician referral may also introduce unanticipated variations in patient selection.

 Often, there are no specific criteria that physicians are asked to consider when identifying appropriate patients, leaving the door open to unintentional biases.

 Appropriate clinician referral into CMPs requires that clinicians have a clear understanding of the characteristics that place patients at higher risk for poor health outcomes, as well as the benefits the CMP could provide.

In this study, we sought to understand the unique patient characteristics associated with both patient identification as high risk for poor outcomes and subsequent clinician referral to care management. We also sought to describe variations in patient identification and selection for care management across PCPs overall, and for important patient characteristics. We examined the selection decisions during the second year of the CMP to avoid biases associated with the ramp-up of the program.

The Partners Healthcare Pioneer Accountable Care Organization (ACO)—1 of 4 Pioneer ACOs operating in eastern Massachusetts&mdash;is among the largest nationwide, and the only Pioneer ACO that exclusively aligns patients to primary care physicians. One of the signature initiatives within the Partners Pioneer ACO is the CMP. The CMP is a longitudinal program that relies on nurse care managers working in conjunction with primary care physicians to improve care quality and control healthcare utilization for referred patients. After a successful trial within the Medicare High Cost Beneficiary Demonstration Program,

 the CMP was implemented within all Partners Healthcare PCPs under the Medicare Pioneer ACO contract, starting in January 2012. Partners Healthcare uses a hybrid approach for patient selection, combining a claims-based algorithm with clinician review to identify high-risk patients who could benefit from the CMP.

The population for this study included traditional Medicare beneficiaries within the Partners Healthcare Pioneer ACO who were identified using a claims-based algorithm as potentially high risk for poor outcomes in the second year of the program (2013). By the second year, all practices had experience performing the clinical review and had 1 or more ACO patients who were actively engaged in CMP. In total, 7651 (12%) of 63,629 Medicare patients were identified by the Partners algorithm. Among individuals with a predictive risk score for future total medical expense via Optum ImpactPro software version 6.0 (Optum, Inc, Eden Prairie, Minnesota) that exceeds a minimum threshold, the Partners algorithm uses combinations of chronic conditions and patterns of healthcare utilization to identify individuals potentially at high risk and appropriate for subsequent clinical review. The Partners algorithm was developed with extensive input from an advisory committee of operational and clinical experts within the organization. During the clinical review process, PCPs receive detailed information on chronic conditions, utilization patterns, and predictive scores that triggered identification by the algorithm. PCPs had considerable latitude when selecting patients for care management, but were provided additional guidance through reference guides and regional medical directors who received orientation and training on the list review process.

In order to study clinician review decisions during the second year, this analysis excluded 2605 patients who had been identified and reviewed in the prior year, as well as 1267 patients who had moved, died before the review process, were not community-dwelling (eg, residing in long-term care), switched to a Medicare Advantage plan, or did not have a relationship with a Partners primary care clinician. Lists of the remaining 3779 patients (5.9% of 63,629 total Medicare ACO patients) were distributed to practice management organizations and, eventually, to PCPs and clinicians for final identification of individuals appropriate for the primary care—based CMP. We then further excluded 788 patients aligned to 44 PCPs that did not make differential review decisions (eg, identifying none or all patients as high risk for poor outcomes or requiring care management), as well as 306 patients who were not classified by the practice prior to the release of the 2014 list. Thus, the final study population consisted of 2685 patients who received primary care within 35 practices.

The dependent variables are the decisions made by primary care clinicians and their practices regarding whether patients were identified as high risk for poor outcomes and whether patients could benefit from CMP. 

 and include patient demographic characteristics, area-level poverty (American Community Survey 2012 5-Year Estimates

), prospective risk score and changes in the number of conditions from calendar year 2012 to the year directly preceding the first list review (May 1, 2012-April 30, 2013) (CMS-Hierarchical Condition Categories Risk Adjustment Model

), healthcare utilization (ED visits, medical hospitalizations, skilled nursing and hospice service use) in the 12 months prior, and characteristics of the patient’s relationship with their PCP. Practice characteristics include practice size, level of affiliation with the healthcare organization, whether the practice is a community healthcare center, years of experience with care management, CMP capacity, and level of practice readiness for patient-centered medical home transformation. Models were also adjusted for primary care physician characteristics, including primary care physician age, full-/part-time status, and duration of employment within the healthcare organization.

Descriptive analyses of the patient- and practice-level characteristics associated with patient identification as high risk for poor outcomes and selection for care management were explored using bivariate analyses. Standardized estimates of patient identification as high risk and selection for CMP were developed using logistic regression models with robust standard errors clustering at the practice. For regression models, we used the subset of explanatory variables that were independently predictive of either patient identification as high risk for poor outcomes or selection for care management at 

 <.20, and used a similar approach to develop standardized estimates at the practice level.

We explored the extent to which practices consistently identified patients with specific characteristics as high risk for poor outcomes. Specifically, we examined 6 patient characteristics that were independently associated with being considered high risk for poor outcomes and that were present in at least 10% of patients in our sample; similar models were used for each, as described above, to develop standardized practice-level estimates of identification as high risk among patients with and without the characteristic. Full interaction models did not enhance predictive ability and decreased the number of practices with stable estimates; therefore, we only included interactions between PCP and the characteristic of interest in our final models.

Global tests were used to test for differences in overall practice marginal estimates and interaction marginal estimates. Chi-square statistics were used to describe the relationship between overall practice marginal estimates and practice estimates for individuals with specific characteristics.

The study was approved by the Partners Institutional Review Board.

Among Medicare ACO patients identified by the claims-based algorithm and who also passed an administrative review (roughly 5.9% of total Medicare ACO patients), 86.3% were thought to be high risk for poor outcomes by their primary care providers; 77.2% of these patients were referred to a nurse-led CMP within the PCP.

A number of patient characteristics were independently associated with patient identification as high risk during the clinician review. Specifically, independent predictors of high risk include advanced age, higher prospective risk score, a recent increase in diagnosed conditions, and higher numbers of medical hospitalizations and days in skilled nursing facility (SNF) care in the prior year. Patients identified as high risk also were more likely to have more visits with their PCP in the prior 12 months, but shorter relationships with their PCP than patients not identified as high risk. The patient characteristics with the strongest independent association with being identified as high risk include age 85 years or older (93.1%), 2 or more hospitalizations in the prior year (91.5%), and 30 or more SNF days during the prior year (92.7%) (Table 1).

Among high-risk patients, advanced age was also an independent predictor of patient referral to a CMP. However, when determining whether high-risk patients would benefit from care management, clinicians were also more likely to select individuals without prior hospice use and those living in lower-income areas. The adjusted proportions of high-risk patients 85 years or older selected for care management was 80.1%, and between 81.2% and 81.9% among patients residing in areas with more than 10% of individuals living in poverty (Table 1).

Lists of patients identified by the claims-based algorithm were reviewed within 35 practices where patients received their primary care. On average, practices reviewed 76 algorithm-identified Medicare ACO beneficiaries (range = 5-248) in 2013. Adjusted proportions of patients identified as high risk and referred to a CMP varied according to practice characteristics. Specifically, after adjusting for patient characteristics and other practice characteristics, practices with 6 or more physicians (88.8%), non—community health centers (87.3%), and practices without the capacity to provide care management to all patients (93.5%) had the largest adjusted proportions of algorithm-identified ACO patients identified as high risk during clinical review. Among high-risk patients, adjusted proportions selected for care management were highest within practices employed by the health delivery organization (80.8%) and non–community health center practices (78.4%) (

After adjusting for patient and primary care physician characteristics and controlling for the clustering of patients within PCPs, practices varied significantly in terms of both the adjusted proportion of patients identified as high risk (36% to 99%; 

 <.001) and the proportion of patients referred to the CMP (18% to 96% ; 

We further explored the extent to which patients with specific characteristics were consistently categorized as high risk across practices. For this exploratory analysis, we selected the 6 patient characteristics that were both independently associated with patient identification as high risk and were present in at least 10% of the patient population. Practices varied significantly in how they categorized patients for 5 of 6 patient characteristics studied (

Hybrid patient identification approaches, that use algorithms to identify a subset of patients who warrant more careful clinical review, hold promise as a feasible method for incorporating clinician judgment and knowledge of patient psychosocial needs into case findings. However, we know little about the clinical review process, including the criteria used by PCPs to identify patients as high risk, how practices identify which high-risk patients would be appropriate for care management, and how these decisions differ across PCPs. 

Consistent with findings from our interviews of primary care physicians and care managers,

 we found that primary care teams used additional information about their patients’ clinical profiles to decide whether a patient on an algorithm-generated list was high risk. Patients identified as high risk tended to be older, sicker, and with greater healthcare utilization, including primary care visits, than patients not identified as such. Patients with shorter relationships with their primary care physician were more likely to be identified as high risk, suggesting that PCPs might either have different thresholds for identifying patients who are new to their practice as high risk or may not have had the opportunity to establish necessary supports for their newer patients. Lack of a long-term patient—PCP relationship may compromise effective communication with patients and their caregivers, which is particularly important in older, complex patients.

As a second step, primary care teams were asked to identify which high-risk patients were appropriate for care management, and when selecting, patient clinical need was far less of a consideration. In addition to patient age and no prior hospice use, patient area—level poverty was independently associated with selection of high-risk individuals for CMP services. Thus, although predicted risk and prior utilization are important for identifying patients who are high risk, they are not the primary factors associated with the decision to refer patients to a CMP.

Our study extends prior work by clearly differentiating clinical assessments of whether patients are high risk from determinations of whether high-risk patients would be appropriate for CMP services. Additionally, our findings support other studies that have found that one of the benefits of including clinician input is the ability to consider nonclinical factors, such as modifiable health risks, patient health literacy, social and home environment challenges, social support, or inadequate coping skills and financial resources.

 Patients appropriate for care management often have issues with medical decision making and care coordination in addition to poorly controlled disease.

 Physicians may also preferentially select patients who they feel are more willing to participate in a CMP.

We found considerable variation in the overall adjusted proportions of patients identified as high risk across practices, as well as significant variations in practice-level identification rates for specific cohorts of patients, suggesting that practices may not use the same criteria to identify high-risk patients. Although practices were provided with central guidance on how to perform the clinical review, patient-by-patient decisions are left to the practices.  Some of this diversity in identification of high-risk patients may appropriately reflect unmeasured differences in practice resources and access to specialty care and services, such as behavioral health specialists and addiction counselors, for their patient population. Differences could also reflect variations in program implementation, such as nurse skill levels or comfort with different patient populations. Care management, by nature, is a diverse intervention for individuals with complex healthcare needs.

 The multifaceted nature of care management not only complicates the identification of patients, but may bias selection toward patients whose needs can be met by the skills of the care manager available to the practice. Training for care managers and feedback to practices regarding the benefits and limitations of CMP may help minimize unintentional disparities in patient selection.

Articles

Hybrid approaches allow for clinician input into case finding for care management, but training and monitoring is required to protect against unintentional biases.

Implementing a Hybrid Approach to Select Patients for Care Management: Variations Across Practices

Hybrid approaches to case finding show promise as a method to increase the success of care management programs (CMPs). A large healthcare system implemented a hybrid approach in which clinicians review algorithm-generated lists of potential high-risk patients within their practice and select the patients most appropriate for the CMP.  We sought to understand the criteria clinicians used when selecting patients.

 We conducted 20 semi-structured interviews with a convenience sample of primary care clinicians and their care managers from a diverse set of practices.

 Two independent reviewers thematically coded interview responses through an iterative process.

 In addition to clinical need (eg, disease severity or multiple comorbidities), interviewees considered a number of nonclinical patient characteristics that they felt placed patients at risk for suboptimal healthcare utilization and poor adherence. These include patients’ predisposing (eg, health literacy or navigation challenges, physical vulnerabilities, insight regarding their health) and enabling characteristics (eg, social and home environment, coping skills, financial resources). Interviewees also considered patients’ existing linkages with the health system and whether other clinicians were already meeting a patient’s care needs.

In selecting patients for a CMP, clinicians considered patient characteristics that are not easily captured in standard clinical and administrative data. A hybrid approach to case finding concentrates clinician review on patients who meet standard clinical and healthcare utilization criteria, and allows clinicians to incorporate knowledge of patients’ predisposing and enabling characteristics that are not readily maintained in clinical data.

	This study identified the factors that primary care clinicians and care managers considered as they reviewed algorithm-generated patient lists to identify a final set of patients appropriate for care management. 

		In addition to the complexity of patients’ medical conditions, clinicians considered patients’ health literacy, physical vulnerabilities, social and home environment, and coping skills when selecting patients for care management. 

		Clinicians also considered the extent to which patients’ care needs were met by strong existing linkages to care providers outside of the primary care practice.

Healthcare delivery systems are increasingly entering into risk contracts which require healthcare providers to manage the utilization of defined patient populations. Care management programs (CMPs), particularly those based in primary care practices, have become a common strategy to meet the cost containment and quality targets in these contracts. Due to the concentration of costs in persons with multiple chronic conditions, these persons are often selected for care management interventions that aim to enhance the coordination of health services and improve patients’ ability to navigate the health system.

 Appropriate case finding, or the identification of patients most likely to benefit from care management, is key to maximizing the cost-effectiveness and sustainability of CMPs.

Many CMPs rely on quantitative approaches to high-risk patient identification, such as risk prediction models based on prior healthcare utilization and clinical conditions. These approaches typically rely on administrative data, such as billing claims, to predict future healthcare utilization, but are often unable to include more nuanced patient characteristics, including socioeconomic and psychosocial circumstances.

 Claims-based approaches often fall short of identifying the patients who are most likely to engage in and benefit from care management because they are unable to detect factors such as patient willingness to participate or barriers to participation.

 Moreover, use of data-driven algorithms often introduces a time lag between when patients need help and when they are identified for the program.

An alternative strategy is to rely on direct referrals from primary care physicians (PCPs) to identify appropriate patients. This strategy is based on the premise that PCPs know their patients best and can consider patient characteristics not routinely available in administrative data or medical records.

 However, PCPs vary widely in their referral patterns for such programs,

 and concerns have been raised about underselection of patients with poor existing linkages with the healthcare system.

 Finally, physicians with a poor understanding of care management goals or approaches may refer inappropriate patients to these programs.

Hybrid approaches that combine predictive modeling and clinician input have been suggested as a way to overcome the biases and limitations of the individual approaches.

 Prior research has explored the extent to which primary care clinician case finding agrees with predictive modeling case finding,

 but has not examined the added benefit of clinical review of a list of potentially high-risk patients generated from a predictive algorithm. We investigated the contribution of clinician input in the selection of high-risk patients within an active primary care-based CMP using semi-structured interviews of PCPs and their nurse care managers (CM). We were particularly interested in the criteria and patient factors clinicians used in practice when determining which potential high-risk patients within their panel should be targeted for care management. Specifically, we examined the extent to which clinicians considered: 1) patient characteristics that are and are not routinely available in clinical and billing data; and 2) existing patient linkages to other providers within the healthcare system when reviewing lists of potential high-risk patients, and identifying patients appropriate for practice-based care management. We hypothesized that clinician involvement enhanced the case selection process by allowing the inclusion of information about the patients not readily available in administrative data.

Description of Care Management Program and Patient Selection

The CMP is a primary care-based, nurse-led longitudinal care program designed to help high-risk patients (ie, chronically ill, medically complex patients) better manage their health and healthcare utilization. Approximately 400,000 Medicare and commercially insured patients 18 years or older were eligible for the CMP in 2013. CMP patient selection starts with the use of a billing claims-based predictive algorithm to identify an initial list of high-risk patients. As a first step, chronic conditions, utilization patterns and a predictive risk score for future total medical expense are generated using the prior year of paid claims using Optum ImpactPro version 6.0 software (Optum Inc, Eden Prairie, Minnesota). Combinations of specific chronic conditions and patterns of healthcare utilization, as well as overall risk score, are then used to identify the initial set of high-risk patients using an algorithm developed with input from an advisory committee of operational and clinical experts within the healthcare organization. Approximately 5% of patients are identified as potentially high risk based on the claims-based algorithm. These patients are then sorted into PCP-specific lists for clinical review. Lists provided to PCPs included the chronic conditions, utilization, and predictive risk score for each patient, as well as patient name and medical number so clinicians could easily recognize their patients. Each PCP, together with their CM, was responsible for reviewing between 1 and 39 potentially high-risk patients for 2 insurance-risk contracts in 2013, the second year of the program. Overall, clinicians selected approximately 61% of patients on the algorithm-generated list for the CMP. Clinicians were asked to classify patients on their list according to whether they were appropriate for the CMP, or they were not appropriate either because they were not considered high-risk or because the patient’s care needs were already being met. PCPs were given considerable latitude in how they classified and selected their patients for the CMP.

Using semi-structured interviews of PCP and CM dyads, we elicited and analyzed the patient characteristics that clinicians considered when selecting patients for CMP. This project was undertaken as a quality improvement initiative, and as such, was not formally supervised by Partners Healthcare’s Institutional Review Board per their policies.

We identified a convenience sample of 20 PCP and CM dyads from 18 ambulatory practices within an integrated healthcare system consisting of 270 primary care practices located in eastern Massachusetts. Physician organization leaders identified participants with the intention of maximizing variation in practice characteristics, such as the level of affiliation with the delivery system, location, and type (

). The CMP director then confirmed willingness to participate.

A physician researcher (CH) and project manager (VH) conducted all interviews, either by phone or in person, at the respective practices. Interviews lasted 45 to 60 minutes and were audio recorded with consent from all participants. All interviews were completed between May and September 2013. A research assistant (LM) transcribed the audiotapes verbatim.

Interviews followed a semi-structured, open-ended interview guide developed through an iterative process by the research team, and designed to obtain general programmatic quality improvement information, as well as information on the characteristics of patients selected for care management. Relevant interview questions included (see the 

] for the complete interview guide): 1) “How do you define a ‘high-risk’ patient and what criteria do you use to determine if a patient is ‘high-risk’?” 2) “What types of patients did you remove because they were ‘not high-risk’?” 3) “What types of patients did you remove because they have their care needs met elsewhere?”

The Andersen Model of Health Services Utilization

 provided a basic framework for organizing interview responses according to whether they reflected patient predisposition to use services (predisposing factors), characteristics that enable or impede healthcare use (enabling factors), and patient need for healthcare (need). We coded and analyzed interview responses using QSR International’s software, NVivo version 9, a qualitative data management and analysis software tool (QSR International Pty Ltd, Doncaster, Australia). In the first level of analysis, an interviewer and a research assistant (VH and LM) independently read and thematically coded all interview transcripts—a process that allows coders to organize and evaluate similar quotations by themes.

 Through an iterative process, content-related minor themes or factors were collapsed into major factors, then minor and major factors were discussed until a mutual agreement was reached. We met with clinician researchers and operational leaders to solicit feedback, present disconfirming evidence, and validate the coding scheme.

 After coding the data into this framework, we retained and summarized the major and minor predisposing, enabling, and need factors mentioned in at least 20% of interviews. Exploratory analyses examined variations in themes by practice affiliation and model.

Although limited to a single large integrated delivery system, our convenience sample of interviewed practices represents a broad range of practice affiliations, locations, and models (Table 1). The major and minor factors that PCPs and CMs considered in deciding which patients would be appropriate for care management within their primary care practice are summarized in 

 and exemplary quotations are provided in 

Most interviewees (85%) considered a patient’s ability to comprehend medical instructions and navigate the healthcare system when determining whether patients were appropriate for the CMP. Specifically, more than half (55%) of interviewed PCPs and CMs reported that they considered patient need for education about their diseases and appropriate management as they selected high-risk patients. Almost half of interviewees suggested that activated patients (ie, patients who are good “advocates for themselves,” “well-versed and knowledgeable about what they need,” and “can take care of themselves”) may not benefit from the additional resources provided by care management. On the other hand, interviewees felt that patients with linguistic or cultural barriers, as well as individuals with impaired executive function, would be more likely to benefit from the services rendered by CMs.

More than half (60%) of interviewees considered whether their patients had physical vulnerabilities, including a high risk of falling, frailty, or advanced age, when determining appropriateness for CMP. Clinicians reported that, in some cases, they would select patients with relatively stable functional status if they had physical vulnerabilities because they felt care management might help facilitate patient care in the event of decompensation.

Patient insight regarding his/her health. 

Just over half (55%) of interviewees considered the extent to which patients accurately perceived their health status and their need for assistance. This included elderly patients who were reluctant to lose their independence, younger patients in denial of their medical conditions, or patients who minimize their health needs. Interviewees felt these types of patients lacked an accurate perception of their limitations and needs, potentially leading them to refuse certain types of medical care or assistance. Interviewees stated that they would include these patients in the program, but that it might be challenging to engage and impact these types of patients.

 All interviews mentioned that they considered their patients’ social and home environment when assessing appropriateness for the CMP. Interviewees observed that relatives often serve as patients’ advocates and caretakers, helping patients navigate the health system, monitoring medication regimens, and noting changes in the patient’s health status. A large majority (90%) of interviews considered the absence of patients’ familial and social supports as reasons to include patients in the CMP. Instability in a patient’s home environment was mentioned as a consideration in a large majority (75%) of interviews, with issues ranging from addiction and verbal abuse to hoarding, as well as spousal illness, hospitalization, or recent death. Some (30%) interviewees mentioned the social isolation of living alone as an added risk factor for elderly and frail chronically ill patients. Finally, lack of access to transportation, home delivery of food, or other community resources were also considerations for CMP inclusion (30%).

Over half (65%) of interviewees noted that they would select patients with underlying mild psychological concerns, such as health anxiety, who often seek reassurance from medical professionals, whether through frequent phone calls or urgent care visits. Such patients may benefit from a CM, who could act as a point-person to help them manage anxiety related to their medical conditions without consuming limited office resources.

A minority (35%) of interviewees mentioned socioeconomic instability (ie, chronic unemployment or unstable housing), and its effect on a patient’s ability to manage his or her chronic illness(es). Medication adherence issues related to financial instability were also considered.

All interviewees considered their patients’ chronic conditions, including the severity or stage of the patient’s disease, and the patient’s complexity or combination of conditions. Interviewees considered whether complicated medication regimens placed their patients at higher risk for decline, complications, and hospitalization. Interviewees specifically highlighted patients with co-occurring medical and psychiatric disorders, such as depression, which they felt further complicated patient care and the patient’s ability to manage his or her medical condition(s).

In identifying patients appropriate for CMP, interviewees considered factors in isolation, but also described the interplay between patient factors, and how the combination of these factors either supported or impeded a patient’s ability to effectively manage his or her health.

Once clinicians determined whether patients were high-risk, interviewees then considered whether the CMP would be the right program for the patient. Interviewees considered whether the patient was already connected with other clinicians, and if the resources available through the CMP would address the patient’s needs. Specifically, they reported considering existing relationships with specialist physicians (eg, psychiatrists, cardiologists) or specialty clinics (eg, transplant, oncology), private CMs, or institutional care providers (eg, group homes, assisted living). Decisions regarding whether care needs were met relied on clinicians’ perceptions of whether there were additional resources or services the CMP could provide, such as transitions to end-of-life care. Moreover, interviewees’ understanding of the CMP also impacted their decisions about whether to include certain subgroups of patients. For instance, some interviewees excluded patients whose primary diagnosis was a psychiatric condition or substance abuse because they felt the CMP did not yet have the resources to meet these patients’ needs.

Our study evaluated the factors clinicians considered important in determining whether potentially high-risk patients within their practice would benefit from a practice-based CMP. Similar to others, we found that clinicians consider patients’ predisposing factors, including health literacy and patient self-efficacy, as well as enabling factors, such as home and family supports, when identifying patients appropriate for care management.

 We found that these factors remained important, even in our hybrid approach to case finding, in which clinician decision making is limited to a set of risk contract patients with prior high-cost care patterns and complex chronic illness. Additionally, clinicians in our investigation considered whether existing medical and community resources were already addressing patient needs and whether CMP would enhance or interfere with existing patient care networks.

Many of the factors clinicians considered in their decision making are not routinely captured in clinical or billing data, and are therefore unavailable for case finding.

 For example, patient home environment and social support, as well as patient health literacy, were factors considered by a large majority of clinicians when identifying patients appropriate for CMP. This information is not contained in billing records and is not routinely captured in electronic health records. 

Previous research found only a small subgroup of patients concordantly selected by PCPs and predictive modeling for care management.

 Compared with PCP-based case selection, a predictive modeling approach tends to identify patients who are at a higher risk of healthcare utilization and have a higher morbidity burden,

 but are less likely to be helped by care management, either because their utilization is not preventable or they are unwilling or unable to participate.

The hybrid approach in this study builds on these findings. Starting with a predictive model to identify a subset of patients that may be appropriate for CMP effectively limits the review burden on clinicians, and focuses the clinical review on the subset of patients who are enrolled in a risk contract and have chronic conditions and patterns of prior healthcare utilization that may be amenable to care management.

More systematic capture of clinical and demographic data may become standard in newer electronic medical records or health information exchanges,

 but many of the nonclinical, nuanced factors clinicians reported considering are unlikely to become standard elements of clinical documentation, either because of privacy concerns or because of their limited relevance to standard clinical care.

Incorporating clinical input in patient selection addresses some of the limitations associated with the use of claims or medical record-based algorithms for case finding, but might also introduce new biases. Physicians and practices may vary in their threshold for CMP referral or how they weigh specific criteria. For example, some practices may have existing programs or supports, such as special programs for patients with heart failure or diabetes, or close ties with cancer centers that might impact their patient selection. Variations in decision making may also be related to differences in knowledge or acceptance of team approaches to patient care. For instance, we found that some clinicians were unsure whether the CMP would have the necessary resources for patients with co-occurring mental health and substance use. More training and education as to the types of patients who would be a good fit for CMP may improve the patient selection process.

Clinicians consider a number of patient predisposing and enabling characteristics not typically available in clinical data systems when selecting high-risk patients for care management.

Clinician Considerations When Selecting High-Risk Patients for Care Management

 To report on the first year of a program using standardized rating scales within a large, multisite mental health system of care for children and to assess the validity, reliability, and feasibility of these scales.

Naturalistic follow-up of clinicians’ ratings.

Clinicians filled out the Brief Psychiatric Rating Scale for Children (BPRS-C) and the Children’s Global Assessment Scale (CGAS) at intake and discharge/90-day follow-up for all new patients.

Data were collected on 2396 patients from all 20 sites serving children in the Partners HealthCare network. Mean scores for both BPRS-C and CGAS showed worst functioning at inpatient sites, followed by Acute Residential Treatment, then partial hospital, then outpatient sites. All patients re-rated at discharge or 90-day follow-up showed a significant improvement in scores. Inter-item reliability on the BPRS-C was acceptable, with Cronbach alphas of .78 and .81. Feasibility at intake was demonstrated in that 66% of all patients had a completed form at intake. Reassessment at discharge also appeared to be feasible in more restrictive levels of care, but less feasible in outpatient sites, where fewer than 25% of all patients had a follow-up form.

This evaluation suggested that the 2 standardized measures appeared to be valid and reliable as part of routine intake and discharge/follow-up in a large child psychiatry system of care. Whether these measures are truly clinically useful remains to be demonstrated because there is at present no gold standard for assessing the quality of treatment or change caused by it.

Two standardized psychiatric rating scales used within a large, diverse, multisite mental health system of care for children were assessed after their first year of use.

The scales were determined to be valid, reliable, and feasible for assessing admissions as required by a managed care insurer.

Collecting data at follow-up/discharge was possible in about two-thirds of inpatient, partial hospital, and acute residential treatment cases, but in fewer than 25% of cases in outpatient settings.

Patient scores on both measures showed statistically significant improvements from admission/

Increasingly, individual insurers as well as governmental and regulatory agencies are requiring mental health clinicians to use standardized instruments to assess patients as a part of clinical care.

 supports the measurement of outcomes in psychiatry, but stresses the importance of using measures that are feasible in clinical settings as well as valid and reliable. We describe the experiences of a large and diverse child mental health system of care, Partners Psychiatry and Mental Health (PPMH), in the first year of implementing a uniform set of outcome measurements across all sites at all levels of care. We focused on (1) examining the extent to which outcomes measures were implemented successfully within the PPMH system and (2) the validity and reliability of the measures themselves as they pertain to this setting.

To comply with contractual terms from the Commonwealth of Massachusetts, the Massachusetts Behavior Health Partnership (a Medicaid managed care carve-out company) required providers to use standardized, validated instruments to assess and track outcomes for all new patients. Beginning July 1, 2005, these patients had to be rated at intake/admission and at regular 90-day follow-up intervals and/or discharge. The mandate also required clinicians to use the information gathered from these forms in patient feedback and treatment planning.

In order to gather systemwide data, all PPMH sites agreed to utilize the same instruments and procedures. In choosing measures, a steering committee of staff from the sites considered ease of use, low cost (public domain preferred), and broad applicability across a wide range of ages, diagnoses, and levels of care. After assessing the available options, the committee chose 2 measures: the Brief Psychiatric Rating Scale for Children (BPRS-C)

 as a measure of symptom severity and the Children’s Global Assessment Scale (CGAS)

The BPRS-C has been used in a number of studies of research and clinical populations including patients with bipolar disorder, schizophrenia, and psychotic disorder and other studies requiring a multidimensional assessment of symptoms.

 The measure consists of 21 discreet symptom areas, each rated for severity

 along a 7-point Likert scale. A total score ranging from 0 to 126 is computed by summing all 21 items, with higher scores indicating greater symptom severity. Changes in this total score are used when evaluating pretreatment to posttreatment change.

 The reliability and consistency of the BPRS-C have been tested over time, and it has become an integral measure in diverse psychiatric venues.

 The CGAS is a practical complement to the BPRS-C because it is used for the assessment of “social and cognitive competence rather than symptoms.”

 PPMH added the CGAS as a second measure in order to obtain an easily understood rating of overall functioning. The CGAS has been used in research for more than 2 decades and is very similar to Axis V of the 

Diagnostic and Statistical Manual of Mental Disorders

 Several studies have used the CGAS to track patients over time.

We embedded the BPRS-C and CGAS in an Outcomes Rating Form that also included several other fields that we needed for this project. The Outcomes Rating Form is a single 2-sided measure that contains the BPRS-C, CGAS, and 10 other variables about the treatment facility, patient, and clinician. Individual clinicians are responsible for filling out all fields; they select their answers by filling in a blank circle on the scanable form. Other information collected on each form includes medical record number, date of administration, patient’s age and sex, patient’s insurance provider, hospital location, and indication whether the form is for an initial visit/ admission, a quarterly follow-up, or a discharge visit. Clinical fields include CGAS score, primary and secondary Axes I and II diagnoses, and the 21 BPRS-C items. To ensure reliability, clinicians were provided with the descriptive anchors for the BPRS-C developed by Hughes and colleagues

 and level descriptions for the CGAS as developed by Shaffer and colleagues

 on separate pieces of paper, and received on-site training in how to apply them.

During spring 2005, before the commencement of the outcomes initiative, members of the Child Outcomes committee traveled to all of the PPMH sites to provide clinicians with background on the initiative and instruction in the procedures and the use of the 2 measures. An off-site professional vendor was selected to scan and process the standardized forms. Beginning on July 1, 2005, completed forms were mailed on a monthly basis from each site to the vendor, who then provided data files to the Institute for Health Policy, a healthcare research organization within 1 of the PPMH hospitals. The Institute for Health Policy maintains the database and is primarily responsible for data analyses and reporting on issues of compliance and data accuracy. Administrative personnel generally collect forms, and at many sites those clinicians with outstanding forms are sent reminders.

Pediatric patients are seen at 20 PPMH sites. These sites are part of academic medical centers (Brigham and Women’s, McLean, and Massachusetts General hospitals), community hospitals (North Shore Medical Center), and community health centers affiliated with these institutions (Massachusetts General Hospital Charlestown, Chelsea, Revere, and North End; Brigham and Women’s South Jamaica Plain and Brookside; North Shore Boston Street and Lafayette Street clinics). The level-of-care array includes inpatient (McLean Franciscan Hospital and North Shore Medical Center Hunt units), combined partial hospital and Acute Residential Treatment (McLean units in Belmont, Brockton, and at the Franciscan Hospital, and the Klarman Eating Disorders Center); freestanding partial hospital (North Shore); and outpatient clinics at 11 of these locations. All sites are in eastern Massachusetts and most are affiliated in some way with the Harvard Medical School. The sites range in size from 2 to 20 full-time-equivalent clinicians.

All patients who were admitted to inpatient, partial hospital, or acute residential treatment, or who had initial visits to outpatient sites between July 1, 2005, and June 30, 2006, were included in the study sample. Patient discharges and follow-up visits after June 30, 2006, that could be matched to admission/initial visits occurring during our study period also were included. The study was reviewed by the Partners Institutional Review Board and classified as exempt.

Because we wanted to assess the degree of compliance with the requirement of obtaining an Outcomes Rating Form for each admission/discharge, we had to calculate the denominator of all new cases for the entire system. We combined data from several PPMH billing databases to determine the total number of admissions and intakes for all sites between July 2005 and the end of June 2006.

The characteristics of patients participating in our initiative were assessed using information from intake/admission forms. Intake and follow-up completion rates across levels of care were used to assess feasibility. As noted above, completion rates for intakes/admissions were calculated by comparing the number of completed forms with the expected number based on billing data. Follow-up completion rates were calculated by comparing the number of patients with both intake/admission forms and follow-up/discharge forms with the number of patients with intake/admission forms.

We hypothesized that overall validity of the 2 measures score over the course of treatment within each level of care; (2) a significant difference in impairment score on each measure according to the level of care (inpatient, partial hospital, acute residential treatment, outpatient); and (3) cross-measure agreement between the 2 instruments. For the BPRS-C, reliability was explored via the Cronbach alpha statistic. Because the CGAS is a single-item rating, inter-item reliability could not be calculated. Changes in scores over the course of treatment were assessed at the patient level by determining the difference between follow-up/discharge ratings and intake/admission ratings for both the BPRS-C and CGAS, with significance testing using paired 

 tests. Differences in intake scores across levels of care were assessed by using analysis-of-variance and contrast tests. We assessed cross-measure agreement between the 2 tools at both intake/admission visit and 90-day follow-up/discharge using Pearson product moment correlations.

During the first year of this project, throughout the entire 20 sites of Partners Child Psychiatry and Mental Health System, 3636 children and adolescents were seen for intake/admission. Of these children, 2396 were rated using the standardized  Outcomes Rating Form for CGAS/BPRS-C. As would be demonstrated if there were (1) an improvement in shown in Table 1, 508 of these forms (21%) were from inpatient sites, 311 (13%) were from partial hospital sites, 555 (23%) were from acute residential treatment sites, and 1022 (43%) were from outpatient sites. Duplicate record numbers were flagged and removed from the dataset if they appeared by date to represent duplicate ratings of the same patient for the same period of time.

Across all levels of care the mean age of patients was 13.4 years. Systemwide there were 1308 (55%) females and 1075 (45%) males. Table 1 shows that about one-third of all patients (33.6%) were covered by some form of commercial Blue Cross Blue Shield product, another 22% by other commercial insurers, and nearly one-third (29%) were covered by Medicaid. In about 16% of all cases, the insurance could not be categorized or was missing.

Table 2 shows the number of admission/intake forms expected according to Management Information System billing data and the number and percentage of forms actually received for each level of care. As shown, intake form return rates ranged from 63% to 70% across the 4 levels of care, with overall intake/admission returns at 66% systemwide.

Table 2 also shows the number of follow-up/discharge forms received at each level of care for patients who had an Outcomes Rating Form completed at admission/intake. For inpatient,  partial, and acute residential treatment hospitals the discharge completion rate ranged from 65% to 79%. At outpatient sites the follow-up completion rate was much lower (22%). Systemwide, the overall rate of follow-up/discharge Outcomes Rating Forms was 49%. It was not possible to calculate the treatment interval between intake and follow-up, because the field on the form asked for date of form completion rather than date of follow-up/discharge. In the outpatient sites the forms were to be given at approximately 90 days into treatment, whereas at inpatient, partial hospital, and acute residential treatment sites the forms were to be completed on the day of discharge, a pre/post interval that could vary considerably.

Given the lower rates of completion for follow-up ratings, within each level of care the full sample at intake was compared with the smaller sample of patients who completed forms at follow-up as well as intake for each of the 3 major demographic variables. Although statistically significant differences were observed for more than half (7/12) of the comparisons, all the differences were small (±10%) and did not appear to show a clear pattern when compared across levels of care. For inpatients there were no significant demographic differences for cases seen for admission and discharge versus those seen for admission alone; for outpatients, only insurance type revealed a significant difference (patients with Medicaid were significantly less likely to have completed follow-up forms). In acute residential treatment and partial hospital cases, however, patients with Medicaid were more likely to complete discharge forms. For acute residential treatment patients, females were less likely to have discharge forms; for partial hospital patients, females were more likely to have discharge forms. For both of these levels of care, patients over age 18 years were less likely to have discharge forms.

Table 3 shows the mean scores, the changes in mean scores, and significance levels of the 

 tests for pairs comparing pretest and posttest data for subjects with valid BPRS-C and CGAS data at both intake/admission and follow-up/discharge. Time 1 means are for only those 1185 subjects who also have time 2 scores (paired data within subjects), not the full sample of 2396 who completed forms at time 1. The number of subjects at time 2 is much smaller due primarily to the large number of outpatients who did not return for follow-up after intake and/ or missing data. The means are given for each level of care separately.

As the table shows, mean scores for both BPRS-C and CGAS showed significant improvements over time (

 <.001 for all) within all 4 levels of care. Paired 

 tests demonstrate that patients in inpatient, acute residential treatment, partial hospital, and outpatient treatment showed statistically significant improvement in functioning (as measured by mean increases in time 2 CGAS scores), as well as decrease in symptoms (measured by mean decreases in time 2 BPRS-C scores) from intake/admission to 90-day follow-up/discharge.

Table 3 also can be viewed from another perspective, illustrating the way in which BPRS-C and CGAS mean scores differed across levels of care, with poorer functioning at more restrictive levels of care. Contrast analyses using generalized linear models show that mean initial CGAS scores were significantly different among all levels of care and that mean initial BPRS-C scores were significantly different between any pair of settings except between the acute residential treatment and partial hospital settings. Specific contrast analysis significance levels are listed in footnote b of Table 3.

Table 4 shows the correlations between the BPRS-C and CGAS scores within each level of care and in the overall sample at intake/admission (time 1) and at follow-up/discharge (time 2). As in Table 3, data shown are only for subjects who had scores at both time 1 and time 2. At intake/admission these 2 measures showed significant correlations with each other ranging from r = -.34 to r = -.47 for the 4 different levels of care. At follow-up/discharge the correlations between CGAS time 2 and BPRS-C time 2 also were significant and in fact were somewhat higher, ranging from r = -.59 to r = -.69 at the 4 levels of care.

For the BPRS-C, the measure clinicians used to rate symptoms of their patients, inter-item reliability was assessed using the Cronbach alpha statistic. Ratings of .78 for initial assessments and .81 for follow-up assessments demonstrate the internal consistency of clinician ratings, suggesting that the BPRS-C was a reliable instrument in this population.

Data from the first 12 months of a systemwide outcomes measurement initiative confirm that it is possible to implement procedures using standardized instruments to rate symptom severity and level of functioning at intake/admission and to collect follow-up/discharge data across the full range of facilities for mental health care for children from inpatient to outpatient. All 20 sites in the current sample were able to incorporate standardized rating forms into their intake procedures, with 66% of all eligible intake cases having assessment forms completed during the first year of the initiative.

Follow-up assessment also appeared to be feasible, at least in the more restrictive levels of care. For inpatient, acute residential treatment, and partial hospital patients, the rates of discharge form completion ranged from about two-thirds to about three-quarters of all admissions, whereas in the outpatient settings follow-up forms were completed for fewer than one-quarter of all intakes. Determining the true feasibility of follow-up form completion in outpatient settings proved to be quite difficult. Calculating a valid follow-up form completion rate for outpatient cases was impossible with the data we had because the denominator of all patients who actually entered treatment was not known. Because data on the number of visits for each case were not collected, it was not possible to tell which patients dropped out after just 1 or 2 intake visits.

An unpublished report titled “Measuring outcomes in outpatient child psychiatry: the contribution of electronic technologies and parent report” by J. M. Murphy and colleagues (Massachusetts General Hospital, 2008) provides additional information on completion of follow-up forms. In 1 large clinic where the actual number of visits could be tracked for each patient, an examination of cases that did not have follow-up forms showed that about half of all children seen for intake did not return for more than 1 or 2 additional visits. Because clinicians were not required to fill out follow-up forms for cases unless there had been at least 3 visits, the other cases would have to be excluded from a valid denominator. Using this figure to estimate the true denominator for outpatient follow-up forms in this 1 outpatient clinic would have led to a completion rate of about 40%. It also is important to note that when the above outpatient setting switched to an electronic data collection  system, it was possible to obtain follow-up forms for more than 80% of cases. Both of these findings suggest that it is possible to make follow-up assessments more feasible in outpatient settings.

Validity of the measures was supported by several findings. First, scores obtained here were comparable to those reported for patients receiving similar levels of care in other studies. Second, in the current study, the 2 measures used were significantly correlated with each other at both intake and followup. Third, scores on both measures improved significantly from intake to follow-up in all 4 levels of care. Fourth, scores on both measures showed significantly more impairment in more restrictive levels of care compared with less restrictive levels of care. A reasonably high level of inter-item reliability was shown for the BPRS-C at both administrations.

Strengths and Limitations of the Study and the Program

Strengths of this study include a relatively large and representative sample drawn from a relatively large and diverse child psychiatry system of care, high rates of form completion at intake under real-world conditions, and the validity and reliability of the measures used. Limitations include lower levels of form completion at time 2, especially in outpatient settings, and lack of information on compliance with the insurers’ other mandates to incorporate score data into treatment planning and patient feedback. The fact that both measures relied only on clinician report to assess improvement leaves open the question as to the degree to which such reports could be biased.

Strengths of the program were the fact that it was able to incorporate outcome measures as a standard of care in a large and diverse system and that it has been sustainable over more than 3 years. Limitations included the time burden the initiative placed on clinicians and administrative staff. Moreover, many clinicians reported that because they themselves provided all of the information on the rating forms, the scores did not give them any new information that would make a significant contribution to treatment planning. As noted by Busch and Sederer,1 the lack of benefit to individual clinicians and patients was probably a negative incentive for completing the forms. Nevertheless, the Child Outcomes committee concluded that the clinician-completed BPRS-C and CGAS should still be used as a systemwide threshold measure with the possibility that parent-completed instruments and/or diagnosisspecific measures might need to be added in the future. Such additions would have to be weighed carefully, considering the additional burden for clinicians and administrative staff.

In the project’s first year, some unexpected benefits as well as burdens were identified. Data from the project gave the system a yardstick to assess functioning and symptom severity at different levels of care (inpatient, acute residential treatment, partial hospital, and outpatient), which has enabled the psychiatry department to provide quality assurance data for a number of administrative reports.

Although it is not possible from the data in this study to determine the extent to which these measures may have been used to provide data for treatment planning or patient feedback (the insurer’s other 2 mandates), it was clear that the original procedure of using paper and pencil forms mailed out for off-site scoring did not lend itself to providing the kind of rapid feedback that would be most useful for these purposes. However, use of many of the newer types of electronic technology (eg, input of the measures through kiosks, fax backs, tablets, or digital pens) could speed up the process considerably. Several of these methods already have been pilot-tested at 1 of the PPMH sites and have demonstrated an ability to substantially increase outpatient form completion rates at both intake and follow-up (Murphy et al, unpublished report, Massachusetts General Hospital, 2008). With or without these methods, because other major insurers now are mandating their own outcomes measures, it seems likely that the use of such measures is here to stay.

 From the McLean Hospital (JG, RJB, MP, RLB), Belmont, MA; the Department of Psychiatry (JG, RJB, MP, CV, MJ, JMM), Harvard Medical School, Boston, MA; the Department of Psychiatry (RAC), North Shore Medical Center, Salem, MA; the MGH Institute for Health Policy (CV, ZL), Boston, MA; the Department of Psychiatry (NTK, RLB, MJ, JMM), Massachusetts General Hospital, Boston, MA; Partners Psychiatry and Mental Health (SJD, KJS), Boston, MA; and the Department of Psychiatry (MJ), Newton Wellesley Hospital, Newton, MA.

The authors (JG, RJB, RAC, MP, CV, NTK, RLB, ZL, MJ, SJD, KJS, JMM) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Concept and design (JG, RJB, RAC, MP, NTK, ZL, MJ, SJD, KJS, JMM); acquisition of data (RJB, MP, NTK, RLB, JMM); analysis and interpretation of data (JG, RJB, RAC, MP, CV, NTK, RLB, ZL, SJD, KJS, JMM); drafting of the manuscript (RAC, MP, NTK, RLB, MJ, JMM); critical revision of the manuscript for important intellectual content (JG, RJB, MP, CV, RLB, ZL, MJ, SJD, KJS, JMM); statistical analysis (CV, ZL, JMM); provision of study materials or patients (RJB, RLB, JMM); administrative, technical, or logistic support (RJB, RLB, MJ, JMM); and supervision (CV, MJ).

J. Michael Murphy, Department of Psychiatry, Massachusetts General Hospital, 55 Fruit St, YAW 6A, Boston, MA 02114. E-mail: mmurphy6@partners.org.

Assessing outcomes in psychiatric practice: guidelines, challenges, and solutions. 

2. Hughes CW, Rintelmann J, Emslie GJ, Lopez M, MacCabe N.

 A revised anchored version of the BPRS-C for childhood psychiatric disorders. 

3. Massachusetts Behavioral Health Partnership (MBHP). 

Clinical outcomes management protocol: performance specifications and phase-in timelines.

http://www.masspartnership.com/provider/outcomesmanagement/Outcomesfiles/Quality%20Alert%2010_%20Final.pdf

4. Shaffer D, Gould MS, Brasic J, et al. 

A Children’s Global Assessment Scale (CGAS). 

5. DelBello MP, Findling RL, Kushner S, et al. 

A pilot controlled trial of topiramate for mania in children and adolescents with bipolar disorder. 

6. Lachar D, Randle SL, Harper R, et al. 

The Brief Psychiatric Rating Scale for Children (BPRS-C): validity and reliability of an anchored version. 

7. Ross RG, Novins D, Farley GK, Adler LE. 

A 1-year open-label trial of olanzapine in school-age children with schizophrenia. 

Multidimensionally impaired: the good news. 

9. Zalsman G, Carmon E, Martin A, Bensason D, Weizman A, Tyano S. 

Effectiveness, safety, and tolerability of risperidone in adolescents with schizophrenia: an open-label study. 

The Brief Psychiatric Rating Scale for Children [Brief Reports and Reviews]. 

12. Gabbard GO, Coyne L, Kennedy LL, et al. 

Interrater reliability in the use of the Brief Psychiatric Rating Scale. 

13. Rufino AC, Uchida RR, Vilela JA, Marques JM, Zuardi AW, Del-Ben CM. 

Stability of the diagnosis of first-episode psychosis made in an emergency setting. 

Two standardized rating scales appeared to be valid and reliable for use at admission and possibly follow-up in a child psychiatry system of care.


Christine Vogeli, PhD

Articles

Implementing a Hybrid Approach to Select Patients for Care Management: Variations Across Practices

Clinician Considerations When Selecting High-Risk Patients for Care Management

Assessing Outcomes in Child Psychiatry