Objective: To describe the development and assessment of the Abbreviated Fine Severity Score (AFSS), a simplified version of the Pneumonia Severity Index (PSI) suitable for providing risk-adjusted reports to clinicians caring for patients hospitalized with communityacquired pneumonia.

Study Design: Retrospective cohort study.

Methods: We defined the AFSS based on data available in administrative and laboratory databases. We downloaded and linked these hospitalization and laboratory data from 2 cohorts (11,030 patients and 6147 patients) hospitalized with community-acquired pneumonia in all Kaiser Permanente Medical Care Program hospitals in northern California. We then assessed the relationship between the AFSS and mortality, length of stay, intensive care unit admission, and the use of assisted ventilation. Using logistic regression analysis, we assessed the performance of the AFSS and determined the area under the receiver operating characteristic curve (c statistic). Using a combination of manual and electronic medical record review, we compared the AFSS with the full PSI in 2 subsets of patients in northern California and Denver, Colorado, whose medical records were manually reviewed.

Results: The AFSS compares favorably with the PSI with respect to predicting mortality. It has good discrimination with respect to inhospital (c = 0.74) and 30-day (c = 0.75) mortality. It also correlates strongly with the PSI (r = 0.87 and r = 0.93 in the 2 medical record review subsets).

Conclusions: The AFSS can be used to provide clinically relevant risk-adjusted outcomes reports to clinicians in an integrated healthcare delivery system. It is possible to apply risk-adjustment methods from research settings to operational ones.

(Am J Manag Care. 2008;14(3):158-166)

More than 4 million cases of community-acquired pneumonia (CAP) occur in the United States each year, with 1.3 million patients hospitalized.^1-3 CAP remains the leading infectious cause of mortality in the United States.4 Its incidence among older persons is 18.3 cases per 10005 and accounts for almost 7% of all US patient hospital costs.^6,7 CAP is also the most common cause of severe sepsis.⁸

Given its importance, managed care organizations seeking to improve quality of care must find ways of addressing practice variation in the management of CAP. The presence and persistence of practice variation have undermined the credibility of hospitals and physicians with purchasers and with the public.^9-12 A major challenge facing managed care organizations seeking to reduce practice variation is how to respond to clinicians’ concerns regarding differences in patient illness severity. The most commonly used administrative data sources for risk adjustment are hospital discharge abstracts, which are based on International Classification of Diseases (ICD) codes (usually grouped into diagnosis-related groups¹³). These readily available sources have 2 major disadvantages. First, they use information that is unavailable at the time of clinical decision making (eg, an ICD code that indicates that a patient experienced assisted ventilation). Second, they do not contain information about a patient’s physiologic state. Research investigations address these limitations by incorporating data acquired through manual medical record review such as vital signs and laboratory test results.^13,14 However, because of the high costs of acquiring such data, it is difficult for managed care organizations to incorporate them into routine reports provided to clinicians.

A few recent studies have shown that it is possible to assign abbreviated forms of severity scores using laboratory data from automated databases^15,16 or from manual data collection in combination with electronic data.¹⁷ Render et al^16,18 developed a method for risk adjusting adult intensive care unit (ICU) outcomes that combines laboratory data with administrative data, while Graham and Cook¹⁷ combined manually acquired diagnostic and electronically captured laboratory data. More recently, Pine et al¹⁴ highlighted the value of supplementing condition-specific riskadjustment models with laboratory data. Although it is clear that incorporation of vital signs, radiologic findings, and other findings of the physical examination in severity scores is desirable, many hospitals do not yet have ready access to these data in electronic format. Consequently, it seems reasonable to use these abbreviated scores during this transitional phase in medicine.

In this article, we describe the development and assessment of such a transitional tool, an abbreviated version of an existing severity of illness score, the Pneumonia Severity Index (PSI).^19,20 Our Abbreviated Fine Severity Score (AFSS) combines available laboratory data with administrative data and is used for routine reporting to clinicians at 18 hospitals in an integrated healthcare delivery system, the Kaiser Permanente Medical Care Program (KPMCP). We chose this score as our starting point because of the clinical importance of CAP and because the PSI is an integral part of the KPMCP CAP clinical practice guideline.²¹

MATERIALS AND METHODS

Study Setting
We developed the AFSS using hospitalization data from 16 Northern California KPMCP hospitals between January 1, 2000, and March 30, 2002 (cohort 1). All of these facilities use the same comprehensive information systems linked by a common medical record number. By the time that ongoing operational reporting using the AFSS became routine, an additional 2 hospitals were in operation. Therefore, cohort 2 consisted of CAP admissions between July 1, 2004, and June 30, 2005, at 18 KPMCP hospitals in northern California.

Limited resources allowed review of medical records for only the following 3 patient subsets: (1) We compared the electronically assigned AFSS with the PSI based on manual medical record review in the Colorado region of the KPMCP, which has 398,706 members, including admissions to Exempla St Joseph Hospital (ESJH) in Denver, Colorado (which is the primary contract hospital for KPMCP patients in Denver), and the Kaiser Permanente outpatient clinics from which patients were referred between November 1, 2004, and April 22, 2005. (2) We performed a similar medical record review using randomly selected hospitalizations from cohort 1 at the KMPCP Oakland Medical Center. (3) We reviewed a randomly selected group of patients from cohort 1 to assess the accuracy of the diagnosis of pneumonia.

We obtained approval from the institutional review boards for the protection of human subjects in the northern California and Colorado regions of the KPMCP. The approval included a waiver of individual informed consent.

Identification of Patients With CAP
Methods used to identify KPMCP patients and to link their hospitalization records to each other (in the case of transported patients) and to their laboratory test results have been described.^22-27 Our initial step was to identify all nonobstetric, nonpsychiatric patients at least 18 years of age hospitalized between January 1, 2000, and March 31, 2002, with at least 1 stay in a KPMCP hospital. From this cohort, we first identified patients with pneumonia and then selected patients who met the following eligibility criteria^19,20: (1) A principal discharge diagnosis of pneumonia, defined according to the list of ICD codes by Fine et al,^19,20 was required. (2) In patients with multiple CAP hospitalizations, only the first was used. (3) Patients with prior hospitalizations within 7 days were excluded. (4) Patients with selected conditions (eg, prior organ transplantation or infection with human immunodeficiency virus) were excluded. (5) Among patients with a  hospitalization involving more than 1 hospital, patients with a first hospital stay outside the KPMCP were excluded. (6) Patients who were not KPMCP members at the time of CAP hospitalization were excluded. For identification of patients at ESJH, we used the same criteria. Membership in the Colorado KPMCP was required instead of membership in the Northern California KPMCP.

Abbreviated Fine Severity Score
The PSI uses 19 predictors (eg, arterial pH and altered mental status) and has a maximum value of 285 plus the patient’s age in years (Table 1).²⁰ Of these 19 predictors, the following 7 are not readily available in hospital discharge abstracts or in laboratory databases and could not be included: whether a patient was a nursing home resident, 5 physical examination components (eg, respiratory rate), and the presence of pleural effusion. The remaining 12 predictors, which constitute the AFSS, are available in KPMCP databases. Based on the PSI scoring scheme, these 12 items permit a maximum number of points equaling 180 plus the patient’s age in years.

We scanned the KPMCP and ESJH hospitalization and laboratory databases and downloaded all relevant test results obtained on a patient during the 24-hour period preceding hospital admission and linked these to the electronic discharge abstracts. In cases in which more than 1 test of a given type was obtained in the time frame, we selected the test result that would give the highest point assignment. If an individual test result was not obtained for a patient (eg, arterial pH), it was imputed as normal, and 0 points were assigned.

Outcomes

We ascertained inhospital mortality, assisted ventilation, ICU admission, and total hospital length of stay (LOS) from KPMCP and ESJH databases in northern California and Denver. To ascertain 30-day mortality, we linked KPMCP records to California or Colorado death certificates and to publicly available Medicare files using previously described methods.²⁸

We ascertained the use of assisted ventilation based on ICD procedure codes 96.7 (other continuous mechanical ventilation), 96.70 (continuous mechanical ventilation of unspecified duration), 96.71 (continuous mechanical ventilation <96 hours), or 96.72 (continuous mechanical ventilation ≥96 hours). To establish total LOS for patients transferred between hospitals, we linked records involving multiple hospital stays, so LOS is defined as the exact time in days and hours between the first hospital admission in a linked hospital stay and the final discharge to home or a skilled nursing facility or death.

Manual Medical Record Abstraction
Because the Colorado KPMCP had an operational outpatient electronic medical record during the study period, we could manually audit all of the Denver patients’ outpatient notes before admission, admission histories and physical examination findings, and dictated discharge summaries. These latter 2 items are retrievable through the electronic medical record.

To compare the AFSS and the PSI in Northern California KPMCP patients, we randomly selected 200 CAP hospitalizations that began at the KPMCP Oakland hospital between January 1, 2000, and March 30, 2002. Of these, 100 were randomly selected hospitalizations in which the patient died or was admitted to the ICU, and 100 were randomly selected hospitalizations in which the patient survived and did not experience ICU admission.

We randomly selected 70 northern California records to verify the CAP diagnosis. We manually reviewed Northern California KPMCP radiology databases, outpatient diagnosis databases, and electronic discharge summaries to confirm the CAP diagnosis by the presence of a new infiltrate on a chest radiograph within 48 hours of admission or by the admitting physician’s physical examination findings. If confirmation or refutation of the diagnosis of pneumonia was not possible in this fashion, we manually reviewed the paper hospital medical records.

Statistical Analysis

During our initial development phase, we did not test the performance of the AFSS as a simple aggregate score but instead grouped the laboratory and comorbidity points into subscores. We did this to educate our target audience (chiefs of hospital-based medicine and ICU directors) with respect to the relative contribution of different factors to patient outcomes. Production reports use only aggregate scores.

We used logistic regression analysis to examine each dichotomous outcome (death, admission to the ICU, and the use of mechanical ventilation) relative to the full AFSS or its components. We assessed the performance of these models using the area under the receiver operating characteristic curve, or c statistic.²⁹ We used the likelihood ratio χ² to estimate the relative contribution of each model predictor. We calculated the marginal increase in this χ² statistic accounted for by each predictor as it was added and removed from the full model.^16,30 We used linear regression analysis and the Pearson product moment correlation coefficient to compare the AFSS with the PSI.

RESULTS

Cohort 1
During the 27-month period of our initial development work using northern California data, 200,957 patients at least 18 years of age experienced 347,522 separate overnight stays, which (after linkage of transfers) comprised 304,013  hospitalizations at 16 study hospitals. Of these hospitalizations, 15,884 met our initial screening criteria (a discharge diagnosis of pneumonia). After exclusions, 11,030 patients comprised the final cohort. The inhospital mortality was 9.2%, while the 30-day mortality was 15.1%. Within the cohort, 1352 patients (12.3%) had ICU admissions and 598 patients (5.4%) experienced assisted ventilation. Among survivors, the median LOS was 3.7 days, with a mean LOS of 5.3 days.

Cohort 2
Between July 1, 2004, and June 30, 2005, in northern California, 6147 patients met the inclusion criteria. Among these patients, the inhospital mortality was 7.7%, while the 30-day mortality was 13.4%. Seven hundred six patients (11.5%) were admitted to the ICU, and 233 patients (3.8%) experienced assisted ventilation. Among survivors, the median LOS was 3.7 days, while the mean LOS was 5.3 days.

Patient Subsets With Manual Medical Record Review
During the 6-month study period at the Denver site, 1005 patients at least 18 years of age were hospitalized at ESJH. Because of problems with the ESJH laboratory system, we were able to obtain complete data for only 395 of these patients. Among these patients, the inhospital mortality was 3.8%, while the 30-day mortality was 14.2%. One hundred sixty-seven patients (42.3%) were admitted to the ICU, and 36 patients (9.1%) experienced assisted ventilation. Among survivors, the median LOS was 4.0, while the mean LOS was 5.7 days. The Denver and Oakland samples were drawn using a case-control design to ensure that adequate numbers of inpatient deaths and ICU admissions were available for analysis. Therefore, these samples had a higher representation of patients who died in the hospital or were in the ICU than cohorts 1 and 2.

Two of us (GJE and MPC) manually reviewed KPMCP paper medical records, electronic physician admission histories and physical examination findings, electronic physician discharge summaries, and dictated radiology reports and were able to verify the diagnosis of pneumonia in 53 of 70 (75.7%) randomly selected records of patients with CAP. Of the medical record review patients with confirmed pneumonia, 80% had positive radiographs, and 20% had positive physical examination findings.

In the 2 cohorts that underwent manual medical record review (200 Oakland patients and 395 Denver patients), we found a strong correlation between the AFSS and the PSI. After adjusting for the sample design, the correlations between the AFSS and the PSI were 0.93 and 0.87 in the Denver and Oakland cohorts, respectively. The c statistics for inhospital mortality and 30-day mortality ranged from 0.75 to 0.79 for the PSI and from 0.66 to 0.73 for the AFSS in the 2 cohorts whose medical records were manually reviewed. At both sites, these values changed little when patients with questionable diagnoses of pneumonia were excluded.

AFSS Predictive Performance
Table 2 gives the characteristics of cohort 1, in which the AFSS scores ranged from 8 to 191, with a median score of 84 and a mean score of 84.6. Table 3 gives the results of the logistic regression model in which the outcome was mortality and the predictors were age, sex, laboratory subscore, and comorbidity subscore. This model predicted mortality with c statistics of 0.74 for inhospital mortality and 0.75 for 30-day mortality. Examination of observed and expected mortality rates across different risk groups showed that the models were well calibrated until they reached a predicted mortality risk of 35%. Above this predicted risk, the number of patients was small. When considering inhospital death, 157 patients (1.4% of all patients) had a predicted mortality risk of at least 35%, and the number of deaths in this group was 48 (30.6%); this group of patients had a mean (SD) age of 83.5 (8.1) years. Their mean (SD) AFSS was 156.1 (10.1). In the model for 30-day mortality, 810 patients (7.3% of all patients) had a predicted mortality risk of at least 35%, and the number of deaths in this group was 313 (38.6%). This group of patients had a mean (SD) age of 84.6 (7.7) years. Their mean (SD) AFSS was 134.9 (13.8).

The roles of laboratory abnormalities and age differed in these models. Laboratory abnormalities played a larger role for inhospital mortality, and age played a larger role for 30-day mortality. As indicated by the likelihood ratio χ² , the relative contributions of laboratory abnormalities to the total explanatory power of the model were 33% for inhospital mortality and 19% for 30-day mortality. In contrast, the relative contributions of age were 34% for inhospital mortality and 50% for 30-day mortality.

The c statistics of the AFSS for the occurrence of assisted ventilation (c = 0.71) and admission to the ICU (c = 0.69) were lower than those for mortality. Figure 1 shows the relationships between the AFSS and mortality, hospital LOS, admission to the ICU, and the use of assisted ventilation. The results in cohort 2 were similar (eg, c = 0.74 for inhospital mortality).

DISCUSSION

The PSI was the result of a federally funded research effort focusing on CAP that included quantitative assessment of predictors for hospital admission and discharge.^19,20,31-36 The PSI has been used to compare antibiotic therapies, need for different degrees of treatment intensity, and efficacy of diagnostic and treatment strategies. ^37-42 In our context, developing a riskadjustment method based on the PSI, rather than a completely recalibrated or a totally new score, was desirable for several reasons. Given the high cost of manual medical record review and the unavailability of vital signs, coded radiologic findings, and other physical examination components in electronic form, adopting the strategy by Render et al16,18 seemed a reasonable way to improve on the existing reporting strategy, which until this point was based entirely on ICD codes. Credibility with clinicians is important, so basing the method on a quantitative tool that had already been endorsed by the department chiefs of internal medicine, emergency medicine, and hospital-based medicine made sense. In addition, using data extracted from the outpatient electronic medical record data repository and from an inpatient data system, we found that the AFSS performed well in Denver. Although the initial investment required in developing the AFSS was substantial because it marked the first time the KMPCP attempted to use laboratory data for routine reporting, subsequent costs have fallen notably, and generating the report now requires only 8 to 12 hours of programmer time per quarter.     

The AFSS has important limitations. The most important of these is the loss of discrimination and calibration because of the absence of physical and radiographic findings. In our study, the AFSS did not discriminate as well as the original PSI (c statistic range, 0.83-0.89) or as well as a recalibrated PSI (c = 0.85).43 However, it performed as well as the manually assigned CURB (confusion, urea nitrogen, respiratory rate, and blood pressure) and CURB-65 scores.^44,45 As noted by others,⁴⁶ it is more difficult to predict the need for intensive care and the use of assisted ventilation, and we found that the overall performance of the AFSS with respect to predicting these outcomes is similar to that reported for medical record review–based indexes, including the complete PSI.⁴⁶ It is important to keep in mind that any score calibrated for a given outcome may not perform as well when used to predict a different outcome, particularly when the different outcome is more dependent on practice style rather than on a discrete physiologic end point (eg, admission to the ICU as opposed to inhospital mortality). The AFSS Hosmer-Lemeshow statistic was significant, and examination of specific risk deciles showed that our abbreviated score did not calibrate as well among patients whose predicted mortality risk exceeded 35%. Examination of this group, which constitutes less than 3% of the cohort, showed that it consisted of a much older and sicker set of patients. This suggests that future predictive models may need to handle older patients differently, perhaps by including age-specific interaction terms for laboratory data and comorbidities, and that collaborative efforts should be made so that greater numbers of these patients are in study datasets to minimize stochastic variation. Given the need to have robust risk-adjustment tools that have credibility with clinicians, we believe that these limitations are offset by the benefits of being able to generate a production report using automated databases.

Reports using the AFSS are not being presented to clinicians as providing conclusive evidence that center x is “better” than center γ. This is important given the sample size and methodological limitations noted by other investigators,^47,48 as well as the limitations of our abbreviated score and the original PSI, which may need to be modified to include other predictors.⁴⁹ Instead, the AFSS and reports based on it are being used (1) as directional indicators that can motivate collaboration between hospitals by serving as a starting point for further investigation of interhospital differences, (2) as a risk-adjustment tool for performing “mini case-control studies” using multivariate template matching,⁵⁰ and (3) as an aid to estimating sample sizes for other clinical investigations in which it is useful to have a sense of the overall severity distribution in a target population. Moreover, the AFSS is not being used for the management of or for any critique of the management of individual patients.

A quarterly report showing interfacility comparisons among different types of patients with CAP entered production in the Northern California KPMCP in March 2006. Since the availability of this report, clinicians in the KPMCP have been using it to assess their management of CAP among patients at different risk levels. For example, a facility whose CAP mortality is low but whose CAP LOS is high is reviewing randomly selected medical records using stratified random sampling based on the AFSS. Clinicians have requested additional analyses and changes in the report. Figure 2 shows a new graphic, similar to one developed by Render et al ¹⁸ for ICU reporting for Veterans Affairs hospitals, that will be incorporated into the production report beginning in 2008. This format permits clinicians to compare risk-adjusted mortality and LOS in survivors. Clinicians also requested that we provide a rough equivalency table comparing the AFSS with the PSI risk classes. The Appendix (available at www.ajmc.com) provides data showing the performance of the AFSS among the later Northern California KPMCP cohort using strata that approximate 5 risk classes based on mortality ranges in the original studies by Fine et al.^19,20 Additional details of our chart validation and current reports are also included in the online Appendix.

Obviously, the method we have defined can only be used by entities with the capability to store and retrieve laboratory data using standardized formats, which does not include all hospitals in the United States. On the other hand, increasing numbers of hospitals are acquiring this capability, so it is important for health services researchers to begin defining ways to use these methods operationally. Moreover, given the fact that vital signs, physical examination components, and coded radiologic findings are going to be available electronically in the near future, the AFSS must be considered a transitional tool that will ultimately be replaced by more robust severity scores.

In conclusion, our findings demonstrate that it is possible to incorporate electronically captured laboratory data in risk-adjustment models for patients with CAP. Organizations with integrated information systems (eg, hospital chains) can identify a cohort of patients with CAP and assign abbreviated point scores based on studies originally developed using paper medical records. These tools may then be used for production reports to support quality improvement efforts.