Currently Viewing:
The American Journal of Managed Care October 2012
Change to FIT Increased CRC Screening Rates: Evaluation of a US Screening Outreach Program
Elizabeth G. Liles, MD, MSCR; Nancy Perrin, PhD; Ana Gabriela Rosales, MS; Adrianne C. Feldstein, MD, MS; David H. Smith, RPh, MHA, PhD; David M. Mosen, PhD, MPH; and Jennifer L. Schneider, MPH
Toward Tailored Disease Management for Type 2 Diabetes
Arianne M. J. Elissen, MSc; Inge G. P. Duimel-Peeters, PhD; Cor Spreeuwenberg, PhD; Marieke Spreeuwenberg, PhD; and Hubertus J. M. Vrijhoef, PhD
Implementation of EHR-Based Strategies to Improve Outpatient CAD Care
Stephen D. Persell, MD, MPH; Janardan Khandekar, MD; Thomas Gavagan, MD; Nancy C. Dolan, MD; Sue Levi, RN, MBA; Darren Kaiser, MS; Elisha M. Friesema, BA, CCRP; Ji Young Lee, MS; and David W. Baker, MD, MPH
Pediatric Integrated Delivery System's Experience With Pandemic Influenza A (H1N1)
Evan S. Fieldston, MD, MBA, MSHP; Richard J. Scarfone, MD; Lisa M. Biggs, MD; Joseph J. Zorc, MD, MSCE; and Susan E. Coffin, MD, MPH
Medicare Part D Claims Rejections for Nursing Home Residents, 2006 to 2010
David G. Stevenson, PhD; Laura M. Keohane, MS; Susan L. Mitchell, MD, MPH; Barbara J. Zarowitz, PharmD, FCCP, BCPS, CGP, FASCP; and Haiden A. Huskamp, PhD
Identification of and Intervention to Address Therapeutic Gaps in Care
Daniel R. Touchette, PharmD, MA; Sapna Rao, BPharm, MS; Purna K. Dhru, PharmD; Weihan Zhao, PhD; Young-Ku Choi, PhD; Inderpal Bhandari, PhD; and Glen D. Stettin, MD
EMR-Based Medication Adherence Metric Markedly Enhances Identification of Nonadherent Patients
Shepherd Roee Singer, MD, MPH; Moshe Hoshen, PhD; Efrat Shadmi, PhD; Morton Leibowitz, MD; Natalie Flaks-Manov, MPH; Haim Bitterman, MD; and Ran D. Balicer, MD, PhD
Financial Incentives and Physician Commitment to Guideline-Recommended Hypertension Management
Sylvia J. Hysong, PhD; Kate Simpson, MPH; Kenneth Pietz, PhD; Richard SoRelle, BS; Kristen Broussard Smitham, MBA, MA; and Laura A. Petersen, MD, MPH
Identifying Frail Older People Using Predictive Modeling
Shelley A. Sternberg, MD; Netta Bentur, PhD; Chad Abrams, MA; Tal Spalter, MA; Tomas Karpati, MD; John Lemberger, MA; and Anthony D. Heymann, MB BS
Currently Reading
Application of New Method for Evaluating Performance of Fracture Risk Tool

Application of New Method for Evaluating Performance of Fracture Risk Tool


We previously published our experience with the Fracture Risk Calculator (FRC; Foundation for Osteoporosis Research and Education) within Kaiser Permanente Northern California.1 Among nearly 95,000 female health plan members 50 years and older who underwent bone mineral density during 1997 to 2003, we compared the 10-year risk of hip fracture estimated using FRC with observed hip fracture outcomes.1 A part of that analysis compared FRC performance with and without input of femoral neck bone mineral density (BMD). Based on area under the receiver operator characteristic curve (AUROC) methodology (c-statistic), we concluded that BMD did not enhance the tool’s performance.

While the AUROC is a widely used method to optimize sensitivity and specificity, other methods may also be used to evaluate risk-assessment tools. One such method is the net reclassification improvement (NRI).2,3 The NRI calculation adds rates of appropriate reclassification of individuals to risk categories (to higher risk category for those who subsequently fracture and to lower risk category for those who do not) and subtracts inappropriate reclassifications; the result is a summative overall reclassifi cation improvement score ranging from –100% to +100%.2

We recently applied this methodology to examine the performance of FRAX, another web-based fracture–risk assessment tool developed by the World Health Organization.4 We found, as with FRC, that AUROC (c-statistic) was not improved with the addition of BMD to the FRAX model.4 However, the addition of BMD to FRAX did result in a significant positive NRI (5.5%, P <.01), using a cutoff derived from the maximum Youden’s J statistic (this identifies the threshold for optimal sensitivity and specifiity).4 We now similarly examine the addition of BMD to the FRC tool and its effect on the NRI using the same methodology.

The Youden’s J statistic identified a threshold of 1.3% for the FRC tool. When using 1.3% as a risk category threshold for assignment to higher versus lower risk,4 we found that the addition of BMD to FRC yielded a statistically non-significant NRI of 0.8% (P = .3). These analyses provide further evidence that the FRC tool’s performance appears to be minimally affected by the addition of BMD.

As methods for assessing risk prediction tools continue to be refined, further studies in larger and more diverse cohorts should be conducted to determine whether the addition of BMD provides improved discrimination in specific population subsets. In the meantime, our findings support the use of the FRC in populations where BMD is not known. Using limited electronic medical record data, this tool can identify higher-risk subgroups for further evaluation and management.

Erica M. Sobel, DO

Bruce Ettinger, MD

Joan C. Lo, MD

Alice R. Pressman, PhD
Author Affiliations: From Department of Medicine (EMS, JCL), Kaiser Permanente Oakland Medical Center; Division of Research (BE, JCL, ARP), Kaiser Permanente Northern California, Oakland, CA.

Authorship Information: Concept and design (JCL, ARP); acquisition of data (JCL, ARP); analysis and interpretation of data (EMS, BE, JCL, ARP); drafting of the manuscript (EMS, BE, JCL); critical revision of the manuscript for important intellectual content (EMS, BE, JCL, ARP); and statistical analysis (EMS, ARP).

Author Disclosures: Dr Ettinger reports that he is a board member of the Foundation for Osteoporosis Research and Education. The other authors (EMS, JCL, ARP) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Address correspondence to: Joan C. Lo, MD, Kaiser Permanente Division of Research, 2000 Broadway, Oakland, CA 94612. E-mail:
1. Lo JC, Pressman AR, Chandra M, Ettinger B. Fracture risk tool validation in an integrated healthcare delivery system. Am J Manag Care. 2011;17(3):188-194.

2. Pencina MJ, D’Agostino RB Sr, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med. 2011;30(1):11-21.

3. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157-172; discussion 207-112.

4. Pressman AR, Lo JC, Chandra M, Ettinger B. Method for assessing fracture risk prediction models: experience with FRAX in a large integrated health care delivery system. J Clin Densitom. 2011;14(4):407-415.
Copyright AJMC 2006-2019 Clinical Care Targeted Communications Group, LLC. All Rights Reserved.
Welcome the the new and improved, the premier managed market network. Tell us about yourself so that we can serve you better.
Sign Up