Understanding the Role of Big Data in the CV Risk Calculator Controversy

Published Online: January 20, 2014
Andrew Smith
Medical scholars on both sides of the very public dispute over new disease prevention guidelines from the American College of Cardiology (ACC) and the American Heart Association (AHA) expected controversy.

Major revisions to treatment protocols typically generate at least some points of genuine dispute, leaving individual professionals to weigh the evidence for themselves until new research and analysis answers questions definitively.

In this particular case, the points of actual contention are few, though serious. The vast majority of the guidelines, along with the vast majority of the procedures for creating them, have won widespread support among those who study atherosclerotic cardiovascular disease. Indeed, even the most famous critics of the new recommendations have praised them for their emphasis

on preventing stroke as well as heart disease, their efforts to differentiate risk by race and gender, and their efforts to translate research into suggestions1:

• Studies correlating low-density lipoprotein (LDL) cholesterol and disease find no specific thresholds separating safety from danger, so the guidelines omit any mention of specific LDL targets along the lines of <70 mg/dL or <100 mg/dL.

• Drug trials have found that many nonstatin medications can lower LDL levels, but no such drug has demonstrated the ability to lower the risk of stroke or heart disease like statins, which is why the new guidelines rarely recommend considering them, not even for statin-intolerant patients with dangerous LDL levels.

• Trials of statins, on the other hand, have distinctly demonstrated that they prevent strokes and myocardial infarction associated with hypertension, diabetes, and other risk factors beyond LDL levels; thus, the new guidelines suggest statins for many patients with healthy cholesterol profiles.2,3

• Analyses that found no evidence of benefit to statins given to older patients have also been incorporated into the new guidelines, which do not recommend them for patients over age 75 years. Such recommendations have raised objections among conservative clinicians who think the changes too radical and from people who wanted more emphasis on lifestyle and less on medication.

Among thought leaders, however, all the suggestions above have received a warm reception. The only real controversial points are formulas that combine nearly a dozen factors to estimate each patient’s 10-year risk of atherosclerotic cardiovascular disease and a recommendation that any patient whose calculated 10-year risk exceeds 7.5% consider statin therapy.

A viewpoint article published by The Lancet on November 19, 2013, just a few days after the ACC and AHA unveiled the new guidelines, began the debate on both those points, starting with the decision to base discussions about statin use on a combined calculation. “No trial of statin therapy has ever used a global risk prediction score as an enrollment criterion, so basing prescriptions on such a metric might be difficult to defend in an evidence-based climate,” wrote Paul Ridker, MD, and Nancy Cook, PhD, both professors at Harvard University’s Brigham & Women’s Hospital.

“In addition, trial data contradict statements that high absolute risk always predicts statin efficacy… The CORONA4 and AURORA5 trials enrolled individuals with very high vascular risk in the settings of heart failure or renal failure and found no evidence of event reduction with statin therapy despite large reductions in LDL cholesterol.”

Other critics have expressed particular frustration that high risk scores generated almost entirely by a patient’s age can easily cross the 7.5% threshold and thus trigger serious consideration of statin therapy.6 After all, they argue, no trial has ever found that statins prevent atherosclerotic cardiovascular disease associated with aging.

Donald Lloyd-Jones, MD, ScM, who co-chaired the risk assessment work group, thinks many of the critics simply misunderstand, or haven’t read, the guidelines. “The guidelines don’t order anyone to use statins. They merely indicate when it makes sense for doctors and patients to consider statins because there is evidence of benefit for patients in those groups,” said Lloyd-Jones, Eileen M. Foell Professor and chairman of the Department of Preventive Medicine at Northwestern University’s Fienberg School of Medicine.

“If your total 10-year risk is lower than 7.5%, the costs associated with statins may outweigh the benefits for many patients, so there may be no reason to use them. Once your risk exceeds 7.5%, though, it may make sense to use them, depending on why your risk exceeds 7.5%.

“If hypertension or diabetes or high LDL contributes to your risk, then statins probably make sense. If age is your only risk factor, then statins may not make sense and these guidelines aren’t pushing statins on you. I just don’t understand how anyone who had bothered to read the documents in full rather than news reports could think otherwise.”

The other major criticism of the guidelines concerns the accuracy of the risk calculator and, again, traces its origin to the Lancet piece. In it, Ridker and Cook report on their efforts to validate the risk calculator on 3 large medical data sets: the Women’s Health Study (WHS), the Physicians’ Health Study (PHS), and the Women’s Health Initiative Observation Study (WHI-OS).

The risk calculator received the initial information from each cohort participant and made predictions that Ridker and Cook then compared against actual patient results. The calculator overestimated risk by 75% to 150% in all 3 cohorts, Ridker and Cook wrote, predicting roughly double the actual amount of observed disease across the 3 groups. “It is possible that as many as 40–50% of the 33 million middle-aged Americans targeted by the new ACC/AHA guidelines for statin therapy do not actually have risk thresholds that exceed the 7.5% threshold suggested for treatment. Miscalibration to this extent should be reconciled and addressed in additional external validation cohorts before these new prediction models are widely implemented.”

Those charges have made much news, but the risk calculator’s creators and backers stand firmly behind their work.

To build the calculator, the ACC/AHA committee started with a list of factors proved to increase the risk of atherosclerotic cardiovascular disease and found several major medical cohorts that collected all the relevant data on large numbers of typical American patients for at least a decade: the Framingham Heart Study, Atherosclerosis Risk in Communities, Coronary Artery Risk Development in Young Adults, and the Cardiovascular Health Study.

They then crunched untold billions of numbers until they had the tool that best translated starting-point risk factors into accurate projections of actual disease occurrence. After finalizing the calculator, the committee attempted to validate it just as Ridker and Cook later did, by seeing how its predictions matched reality among patients in other cohorts, namely the Multiethnic Study of Atherosclerosis and Reasons for Geographic and Racial Differences in Stroke studies. These validation efforts also found that the calculator systematically overestimated risk. The overestimates tended to be small for people at low risk, but often exceeded 50% for patients at higher risk.

Committee members began to investigate and quickly found a plausible explanation for the error: many of the patients with high initial risk in both cohorts had started taking statins, which had prevented many of them suffering the strokes and heart problems that the calculator predicted. They suspect the same explanation would account for some of the overestimation that Ridker and Cook found and that missing data, along with the unusual patient populations of the WHS, PHS and WHI-OS cohorts, would explain most of the rest.

“We considered using all 3 of those cohorts, either to build or test the risk calculators, but we decided against it on grounds that they lacked the data needed to be helpful and could well end up being misleading,” said David Goff, MD, PhD, FACP, FAHA, dean of the Colorado School of Public Health and co-chair of the Risk Assessment Committee.

The data shortfalls ranged from the duration of patients’ records (less than a decade) to the information recorded (incomplete accounts of some risk factors and medications) to the way the information was collected (often selfreported). The patient populations of all 3 cohorts posed even greater concerns. Neither WHS nor PHS nor WHI-OS tracks anything like a normal cross-section of Americans. All 3 abound with the sort of affluent and educated patient who consistently enjoys far better health than poorer people with ostensibly similar medical starting points.

Barring both a decision by Ridker and Cook to hand over all their data and calculations and the discovery that Ridker and Cook devised a very clever way to compensate for the unusually healthy patient populations, Goff and his colleagues see little reason to scrap a risk calculator that, they say, works on normal populations better than any other available tool.

PDF is available on the last page.
Recommended Articles
Cardiovascular disease may be the leading cause of death in the United States, but half of those deaths can be prevented, according to researchers from Emory’s Rollins School of Public Health.
Metastatic colorectal cancer patients with higher body mass index (BMI) outlived patients with the lowest BMI by an average of 2-and-a-half months, according to new data presented at the European Society for Medical Oncology 17th World Congress on Gastrointestinal Cancer 2015.
Scientists at the Imperial College London have identified a genetic link to obesity and diabetes in an extremely obese woman and her family members.
A week after JAMA released a report that found more than two-thirds of Americans age 25 years and older are overweight or obese, the American Academy of Pediatrics presented updated guidelines on preventing childhood obesity and recommendations for parents to help their children maintain a healthy weight.
The JAMA authors recommend that HHS and the US Department of Agriculture pay attention to DGAC's recommendations on fat intake, which do not include an upper limit on total fat consumption.