An Expanded Portfolio of Survival Metrics for Assessing Anticancer Agents

Jennifer Karweit, MS; Srividya Kotapati, PharmD; Samuel Wagner, PhD; James W. Shaw, PhD, PharmD, MPH; Steffan W. Wolfe, BA; and Amy P. Abernethy, MD, PhD

Cancer is an ever-growing public health problem,1 with an estimated 14.1 million new cases and 8.2 million associated deaths per year worldwide.2 Although advances in cancer therapy have led to dramatic gains in survival and quality of life (QOL), the cost of care has risen sharply.1,3 These costs, which are outpacing those of other areas of healthcare,4,5 are attributed to multiple factors, including the adoption of new and more expensive therapies,3,6 a shift toward personalized care,1 increased cancer survivorship,7 and the expanding number of cancer cases with a growing and aging population.1,7 Several key initiatives are therefore underway to facilitate a dialogue around improving care while containing costs.1,8,9
Much attention has been directed at defining the value of cancer treatments,1,10,11 but there is no consensus on what constitutes value as it has different meanings to various stakeholders (eg, patients, caregivers, physicians, payers).5,12 Recent analyses suggest that cost may not be associated with large survival gains13,14; and metrics currently used in oncology trials, such as response rate or even median overall survival (OS), may not characterize the full clinical value of new agents,13,15 which result in prolonged survival in a proportion of patients.16-18 Metrics are available that characterize changes in Kaplan-Meier curves with new therapies (eg, median and mean OS show a shift to the right and 1-year survival rate shows an upward shift in the tail), but when used alone, may not present the complete value picture. Therefore, a wide range of outcome measures are needed.10,11,15
The objective of this preliminary qualitative and quantitative analysis was to assess the utility of an expanded portfolio of survival metrics in differentiating the value among anticancer agents. This analysis used survival outcomes from several randomized trials with metastatic solid tumors. In addition, we developed a new cost-value analysis tool that can be easily applied to clinical trial data and may be useful to payers and providers in managed care in determining treatment choice.
Survival Metrics
This analysis used a survival metric portfolio consisting of median OS, mean OS, 1-year survival rate, and number needed to treat (NNT) to avoid 1 death at 1 year. Survival information for each agent was derived from the product’s regulatory submission documents to the FDA (prescribing information) or the European Medicines Agency (EMA) (summary of product characteristics), or the respective trial publications presented with the regulatory submission documents, depending on the availability of presented Kaplan-Meier curves.
Median OS—defined as the time from treatment randomization at which half of the patients remain alive, with deaths attributed to any cause19,20—was included because it is the most frequently used efficacy endpoint for anticancer agents.21,22 Mean OS—defined as the average length of time that patients are alive during the trial study period23,24—was estimated by calculating the area under the curve (AUC).25,26 Mean OS was truncated because AUCs had cut-offs based on the study duration, which was assumed to have been deemed appropriate by the regulatory bodies approving each treatment. The GetData software package version 12 (GetData Pty Ltd, Kogarah NSW, Australia) was used to digitize Kaplan-Meier curves for the investigational and comparator arms, and these curves were digitally replotted to estimate x and y coordinates.
Survival rate is defined as the percent of patients alive at a specific key time milestone (eg, 1 year) that reflects a meaningful period of a patient’s life, especially in tumors with a short survival prognosis.25 One-year survival rate data were acquired from each agent’s Kaplan-Meier curves. Survival rates for subsequent years were not included, because trial durations varied across agents, and some trials were designed to last less than 2 years.25,26
NNT is the average number of patients who need to be treated to prevent 1 event (eg, death) based on a time period (eg, 1 year).27,28 NNT measures the investment in number of treated patients required to receive a return benefit of treatment in the population. Although there are no set limits for NNT to be considered clinically effective, a lower NNT (closer to 1) is considered favorable across different disease interventions.28 In our analysis, NNT is the number of patients needed to treat to avoid 1 death at 1 year. As the inverse of the absolute risk reduction, NNT is calculated as follows:
NNT = 1 / (investigational arm event rate − comparator arm event rate)
Comparison of Survival Results
Survival results were compiled and compared both within and across trials by tumor type. Absolute improvement in mean or median OS was calculated (in months) by subtracting the median or mean OS of the comparator arm from that of the investigational arm. Absolute improvement in the 1-year survival rate was calculated (in percentages) by subtracting the survival rate of the comparator arm from that of the investigational arm.
Cost-Value Analysis
A cost-value analysis facilitated comparisons across tumor types. Total drug cost was calculated by multiplying the agent’s cost per month by duration of therapy. Drug costs were in US dollars for October through November 2013 based on Price Rx/Medi-Span, a public database of Wolters Kluwer. Duration of therapy was determined based on the agent’s median duration of administration, median progression-free survival (PFS), or median time to progression, as listed in the product labels, with these data elements chosen based on data availability. Total drug cost also factored in the loading dose, as indicated by the product label, for applicable agents. Because both ipilimumab and sipuleucel-T have limited fixed-dose durations relative to the other agents, their total drug costs were calculated by dividing drug cost by median OS; median OS was felt to present a more comparable and conservative measure of duration than mean OS due to the differential impact of censoring on the mean across trials.
To determine the cost-value relationship, each agent was plotted with the x-axis reflecting total drug cost and the y-axis reflecting absolute improvement in median OS, absolute improvement in mean OS, or 1-year NNT to avoid 1 death. A fitted regression line (with the intercept set at 0 to standardize the progression across metrics) indicating an average cost-to-outcome ratio was plotted for each graph to distinguish agents that were above and below the average for the specific metric. Agents above the regression line had a lower-than-average cost relative to outcome benefit, whereas those below the line had a higher-than-average cost relative to outcome benefit.
Disease and Agent Selection
The following metastatic solid cancers, which have been the focus of clinical investigation, were selected: breast cancer (BC), colorectal cancer (CRC), melanoma, non–small cell lung cancer (NSCLC), prostate cancer (PC), and renal cell cancer (RCC). The analysis was limited to metastatic disease—the most advanced form of cancer—as it accounts for more than 90% of cancer-related deaths,29 and because drug development is occurring in this setting.
Anticancer agents in this analysis were those that met all of the following criteria: used for the treatment of stage 3 or 4 metastatic or refractory disease; included as part of a comparative, multiple-arm, phase 3 trial, reported in regulatory submissions with any comparator and OS as the primary or secondary endpoint; having documented median OS benefit reached at the time of regulatory approval from the FDA or EMA; having follow-up data available for at least 1 year; having an available Kaplan-Meier curve; and having been launched in the last 10 years in both the United States and Europe (eAppendix [available at]). Other agents were excluded as a result of not meeting the criteria (eg, crizotinib). This analysis considered all lines of therapy and was performed in the context of limited head-to-head comparison trials.
Survival Metric Portfolio
In a few cases, survival metrics were consistent for a particular anticancer agent within tumor types. In 3 of the 6 tumor types, both the highest median OS improvement and the highest mean OS improvement occurred with the same anticancer agent (ipilimumab with melanoma, pemetrexed with NSCLC, and sunitinib with RCC) (Table 1, see bolded numbers). Also, the highest improvement in the 1-year survival rate and the lowest NNT occurred together with the same anticancer agent in 5 tumor types (bevacizumab with CRC, ipilimumab with melanoma, erlotinib with NSCLC, abiraterone with PC, and temsirolimus with RCC). In other cases, the survival metrics were not consistent within tumor types. In no tumor types did a given agent have the highest improvement in 1-year survival rate or the lowest NNT along with the highest improvement in mean OS. Also, in melanoma, NSCLC, PC, and RCC, improvements in 1-year survival rates appeared to outpace improvements in median OS, specifically for biologics and immunotherapy; this pattern was mixed in BC and less apparent in CRC.
In many cases, survival metric results varied by agent within each tumor type (Table 1, see bolded numbers). For BC, ado-trastuzumab emtansine demonstrated the greatest improvement in median OS, pertuzumab in mean OS, eribulin mesylate in 1-year survival rate; trastuzumab demonstrated the lowest NNT. For CRC, bevacizumab demonstrated the greatest improvements in median OS and 1-year survival rate, as well as the lowest NNT, whereas cetuximab had the greatest improvement in mean OS. Capecitabine, the only agent assessed in CRC with a nonplacebo comparator, showed the least improvement across all 3 outcomes. For melanoma, ipilimumab showed the greatest benefit with all 3 outcomes. For NSCLC, pemetrexed exhibited the greatest improvements of median and mean OS, whereas erlotinib showed the greatest improvement in 1-year survival rate and the lowest NNT. For PC, enzalutamide demonstrated the greatest improvement in median OS, sipuleucel-T showed the greatest improvement in mean OS, and abiraterone had the greatest improvement in 1-year survival rate and the lowest NNT. For RCC, sunitinib demonstrated the greatest improvements in both median and mean OS, whereas temsirolimus had the greatest improvement in 1-year survival rate and the lowest NNT.
Cost-Value Analysis
The results of the cost-value analysis varied depending on the applied metric, as some agents achieved a higher than average cost value (appearing above the fitted regression line) with some metrics, but not with others (Figure). The greatest cost value based on median OS, mean OS, and NNT was provided by ado-trastuzumab emtansine (second-line BC; Figure [a]), ipilimumab (first- and second-line melanoma; Figure [b]), and ipilimumab (first- and second-line melanoma; Figure [c]), respectively. Higher clinical value based on the individual survival metrics (Table 1) did not necessarily translate to a higher cost-value benefit (Figure).
The cancer therapeutic landscape is changing, with novel agents being introduced with differing response durability and disease outcomes. When assessing treatment, it is important to look at survival metrics by disease category because baseline outcomes differ so remarkably (eg, in the last decade, median OS was 25 months in metastatic BC30 vs 8 months in metastatic melanoma31). Our analysis suggests a pattern of evolving disease outcomes (eg, increasing 1-year OS rates that outpace increasing median and mean OS) aligned by disease and its relative emerging therapeutic landscape. Although metastatic melanoma, NSCLC, and RCC have been historically recognized as diseases with very poor survival rates, the advent of new agents is shifting the placement and contour of the survival curve.16-18
Several metrics are currently used in cancer trials, each with positive and negative characteristics (Table 2); among them, median OS is the most commonly used metric.22 However, median OS requires a long data collection period and a large sample size,19 and it may underestimate the full survival benefit because it overlooks patients who are alive at the end of the study follow-up.10,16,32 This scenario may be encountered with new treatments that prolong survival in a nontrivial proportion of patients (eg, immuno-oncology agents).16-18 For example, ipilimumab (a monoclonal antibody that blocks the immune checkpoint inhibitor molecule cytotoxic T-lymphocyte antigen-4) demonstrated a plateau in the survival curve that began approximately 3 years after initiating treatment in 21% of patients and continued for up to 10 years.17,33
Mean OS, which represents the area under the full survival curve, is more sensitive than median OS to the shape of the final portion of the curve, because it takes into account all patients, not just the surviving 50% (Table 2).10,22 Mean OS is also considered the standard metric for determining cost-effectiveness of cancer treatments,22 and, when estimated alone or as part of a cost-effectiveness analysis, it is required by many healthcare payers and health technology assessment agencies (eg, the United Kingdom’s National Institute for Health and Care Excellence).22 However, unlike median OS, mean OS is not as good an indicator of the central tendency of survival times.10,22
PFS was not included in our assessment tool, as it is frequently considered a surrogate outcome and a proxy for median OS (Table 2). Its role in estimating OS frequently comes into question, however, as PFS is subject to measurement error. It is possible that some agents—especially those targeting cell signaling and angiogenesis—may, with chronic administration, delay progression for a time, but lead to evolutionary changes in tumors, thereby producing a more aggressive phenotype, and subsequently offsetting the earlier delay in progression.34
Survival rate can be highly effective in comparing therapies within a tumor type,35 and NNT is useful for assessing the effectiveness of agents across populations (Table 2).27,28 However, survival rate does not specify whether survivors are still undergoing treatment or have achieved remission, and NNT is limited by difficulties in interpreting results when the treatment or follow-up period is not stated,36,37 as well as because it is not an absolute value and depends on the comparison of 2 treatment groups.38
For this cost-value analysis, we assigned a cost to the therapy based on a reported cost per month multiplied by the duration of therapy. The duration of therapy was based largely on data availability and varied between median duration of administration, median PFS, and median time to progression. Using medians for assessing duration of therapies would theoretically lead to underestimations of costs compared with analysis based on means. However, if costs are estimated similarly for all comparators, the bias should be fairly consistent across treatments.
Our preliminary quantitative and qualitative assessment shows that there is no unique or preferred relative clinical value metric for anticancer agents, but rather that an expanded metrics portfolio may be required. As shown here, an individual agent can have a high relative clinical value using 1 survival outcome and low value using another. In particular, there is variation between median OS and the other metrics. To assess relative clinical value, we developed a new cost-value analysis tool that graphically plots total drug cost versus the survival outcome. Using this tool, a higher relative clinical value based on a particular survival outcome did not necessarily translate into a greater economic value, further underscoring that no single metric is optimal. Future research should seek to provide guidance on how to determine optimal sets of metrics to assess value, particularly within the context of a specific indication.
The relative value assessment (RVA) tool we describe may fit into the spectrum of other value assessment tools/metrics, such as the American Society of Clinical Oncology (ASCO) Value Framework,39 the European Society for Medical Oncology’s Magnitude of Clinical Benefit Scale (ESMO-MCBS),40 the National Comprehensive Cancer Network Value Pathways,41 the Institute for Clinical and Economic Review Value Assessment Project,42 and Memorial Sloan Kettering Cancer Center’s DrugAbacus. These value assessment tools/metrics differ with regard to methodology and parameters assessed. For example, the ASCO Value Framework compares a new agent with the current standard of care for an indication using efficacy and safety data derived from a prospective randomized trial,39 whereas the ESMO-MCBS assesses an agent’s value through survival, QOL, and safety data from comparative outcome studies (randomized/comparative cohort studies or meta-analyses).40 Although these value frameworks have their respective limitations (eg, lack of assessment of potential cost to patients and complex user methodology), each offers important insights into the value of cancer treatments that may aid treatment decision making. Our approach using a portfolio of outcomes molded to a specific cancer indication can supplement these solutions.
Although our results provide insights into the assessment of the relative value of anticancer agents, several factors limit the data interpretation. First, determination of clinical value is subjective and can be approached in a number of ways, not just the ones used here. Additionally, the data sets in this analysis may not be fully mature because they were derived from phase 3 studies with predefined cut-off periods. As patient-level data were not available for all agents, mean OS, 1-year survival rate, and NNT were also estimated from digitized Kaplan-Meier curves rather than actual data points; this may result in underestimation of mean survival due to truncation of the data. Furthermore, the analysis incorporated indirect (naïve) comparisons among trials that failed to account for a tumor-specific prognosis and differences in patient characteristics (eg, number of lines of prior therapy). This analysis may also suffer from selection bias because evaluated agents were required to meet the predefined inclusion criteria and may not have been representative of the entire treatment landscape. Moreover, the cost value assigned did not take into account symptom burden, drug toxicity, or QOL, which is particularly important with newer therapies that are frequently less toxic than traditional chemotherapies. In real life, patients with cancer are treated with a variety of drugs simultaneously, making it difficult to attribute value to an individual drug. Hence, results of this model pertain to value expectations at the beginning of therapy, and should not be used to guide continued treatment. Lastly, our cost-value analysis did not take into account insurance discounting of drug costs.
In a healthcare environment hampered by intensive budgetary constraints, stakeholders struggle to contain costs while providing the best care possible. As novel and more effective oncology products—many of which have high price tags—are introduced, new methods for estimating relative clinical value are sought. Our preliminary qualitative and quantitative analysis, which used more and different metrics than may be the standard, suggests that a broad array of survival outcomes are required to fully assess and benchmark the relative clinical value of anticancer agents. This approach becomes progressively more important as drugs transition from clinical development to regulatory approval and widespread application. The portfolio of measures assessing impact needs to be more broadly meaningful in general populations; our concept of a measure portfolio starts to move in that direction. Including more therapeutic areas, basing the models on actual data points, and incorporating QOL measurements and other patient-focused concerns could further enhance this tool. Further research should concentrate on aligning best-value metrics and creating guidelines for prioritizing metrics when results differ. A mature RVA would enable more informed decisions by payers and providers in managed care, while guidelines for prioritizing metrics may decrease disagreement between stakeholders.

Professional medical writing assistance was provided by Mark Palangio and professional editing assistance was provided by Matthew Dougherty at StemScientific; these were funded by Bristol-Myers Squibb. This study was presented in part at the International Society for Pharmacoeconomics and Outcomes Research 19th Annual International Meeting, May 31 to June 4, 2014, Palais des Congrès de Montréal, Montréal, QC, Canada.
Print | AJMC Printing...