Scott D. Ramsey, MD, PhD

The cost of cancer care is one of the fastest growing components of healthcare spending in the United States, with expenditures expected to increase from $125 billion in 2010 to an estimated $158 billion in 2020.

 With recognition that there is varying and often suboptimal use of costly cancer care services, several groups have called for interventions to enhance the value of cancer care by improving outcomes while maintaining or reducing costs faced by patients, payers, and society.

 Efforts to improve value will entail establishing metrics that capture value, implementing programs to improve value, and tracking trends over time. However, agreeing on the denition of “value” in cancer care, and how it should be measured, is a potentially contentious process. With these issues in mind, we describe a multistakeholder, data-driven, research-enabled regional network that is focused on dening, measuring, and improving value in cancer care for patients, their families, and society. We believe that our “bottom-up” regional strategy may provide a model for developing, implementing, and evaluating promising interventions aimed at improving cancer care.

In 2013, the Fred Hutchinson Cancer Research Center’s Hutchinson Institute for Cancer Outcomes Research (HICOR) established a regional consortium consisting of cancer care providers, payers, patient advocates, and researchers committed to improving outcomes and reducing costs of cancer care: the Value in Cancer Care Initiative. The rationale for the multistakeholder process is grounded in theory and experience that a change in practice is best achieved by direct participation by those who are impacted and who have both control of, and a stake in, the process.

 A steering committee with representatives from the stakeholder groups organized a regionwide meeting to introduce the concept of improving value in cancer care. The meeting, held in January 2014, included nearly 70 individuals representing 20 organizations from across the region.

 The meeting organizers tasked this group with identifying 3 to 5 metrics capable of measuring “value” from their perspective. Rather than focus on a single definition of value,

 the organizers asked stakeholders to consider their own denition of value, with the caveat that a value metric must minimally consider costs and clinical outcomes from their perspective as patients, payers, or providers. Additionally, attendees were informed that the value metrics should be both meaningful from their perspective—ie, represent issues that improve either the efficiency of care, or the lives of cancer patients, or both&mdash;as well as actionable, meaning amenable to intervention within the healthcare system. Importantly, the organizers stressed that it should be feasible to generate the metrics without undue burden on any 1 stakeholder group. It would be the task of HICOR to create the final metrics and report back on them at the next summit.

During the conference, multistakeholder “working groups” independently created lists of possible value measures across 3 phases of cancer care: treatment, survivorship, and end of life. Although approximately 750 metrics within 9 domains were initially offered by the working groups, a final prioritization exercise involving the full group identied 6 domains as standouts on the 2 dimensions of meaningfulness and actionability (

LESSONS LEARNED: KEY ELEMENTS OF THE STAKEHOLDER ENGAGEMENT MODEL FOR GENERATING VALUE METRICS 

The aspirational goal of improving outcomes in cancer care while reducing costs confronts the reality that while diverse stakeholder groups must work together to achieve this goal, they may have competing or even diametrically opposed interests and goals. A health insurer’s cost can be a clinician’s revenue; patients may distrust the actions of insurers and even physicians to restrict “unnecessary” care if they feel it threatens their options or potential well-being. Therefore, the necessary task of trying to achieve concordance on which metrics are worthy of attention and support is often delegated “top- down” from a single stakeholder group, creating antagonism early in the process and ultimately reducing the likelihood of achieving constructive change. We have outlined several aspects of our process, listed below, that we believe are replicable in other regions and that may increase the likelihood of implementation and ultimately positive change. The latter goal remains to be proved.

1. Bottom-up multistakeholder engagement in all phases of the process:

• Defining relevant clinical questions

• Selecting potential interventions to improve care where the value is low

2. Emphasis on actionable and meaningful metrics. Actionable measures create near-term opportunities to improve outcomes and lower costs. Meaningful metrics provide all stakeholder groups with the opportunity to support them from the perspective of their own organization. The most meaningful metrics are self-apparent (eg, survival) and thus require less translation when tools have been developed to capture them, summarize the results, and disseminate the findings. Other metrics require translation, which should be done from the patient’s perspective. For example, “chemotherapy at the end of life” might be better referred to as “overtreatment,” to convey the notion that the treatment does not improve quantity or quality of life.

3. Willingness to accept some metrics that might not be personal priorities or may be difficult to define and implement. Stakeholders were reminded that no single metric can be everyone’s top priority. From a research perspective, some domains from the 

 will be difficult to define in the short term but can be refined over time as other measures are reported.

4. Manageable number of metrics. Performance measurement is burdensome to systems that must collect the data, especially if measures are not aligned with data that are routinely collected or that interfere with work flows. Large numbers of performance measures can quickly overload systems and may reduce participation and the likelihood of meaningful change related to anyone. This can be discouraging and reduce cooperation with future performance measurement efforts.

5. Third-party facilitation. HICOR positions itself as a neutral third party that serves all stakeholder groups; thus, the fact that the organization is affiliated with a cancer care delivery system could be seen as problematic by competitors and health insurers. We address this issue with transparency of our methods. There is probably no solution that is completely free of all bias and conflicts of interest.

The metric development process within HICOR’s Value in Cancer Care Initiative is the first step in a larger vision of designing, implementing, and evaluating approaches to improve cancer care. Like the process of identifying the metrics themselves, we believe the interventions aimed at improving performance should be developed through the multistakeholder engagement process, supported by experts in cancer care delivery, and conducted as prospective, controlled studies. We describe these elements and our plans below.

Using commercial and public health insurance enrollment and claims records linked to the region’s Surveillance, Epidemiology, and End-Results cancer registry, HICOR is quantifying metrics in 4 of the 6 domains shown in the 

. The active participation of the health insurance community has facilitated timely transfer of insurance records, making the available metrics as close to “real time” as possible. Based on input from the clinical community, metrics are available at the clinic level (vs the provider level). Metric development is an iterative scientic approach that requires the engagement and expertise of clinical, statistical, and scientic programming professionals. The process of creating and reporting metrics requires careful thought, since difficulties with numeracy have been shown to be a barrier to understanding data for laypersons and even professionals.

 A regional, stakeholder-based approach allows the opportunity for individual participation and feedback to the approach as it unfolds, and for one-on-one interaction and modication when necessary to improve the face validity of the metrics.

Initially, HICOR is reporting the methods and results privately to the clinics, to ensure that data quality and accuracy issues can be resolved before public reporting of clinic-level records. However, HICOR is dedicated to publicly reporting clinic-specific performance metrics and to providing consumer and patient-facing tools to support decision making such as providing estimates of out-of-pocket costs for treatments with equivalent outcomes. Optimal display of the information is critical to the overall process. Displaying complex data to a diverse audience has necessitated the engagement of experts in data visualization and user-centered design. For example, providers may wish to monitor their adherence to performance metrics regularly so that clinics can evaluate their progress toward pre-specied goals. Patients, on the other hand, may use performance measurement as a way of selecting providers, or as an opportunity for patient-provider interaction regarding particular aspects of care (eg, tumor marker testing following treatment of local-stage breast cancer). To address these issues, HICOR is developing a Web-enabled tool to display reports generated from the linked database noted above. This tool, aimed at providing regional partners with updated reporting on trends in oncology, is now in the beta testing phase with partners. The ultimate goal: to provide results on an interactive website accessible to patients, providers, health insurers, and the purchaser community.

Using the Metric Infrastructure as a Platform for Research-Driven Approaches to Improve Care and Reduce Costs 

Performance measurement can be used for Continuous Quality Improvement (CQI) programs, with an emphasis on observational methods and pre- and post evaluations. Our model of stakeholder engagement and regularly updated data linkage provides an opportunity for more formal research designs such as prospective, controlled trials and evaluation of interventions designed to improve quality and reduce costs. When feasible, our goal is to conduct randomized controlled experiments, implemented in the practice setting, and monitored using existing data systems that are already part of standard work flows (eg, insurance claims and electronic medical records)—previously referred to as pragmatic trials.

 We believe participation of regional stakeholder groups in the design of the intervention and actual conduct of the trial is key to this process. Input from clinical oncology researchers and experts in trial design will support the development of interventions that can be tested in randomized trials and formally evaluated.

Robustness of the Model to a Wide Variety of Interventions 

Many different interventions may be effective in improving oncology care where value or performance fails to meet expected targets. Examples range from novel payment models (eg, pay-for-performance methods) to delivery system changes (eg, automated order entry systems) to tools for patient or provider behavior change (eg, interactive videos, shared decision-making tools). Our network provides us with the opportunity to test and monitor the impact of a wide range of interventions in clinical practice settings, and then to scale up those interventions that appear promising for formal testing via controlled studies. Interventions that do not show improvement can be quickly shelved. Such rapid evaluation and redirection of effort can be difficult and cumbersome in the traditional CQI model.

The Value in Cancer Care Consortium is the rst step toward linking national quality and value efforts with real-world clinical practice and patient experience in a local region. We can rapidly develop and evaluate practical, applied models for translating policy to practice in a grassroots or real-world setting. While healthcare policies are often developed nationally, the implementation of those policies and their impact on patients, providers, and payers is most evident at the local or regional level, potentially allowing more rapid adaptation or intervention. Our strategy provides a real-world laboratory for developing and accessing promising interventions to improve care. Those that are successful can then be scaled and evaluated elsewhere, such as within integrated regional delivery systems, in clinical trial research networks, or through federal programs such as CMS/Center for Medicare and Medicaid Innovation programs.

The cost of cancer care continues to increase at an unsustainable rate, and the quality of cancer care delivery has been found to vary considerably. The need for efforts to improve the value of cancer care by improving outcomes while controlling or reducing costs has never been more apparent. In partnership with multiple committed stakeholders—patient groups, health plans, and providers&mdash;HICOR is generating consensus-based, clinically relevant, and actionable performance measures. These measures can be employed to:

• Design and evaluate practice changes in healthcare delivery

• Monitor the impact of the changes at the clinic, provider, and patient levels

• Develop new interventions, including performance improvement programs, as well as actual experiments, rapidly and efficiently

This multistakeholder regional partnership is providing a real-world platform for the development and evaluation of promising interventions to improve the quality and value of cancer care in the 21st century, both regionally and nationally. 

Scott D. Ramsey, MD, PhD, is director, Hutchison Institute for Cancer Outcomes Research, Division of Public Health Sciences, Fred Hutchinson Cancer Research Center, Seattle, WA; professor, Department of Medicine, University of Washington, Seattle, WA.  

Gary H. Lyman, MD, MPH, is co-director, Hutchison Institute for Cancer Outcomes Research, Division of Public Health Sciences, Fred Hutchinson Cancer Research Center, Seattle, WA; professor, Department of Medicine, University of Washington, Seattle, WA.

1. Mariotto AB, Yabroff KR, Shao Y, Feuer EJ, Brown ML. Projections of the cost of cancer care in the United States: 2010-2020. 

2. Committee on improving the quality of cancer care: addressing the challenges of an ageing population. Delivering High-Quality Cancer Care: Charting A New Course For A System In Crisis. Washington, DC: Institute of Medicine, Board on Health Care Services; 2013.

3. Schnipper LE, Smith TJ, Raghavan D, et al. American Society of Clinical Oncology identifies five key opportunities to improve care and reduce costs: the top five list for oncology. 

4. Schnipper LE, Lyman GH, Blayney DW, et al. American Society of Clinical Oncology 2013 top five list in oncology. 

5. ASCO in action brief: value in cancer care. American Society of Clinical Oncology website. http:// www.asco.org/advocacy/asco-action-brief-value- cancer-care. Published January 21, 2014. Accessed March 16, 2015.

6. Thariani R, Wong W, Carlson JJ, et al. Prioritization in comparative effectiveness research: the CANCERGEN Experience. 

7. Deverka PA, Lavallee DC, Desai PJ, et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. 

Articles

Creating Value-Based Metrics for Cancer Care: A Stakeholder-Informed, Evidence-Driven Approach

	From his vantage point as director of the Cancer Technology Assessment Group at the Fred Hutchinson Cancer Research Center in Seattle, Washington, 

sees the financial fallout of cancer care like few do. An internist and health economist, Ramsey asserts that the entire approach to clinical trials in recent years has contributed to escalating costs of drugs, and that this trend cannot continue. His talk was titled, “Where Does HEOR Fit in the Oncology Model?”

	The connection between economics and healthcare is broken, at least in the United States; so, too, is the pricing model for pharmaceuticals, according to 

 He began his talk by describing the futility of even trying to find a good definition of health economics outcomes research (HEOR) in the literature; in his view, clinical research has had an increasingly narrow focus on whether new cancer drugs produce a short-term benefit, with little regard paid to whether that benefit is worth it financially. This must end, he said.

	“Unless we look at value, unless we look at cost, the outcomes research (OR) is somewhat pointless, because we’re not going to be able to make good decisions and improve value for our patients until we take cost into account, and what we get for what we pay,” Ramsey said.

	Ramsey said that while some in medicine are having discussions about whether new cancer drugs are worth the price, in clinical trials, the structure often looks at end points that are favorable to the pharmaceutical companies—regardless of whether the realworld results that include price have value for patients.

by a Canadian team that found trials are increasingly using measures of progression-free survival (PFS), “which may or may not correlate with symptoms.” The authors found that since the 1980s, when PFS was never an end point, this end point accounted for “nearly a quarter of all the trials in the 2000s.”

 In the palliative setting, the share rose from zero to 45% in that same period, he said (

	“If progression-free survival correlated precisely with patient quality of life or overall survival, that might make sense to me, but we know from multiple studies that that is a very inconsistent relationship,” Ramsey said. “We’re getting away from measuring things that I think patients value the most.”

	By contrast, data that show overall survival (OS) in metastatic colorectal cancer increasing from 8 to 18 months since the 1990s represent “real progress.”

	The problem, as Ramsey sees it, is the lack of relationship between meaningful survival results and soaring drug costs. “The costs of the drugs are increasing regardless of the incremental benefit,” he said. Groups like the National Comprehensive Cancer Network (NCCN) do not help when they equate a drug that costs $840 a month with a few side effects to a string of oral therapies that together cost $60,000 for a treatment course.

	“What we have is a disconnect between what I think would be the ideal and what is real,” he said. US drug development and approval processes differ vastly from those in Europe, which take into account what support payers are willing to provide. By contrast, the biggest US payer of all—the Centers for Medicare & Medicaid (CMS)&mdash;is politically hamstrung from negotiating directly with drug companies.

	The question is: “Will the change in our payment paradigms change the health economics and outcomes research paradigm? In other words, will it make health economics more relevant? And my answer is, I’m not certain.”

	So far, risk sharing takes place between the payer and the patient, but pharmaceutical companies do not accept a role. This creates no incentives to contain prices, Ramsey said.

	If nothing changes, “We’re going to have to stop giving patients access; oncology practices are going to go bankrupt; patients are going to go bankrupt. It’s just not a supportable system.” 

	1. Booth CM, Eisenhauer EA. Progression-free survival: meaningful or simply measurable? JCO. 2012;30(10):1030-1033.

From his vantage point as director of the Cancer Technology Assessment Group at the Fred Hutchinson Cancer Research Center in Seattle, Washington, Scott D. Ramsey, MD, PhD,sees the financial fallout of cancer care like few do.

Putting the 'Economics' Back Into Health Economics Outcomes Research

According to Dr Ramsey, both the Health Economics Outcomes Research (HEOR) model and the pharmaceuticals pricing model in the United States are broken. He says: “Unless we, as a community, figure out a way to use health economics in our decision making, we are not going to stop this train of unsupportable price increases in pharma.”

HEOR is defined as a multidisciplinary approach that is aimed at estimating value. This value usually includes costs and outcomes, as well as factors such as looking at the budget impact of new technologies, patient-reported outcomes, practice patterns, comparative effectiveness, and cost- effectiveness.

Dr Ramsey argues that unless we look at value and cost, the outcomes and research have little meaning. We cannot make good decisions and improve value for patients until cost and quality are taken into account.

“I think we need to put the H-E back into HEOR and figure out ways, as a community, for using that as a leverage point for breaking the current price trend in oncology,” he says.

According to Dr Ramsey, both the Health Economics Outcomes Research (HEOR) model and the pharmaceuticals pricing model in the United States are broken. He says: Unless we, as a community, figure out a way to use health economics in our decision making, we are not going to stop this train of unsupportable price increases in pharma.

Presentation 7: Where Does HEOR Fit in the Oncology Model?

 To systematically assess clinical and economic evidence for oncology orphan drugs marketed in the United States and to highlight the challenges and opportunities for evidence development within this pharmaceutical category.

 We conducted systematic literature searches of the Medline and Embase databases for clinical and cost-effectiveness studies published before June 2010 for all oncology orphan drugs marketed in the United States. We used the Grading of Recommendations Assessment, Development and Evaluation method and the Quality of Health Economic Studies criteria to assess the quality of the selected studies.

We identified 60 randomized controlled trials and 21 cost-effectiveness analyses to support 47 oncology orphan drugs. A total of 21 drugs had moderate or high-quality bodies of clinical evidence, 11 had low-quality or very low quality clinical evidence, and 15 drugs could not be evaluated because we were unable to identify clinical evidence that met our inclusion criteria. The Spearman rank correlation coefficient for the level of evidence for oncology orphan drugs and disease prevalence was 0.3 (95% confidence interval, 0.0-0.5). The cost-effectiveness analyses received quality scores between 72 and 100 (range 0-100), with a mean score of 85.

The results of our study show that oncology orphan drugs marketed in the United States have varying levels and quality of clinical evidence and a paucity of evidence regarding economic value. Innovative analytic and policy approaches are needed to develop and implement a decision-making framework for this pharmaceutical category that is consistent with evidence-based medicine and comparative effectiveness research.

	We conducted a systematic literature review to critically assess the clinical and economic evidence supporting oncology orphan drugs marketed in the United States.

		Oncology orphan drugs marketed in the United States have varying levels and quality of clinical evidence, and a paucity of evidence demonstrating their economic value.

		The current levels of clinical and economic evidence present challenges for decision making about oncology orphan drug availability and accessibility.

		Innovative analytic and policy approaches are necessary to develop and implement a decision-making framework for oncology orphan drugs that is consistent with evidence-based medicine and comparative effectiveness research.

	Comparative effectiveness research (CER) can be described as the assessment of a medical intervention against alternative interventions with the intent of identifying treatment strategies that are likely to have preferable benefit-risk profiles or are considered cost-effective in real-world clinical settings. The purpose of CER is to assist healthcare providers, payers, patients, and decision makers in making informed healthcare decisions that will improve individual and population health.

	Definitions of a rare disease differ around the world and the prevalence threshold varies between countries.

 In the European Union, a disease is considered rare if it affects fewer than 215,000 individuals, while in the United States, a rare disease is described as a condition with prevalence of less than 200,000 or a disease with distinct subpopulations consisting of fewer than 200,000 individuals nationwide.

 As of 2010, 362 drugs indicated for rare diseases (orphan drugs) have received US Food and Drug Administration (FDA) market approval, and oncology therapies comprise the largest clinical subcategory.

 In general, orphan drugs have relatively higher costs than other drugs because manufacturers must rely on smaller patient populations to recoup development investments. In the United States, 15 orphan drugs were commercialized between 2006 and 2008; 6 of these cost more than $100,000 per patient per year.

	Previous studies have suggested that orphan drugs have less robust bodies of evidence because smaller patient populations and limited knowledge of rare conditions constrain the design, conduct, analysis, and interpretation of clinical trials.

 Due to the rapid rate of oncology orphan drug development and the significant financial burden associated with cancer treatments, there have been increasing pressures on drug manufacturers to demonstrate the value of their products. Less robust bodies of evidence will present particular challenges to CER and will make decision making about orphan drug accessibility difficult. The comparative effectiveness data supporting oncology orphan drugs marketed in the United States have not been well studied. The primary objective of this study was to systematically and critically assess the level and quality of clinical and economic evidence currently available for all oncology orphan drugs marketed in the United States. A secondary objective was to highlight the challenges and opportunities for evidence development within this pharmaceutical category.

	The Cumulative List of Designated Approved Orphan Products (www.fda.gov/orphan/designate/allap.rtf) describes drug products that have received orphan designation and have ever received marketing approval from the FDA.

	We used this list, updated May 5, 2009, by the FDA, to identify products indicated to treat rare cancers. Orphan products indicated for diagnosis, palliative care, or treatment of secondary conditions associated with cancer, such as neutropenia, were not included in this study. Any product withdrawn or discontinued from the US market as of June 2010 was also excluded.

	For each oncology orphan drug included in this study, we conducted a literature search in the Medline and Embase databases for randomized controlled trials (RCTs) and cost-effectiveness analyses published prior to June 2010, using search terms that included the drug’s US brand name, generic name, disease indication, and the terms “randomized,” “efficacy,” “cost-effectiveness,” and “economic.” Our priority was to identify RCTs, but we also used the literature search to identify observational studies (prospective or retrospective) or other studies that described the treatment effect of the drug. We also identified published articles through information provided on the FDA Web site for new drug approvals and manual searches of article references. We defined cost-effectiveness analysis using the definition established by the Panel on Cost-Effectiveness in Health and Medicine as “An analytic tool in which costs and effects of a program and at least one alternative are calculated and presented in a ratio of incremental cost to incremental effect.”

	For each literature search, 1 author (MMC) reviewed all of the titles and abstracts of studies likely to meet the following inclusion criteria: (1) original RCT or cost-effectiveness analysis evaluating the orphan drug used for its approved orphan indication, or observational study or published article describing the treatment effect of the orphan drug used for its approved orphan indication; (2) comparative RCTs in which the orphan drug is compared with placebo or a clinically relevant comparator; (3) if the orphan drug (eg, drug A) is used as part of a combination therapy regimen, the trial is designed so that the clinical effect of the orphan drug can be isolated and directly assessed (eg, drugs A, B, and C vs drugs B and C); (4) article is published in the English language; and (5) entire article is available for review. 

 presents a flow diagram that describes the literature search methods and the restrictions applied to our search.

	For each clinical study that met the inclusion criteria, we abstracted information about the comparator, patient characteristics, study design and treatment allocation, primary outcome measure, statistical analytic method, reporting of treatment effect with uncertainty, and study sponsor. For each cost-effectiveness analysis that met the inclusion criteria, we abstracted information about the comparator, study perspective, methods, data sources, primary outcome measure, base-case and sensitivity analysis results, and study sponsor (if reported).

	We used the Grading of Recommendations Assessment, Development and Evaluation (GRADE) method, an evaluation system created by a diverse group of international guideline developers, to assess the quality of clinical bodies of evidence. The GRADE system assesses the quality of a body of evidence by focusing on 4 main components: study design, quality, consistency of evidence, and directness of comparator, population, and intervention. The method also evaluates limitations, potential biases, and uncertainty to assign 1 of 4 possible grades: high, moderate, low, and very low.

 We adapted the grading system to our study by including a fifth grade category, “not able to assess,” to describe circumstances where we could not identify published studies that met our inclusion criteria. 

 details the GRADE methodology and how it was implemented in this study, and defines each quality grade.

	We used the Quality of Health Economic Studies (QHES) grading criteria, a validated quantitative instrument developed by health economists, to assess the quality of cost-effectiveness studies. The QHES grading system consists of 16 criteria that are described in 

 Each criterion is associated with a weighted score that was\ assigned in its entirety for each criterion perceived by the lead author to be satisfactory. Zero points were assigned to each criterion that was perceived to be unsatisfactory. Certain criteria, such as subgroup analysis, were not applicable for all studies. In these circumstances, we used both a best-case and worst-case scoring method where full points and zero points were assigned to each inapplicable criterion. The weighted scores were summed across all criteria for a total of 100 possible points for each study.

	We graphically explored whether disease prevalence was associated with the level and quality of evidence for oncology orphan drugs and calculated the Spearman rank correlation coefficient to estimate the strength of the relationship. We obtained estimates of disease prevalence from Orphanet, an electronic reference for rare disease information in Europe (www.orpha.net).

 This resource was the only one we identified that provided a collection of prevalence estimates for rare diseases. Although the data were derived in Europe, we believe the estimates are of similar magnitude in the United States, assuming similar risk factors for rare cancers. The assessment of correlation excluded 5 drugs: aldesleukin for metastatic melanoma, valrubicin for urinary bladder carcinoma, doxorubicin liposome for metastatic ovarian cancer, and toremifene citrate and exemestane for breast cancer. These cancers are not considered rare. The drugs are indicated for smaller subpopulations of patients with specific disease, physical, or genetic characteristics. Prevalence estimates for patients with these specific characteristics were not reported. We also explored whether the level and quality of evidence for oncology orphan drugs is associated with receipt of FDA accelerated approval.

	We initially identified 48 oncology orphan drugs for inclusion in this study from the Cumulative List of Designated Approved Orphan Products. Of these drugs, 3 (Idamycin, Vesanoid, and Mylotarg) have been discontinued or withdrawn from the US market, and 5 drugs (Gleevec, Sprycel, Tasigna, Velcade, and Temodar) were indicated to treat more than 1 orphan indication or patient population; each indication was independently evaluated for these drugs. We did not include expanded or supplemental approvals for nonorphan indications. In total, 47 orphan drugs and indications were included in this study and are listed in 

 by their brand and generic names. The majority of oncology orphan drugs are indicated for rare blood cancers (n = 30). Table 3 also describes the date of full and/or accelerated regulatory approval, and lists the number of RCTs identified in the published literature that met the inclusion criteria with citations of all published articles that were used to inform the body of evidence.

	We identified a total of 60 RCTs that met our inclusion criteria. The greatest number of RCTs were conducted for interferon alpha-2a indicated for chronic myelogenous leukemia (CML) (n = 7), aldesleukin indicated for metastatic renal cell carcinoma (n = 7), and rituximab indicated for non-Hodgkin B-cell lymphoma (n = 6). The majority of trials we identified comprised the evidence base used for regulatory approval.

 summarizes the clinical evidence grades assigned in this study. Twelve drugs had moderate-quality evidence, including cladribine for hairy cell leukemia and arsenic trioxide for acute promyelocytic leukemia. Although we did not identify any RCTs for these drugs that met our inclusion criteria, their clinical bodies of evidence consisted of observational studies that consistently reported similar treatment effects or a dose-response association with little threat to validity. Individual GRADE ratings assigned to each drug and their clinical bodies of evidence are listed in Table 3.

 shows that there are drugs indicated for cancers with lower prevalence that had high-quality evidence and drugs indicated for cancers with higher prevalence that had low-quality evidence or lacked a sufficient body of evidence for review. The Spearman rank correlation coefficient between disease prevalence and the level of evidence for oncology orphan drugs was 0.3 (95% confidence interval [CI], 0.0-0.5). This indicates that there may be a weak correlation between the prevalence of rare cancers and the number of RCTs conducted for drugs indicated to treat rare cancers. A total of 14 drugs in our study received accelerated approval for the orphan indication. Of these, zero had high-quality bodies of evidence and 5 had moderate-quality bodies of evidence. Although our sample size was small, these results provided some indication that the accelerated approval provision does not optimize high-quality evidence generation for oncology orphan drugs.

	We identified 21 cost-effectiveness studies that met our inclusion criteria. Of those studies, 10 evaluated the use of either interferon alpha or imatinib mesylate, or compared both for CML. 

 summarizes the cost-effectiveness studies included in this assessment and describes comparators, study perspectives, base-case incremental cost-effectiveness ratios, and results of sensitivity analyses.

	The cost-effectiveness analyses received QHES scores ranging from 72 to 100, with a mean score of 85. The clinical data used to inform the studies were largely obtained from either randomized trials or patient medical records. The majority of studies implemented modeling techniques to project longer-term health outcomes that could not be obtained from limited clinical data. Cost inputs were obtained from administrative databases and the published literature. The most common shortcomings were as follows: 11 studies used data inputs from sources that might potentially incorporate bias and did not provide explicit discussion about the direction and magnitude of potential biases; 6 studies did not specify the source of funding for the study, which is recommended to add transparency; and 5 studies either failed to specify the time horizon for analysis, conducted analysis using a short time horizon that is not expected to fully capture clinically significant costs and effects, or did not implement adequate discounting for both future costs and outcomes. Table 4 reports the quality scores assigned to each cost-effectiveness study using the best-case scoring method where full points were assigned to inapplicable criteria. Scores assigned using the worst-case approach, where zero points were assigned to inapplicable criteria, are reported in parentheses.

	We conducted a systematic literature review to identify and assess clinical and economic evidence for 47 oncology orphan drugs marketed in the United States and applied 2 independent grading frameworks to the selected studies to critically assess the quality of each body of evidence. The supporting bodies of evidence available for marketed oncology orphan drugs vary in quality, with limited evidence demonstrating their economic value.

	Although the GRADE methodology is relatively explicit, reviewer judgment is required. In particular, we found that it was necessary to grade certain criteria subjectively in order to better accommodate oncology orphan drug characteristics. For example, the majority of RCTs for oncology orphan drugs had relatively small sample sizes that could lead to substantial imprecision (eg, lack of power, wide confidence intervals). We assigned lower ratings for studies that did not meet predetermined enrollment criteria or appeared underpowered to detect differences in their primary end point, but we did not consider a smaller sample size in itself to be a limitation. We critically assessed each study and lowered ratings when we identified instances of bias that were not explicitly addressed. We also lowered ratings when studies were poorly reported, or when bodies of evidence consisted of studies that reported inconsistent treatment effects.

	The number of RCTs identified in our study is comparable to the number in a study conducted by Tsimberidou and colleagues that evaluated the long-term marketing outcome of oncology drugs approved by the FDA without a randomized trial.

 In their study, 68 approved oncology drugs were identified and almost half (n = 31) were approved without an RCT. Kesselheim and colleagues recently evaluated pivotal trial characteristics of orphan and nonorphan drugs approved between 2004 and 2010 to treat cancer and concluded that pivotal trials for approved oncology orphan drugs were more likely to be smaller, nonrandomized, and unblinded, and to use surrogate end points to assess efficacy.

 The study is limited in that it only evaluated pivotal trials and did not account for other available clinical information. Also, the study was restricted to recently approved oncology orphan drugs and did not review the entire pharmaceutical category. We observed the same characteristics in the clinical trials that were included in our study. We also observed that there is a lack of published post-marketing studies and information about longer-term safety and efficacy.

	We hypothesized that the level of evidence available for oncology orphan drugs would have a strong correlation with disease prevalence in that larger patient populations enable better evidence generation. However, our results suggested a potentially weak correlation between the prevalence of rare cancers and the level of evidence available for drugs indicated to treat rare cancers. A weak relationship suggests that evidence development for oncology orphan drugs may not depend as much on the size of a particular patient population. Instead, evidence development in this pharmaceutical category may depend more on other factors, possibly regulatory requirements or reimbursement considerations.

	The overall dearth of cost-effectiveness studies in this pharmaceutical category, and in oncology as a whole, may reflect evidence limitations or publication bias, where studies of drugs with higher costs, greater benefit-risk uncertainty, or lower effectiveness are not published. In assessing the economic challenges of orphan drugs, Drummond and colleagues stated: “In short, if standard health technology assessment (HTA) procedures were to be applied to orphan drugs, virtually none of them would be ‘cost-effective’.”

 This conclusion was largely based on 2 factors: (1) high incremental cost per quality-adjusted life-year and (2) insufficient breadth and quality of clinical evidence for orphan drugs compared with drugs for more common diseases.

 Our results demonstrated that, contrary to these prior suggestions, it is feasible for some oncology orphan drugs to be considered cost-effective in specific healthcare settings using standard methods of health technology assessment. The clinical and economic value of each orphan drug should be assessed individually, on a case-by-case basis. We observed that all of the drugs considered cost-effective in their respective studies had moderate-quality to high-quality bodies of clinical evidence. This finding suggests there may be a relationship between the level and quality of clinical evidence and the likelihood that an oncology orphan drug has published estimates of cost-effectiveness.

	Evidence development is costly and challenging for all healthcare interventions, but particularly for orphan drugs, due to smaller patient populations and limited clinical knowledge of rare conditions. The trade-offs to generating more robust bodies of evidence may include delayed product accessibility, higher costs, or reduced availability of therapies. For conditions that have therapeutic alternatives, these trade-offs may be acceptable. However, for life-threatening conditions with limited therapy options, these trade-offs may not be considered acceptable. For rare diseases with only a single treatment option, one may question whether economic analysis should apply. Cost-effectiveness analysis is useful in that it provides information about the value of a health intervention compared with an alternative, which may be best supportive care or no therapy. From an equity perspective, McCabe and colleagues argue that there is no sustainable reason why the cost-effectiveness of orphan drugs should be evaluated differently from other drugs.

 However, healthcare systems may wish to consider additional factors when making decisions about orphan drugs, such as budget impact, disease severity, availability of alternative therapies, or societal preferences toward patients with rare diseases.

	Our study highlights 3 important policy questions. First, what types of study designs, incentives, or methods can be used to encourage better evidence development for oncology orphan drugs? Second, how much evidence is necessary or considered sufficient to healthcare decision makers, and what types of evidence should be generated prior to and after marketing approval? Finally, what are the process and decision-making criteria for evaluating comparative effectiveness data for oncology orphan drugs? Emerging private and public initiatives are attempting to address these questions to some extent with evolving methods such as coverage with evidence development

 but additional innovative analytic methods and policy approaches are necessary.

	We recognize several limitations to this study. The most important limitation is that the GRADE assessment framework provides information about the amount of confidence that can be associated with a body of evidence and does not relay any information about the magnitude of clinical benefit or safety of a drug. Certain oncology orphan drugs with low-grade published evidence may yield significant clinical benefits, and conversely, certain drugs with high-grade evidence may yield very little benefit. Currently, there is no established framework for quantifying the magnitude of benefits or risks from health interventions. Garrison and colleagues suggest pairing CER and benefit-risk analysis into 1 framework and describe how cost-effectiveness analysis models could be adapted to conduct quantitative benefit-risk assessments.

 It is important for manufacturers to engage in continuous evidence generation, even after a product is commercialized, so that the effectiveness and safety of their products can be accurately assessed. Comparative effectiveness research methods that capture and incorporate nonpublished clinician or patient experiences could also be useful to help better identify the real-world value of marketed oncology orphan drugs.

	Given the large number of drugs included in this review and the numerous keywords we could have used to search for potential clinical and economic studies, this review may not have captured all relevant drugs or studies. We used broad search terms in 2 major databases and also conducted manual searches of reference citations; this methodology allowed us to screen as many studies as possible. Only 1 author (MMC) reviewed potential studies, abstracted information, critically assessed articles, and assigned quality ratings. Although attempts were made to be accurate and consistent, it is possible that unintentional errors and inconsistencies could have reduced or improved the quality rating for a body of evidence. The QHES is limited in its ability to identify poorly analyzed studies and does not have a benchmark for total scores, which limits its ability to quantitatively categorize and distinguish high-quality studies from low-quality studies.

	The results of our study show that oncology orphan drugs marketed in the United States have varying levels and quality of clinical evidence and a shortage of evidence demonstrating economic value. It is uncertain whether the current evidence levels for oncology orphan drugs marketed in the United States are sufficient to support decision-making practices consistent with principles of evidence-based medicine and CER. This issue remains an open policy question that requires additional evaluation.

This study critically assessed the published clinical and economic evidence supporting oncology orphan drugs marketed in the United States.

Scott D. Ramsey, MD, PhD

Articles

Creating Value-Based Metrics for Cancer Care: A Stakeholder-Informed, Evidence-Driven Approach

Putting the 'Economics' Back Into Health Economics Outcomes Research

Presentation 7: Where Does HEOR Fit in the Oncology Model?

Systematic Review of Comparative Effectiveness Data for Oncology Orphan Drugs