Louis P. Garrison, PhD

 To systematically assess clinical and economic evidence for oncology orphan drugs marketed in the United States and to highlight the challenges and opportunities for evidence development within this pharmaceutical category.

 We conducted systematic literature searches of the Medline and Embase databases for clinical and cost-effectiveness studies published before June 2010 for all oncology orphan drugs marketed in the United States. We used the Grading of Recommendations Assessment, Development and Evaluation method and the Quality of Health Economic Studies criteria to assess the quality of the selected studies.

We identified 60 randomized controlled trials and 21 cost-effectiveness analyses to support 47 oncology orphan drugs. A total of 21 drugs had moderate or high-quality bodies of clinical evidence, 11 had low-quality or very low quality clinical evidence, and 15 drugs could not be evaluated because we were unable to identify clinical evidence that met our inclusion criteria. The Spearman rank correlation coefficient for the level of evidence for oncology orphan drugs and disease prevalence was 0.3 (95% confidence interval, 0.0-0.5). The cost-effectiveness analyses received quality scores between 72 and 100 (range 0-100), with a mean score of 85.

The results of our study show that oncology orphan drugs marketed in the United States have varying levels and quality of clinical evidence and a paucity of evidence regarding economic value. Innovative analytic and policy approaches are needed to develop and implement a decision-making framework for this pharmaceutical category that is consistent with evidence-based medicine and comparative effectiveness research.

	We conducted a systematic literature review to critically assess the clinical and economic evidence supporting oncology orphan drugs marketed in the United States.

		Oncology orphan drugs marketed in the United States have varying levels and quality of clinical evidence, and a paucity of evidence demonstrating their economic value.

		The current levels of clinical and economic evidence present challenges for decision making about oncology orphan drug availability and accessibility.

		Innovative analytic and policy approaches are necessary to develop and implement a decision-making framework for oncology orphan drugs that is consistent with evidence-based medicine and comparative effectiveness research.

	Comparative effectiveness research (CER) can be described as the assessment of a medical intervention against alternative interventions with the intent of identifying treatment strategies that are likely to have preferable benefit-risk profiles or are considered cost-effective in real-world clinical settings. The purpose of CER is to assist healthcare providers, payers, patients, and decision makers in making informed healthcare decisions that will improve individual and population health.

	Definitions of a rare disease differ around the world and the prevalence threshold varies between countries.

 In the European Union, a disease is considered rare if it affects fewer than 215,000 individuals, while in the United States, a rare disease is described as a condition with prevalence of less than 200,000 or a disease with distinct subpopulations consisting of fewer than 200,000 individuals nationwide.

 As of 2010, 362 drugs indicated for rare diseases (orphan drugs) have received US Food and Drug Administration (FDA) market approval, and oncology therapies comprise the largest clinical subcategory.

 In general, orphan drugs have relatively higher costs than other drugs because manufacturers must rely on smaller patient populations to recoup development investments. In the United States, 15 orphan drugs were commercialized between 2006 and 2008; 6 of these cost more than $100,000 per patient per year.

	Previous studies have suggested that orphan drugs have less robust bodies of evidence because smaller patient populations and limited knowledge of rare conditions constrain the design, conduct, analysis, and interpretation of clinical trials.

 Due to the rapid rate of oncology orphan drug development and the significant financial burden associated with cancer treatments, there have been increasing pressures on drug manufacturers to demonstrate the value of their products. Less robust bodies of evidence will present particular challenges to CER and will make decision making about orphan drug accessibility difficult. The comparative effectiveness data supporting oncology orphan drugs marketed in the United States have not been well studied. The primary objective of this study was to systematically and critically assess the level and quality of clinical and economic evidence currently available for all oncology orphan drugs marketed in the United States. A secondary objective was to highlight the challenges and opportunities for evidence development within this pharmaceutical category.

	The Cumulative List of Designated Approved Orphan Products (www.fda.gov/orphan/designate/allap.rtf) describes drug products that have received orphan designation and have ever received marketing approval from the FDA.

	We used this list, updated May 5, 2009, by the FDA, to identify products indicated to treat rare cancers. Orphan products indicated for diagnosis, palliative care, or treatment of secondary conditions associated with cancer, such as neutropenia, were not included in this study. Any product withdrawn or discontinued from the US market as of June 2010 was also excluded.

	For each oncology orphan drug included in this study, we conducted a literature search in the Medline and Embase databases for randomized controlled trials (RCTs) and cost-effectiveness analyses published prior to June 2010, using search terms that included the drug’s US brand name, generic name, disease indication, and the terms “randomized,” “efficacy,” “cost-effectiveness,” and “economic.” Our priority was to identify RCTs, but we also used the literature search to identify observational studies (prospective or retrospective) or other studies that described the treatment effect of the drug. We also identified published articles through information provided on the FDA Web site for new drug approvals and manual searches of article references. We defined cost-effectiveness analysis using the definition established by the Panel on Cost-Effectiveness in Health and Medicine as “An analytic tool in which costs and effects of a program and at least one alternative are calculated and presented in a ratio of incremental cost to incremental effect.”

	For each literature search, 1 author (MMC) reviewed all of the titles and abstracts of studies likely to meet the following inclusion criteria: (1) original RCT or cost-effectiveness analysis evaluating the orphan drug used for its approved orphan indication, or observational study or published article describing the treatment effect of the orphan drug used for its approved orphan indication; (2) comparative RCTs in which the orphan drug is compared with placebo or a clinically relevant comparator; (3) if the orphan drug (eg, drug A) is used as part of a combination therapy regimen, the trial is designed so that the clinical effect of the orphan drug can be isolated and directly assessed (eg, drugs A, B, and C vs drugs B and C); (4) article is published in the English language; and (5) entire article is available for review. 

 presents a flow diagram that describes the literature search methods and the restrictions applied to our search.

	For each clinical study that met the inclusion criteria, we abstracted information about the comparator, patient characteristics, study design and treatment allocation, primary outcome measure, statistical analytic method, reporting of treatment effect with uncertainty, and study sponsor. For each cost-effectiveness analysis that met the inclusion criteria, we abstracted information about the comparator, study perspective, methods, data sources, primary outcome measure, base-case and sensitivity analysis results, and study sponsor (if reported).

	We used the Grading of Recommendations Assessment, Development and Evaluation (GRADE) method, an evaluation system created by a diverse group of international guideline developers, to assess the quality of clinical bodies of evidence. The GRADE system assesses the quality of a body of evidence by focusing on 4 main components: study design, quality, consistency of evidence, and directness of comparator, population, and intervention. The method also evaluates limitations, potential biases, and uncertainty to assign 1 of 4 possible grades: high, moderate, low, and very low.

 We adapted the grading system to our study by including a fifth grade category, “not able to assess,” to describe circumstances where we could not identify published studies that met our inclusion criteria. 

 details the GRADE methodology and how it was implemented in this study, and defines each quality grade.

	We used the Quality of Health Economic Studies (QHES) grading criteria, a validated quantitative instrument developed by health economists, to assess the quality of cost-effectiveness studies. The QHES grading system consists of 16 criteria that are described in 

 Each criterion is associated with a weighted score that was\ assigned in its entirety for each criterion perceived by the lead author to be satisfactory. Zero points were assigned to each criterion that was perceived to be unsatisfactory. Certain criteria, such as subgroup analysis, were not applicable for all studies. In these circumstances, we used both a best-case and worst-case scoring method where full points and zero points were assigned to each inapplicable criterion. The weighted scores were summed across all criteria for a total of 100 possible points for each study.

	We graphically explored whether disease prevalence was associated with the level and quality of evidence for oncology orphan drugs and calculated the Spearman rank correlation coefficient to estimate the strength of the relationship. We obtained estimates of disease prevalence from Orphanet, an electronic reference for rare disease information in Europe (www.orpha.net).

 This resource was the only one we identified that provided a collection of prevalence estimates for rare diseases. Although the data were derived in Europe, we believe the estimates are of similar magnitude in the United States, assuming similar risk factors for rare cancers. The assessment of correlation excluded 5 drugs: aldesleukin for metastatic melanoma, valrubicin for urinary bladder carcinoma, doxorubicin liposome for metastatic ovarian cancer, and toremifene citrate and exemestane for breast cancer. These cancers are not considered rare. The drugs are indicated for smaller subpopulations of patients with specific disease, physical, or genetic characteristics. Prevalence estimates for patients with these specific characteristics were not reported. We also explored whether the level and quality of evidence for oncology orphan drugs is associated with receipt of FDA accelerated approval.

	We initially identified 48 oncology orphan drugs for inclusion in this study from the Cumulative List of Designated Approved Orphan Products. Of these drugs, 3 (Idamycin, Vesanoid, and Mylotarg) have been discontinued or withdrawn from the US market, and 5 drugs (Gleevec, Sprycel, Tasigna, Velcade, and Temodar) were indicated to treat more than 1 orphan indication or patient population; each indication was independently evaluated for these drugs. We did not include expanded or supplemental approvals for nonorphan indications. In total, 47 orphan drugs and indications were included in this study and are listed in 

 by their brand and generic names. The majority of oncology orphan drugs are indicated for rare blood cancers (n = 30). Table 3 also describes the date of full and/or accelerated regulatory approval, and lists the number of RCTs identified in the published literature that met the inclusion criteria with citations of all published articles that were used to inform the body of evidence.

	We identified a total of 60 RCTs that met our inclusion criteria. The greatest number of RCTs were conducted for interferon alpha-2a indicated for chronic myelogenous leukemia (CML) (n = 7), aldesleukin indicated for metastatic renal cell carcinoma (n = 7), and rituximab indicated for non-Hodgkin B-cell lymphoma (n = 6). The majority of trials we identified comprised the evidence base used for regulatory approval.

 summarizes the clinical evidence grades assigned in this study. Twelve drugs had moderate-quality evidence, including cladribine for hairy cell leukemia and arsenic trioxide for acute promyelocytic leukemia. Although we did not identify any RCTs for these drugs that met our inclusion criteria, their clinical bodies of evidence consisted of observational studies that consistently reported similar treatment effects or a dose-response association with little threat to validity. Individual GRADE ratings assigned to each drug and their clinical bodies of evidence are listed in Table 3.

 shows that there are drugs indicated for cancers with lower prevalence that had high-quality evidence and drugs indicated for cancers with higher prevalence that had low-quality evidence or lacked a sufficient body of evidence for review. The Spearman rank correlation coefficient between disease prevalence and the level of evidence for oncology orphan drugs was 0.3 (95% confidence interval [CI], 0.0-0.5). This indicates that there may be a weak correlation between the prevalence of rare cancers and the number of RCTs conducted for drugs indicated to treat rare cancers. A total of 14 drugs in our study received accelerated approval for the orphan indication. Of these, zero had high-quality bodies of evidence and 5 had moderate-quality bodies of evidence. Although our sample size was small, these results provided some indication that the accelerated approval provision does not optimize high-quality evidence generation for oncology orphan drugs.

	We identified 21 cost-effectiveness studies that met our inclusion criteria. Of those studies, 10 evaluated the use of either interferon alpha or imatinib mesylate, or compared both for CML. 

 summarizes the cost-effectiveness studies included in this assessment and describes comparators, study perspectives, base-case incremental cost-effectiveness ratios, and results of sensitivity analyses.

	The cost-effectiveness analyses received QHES scores ranging from 72 to 100, with a mean score of 85. The clinical data used to inform the studies were largely obtained from either randomized trials or patient medical records. The majority of studies implemented modeling techniques to project longer-term health outcomes that could not be obtained from limited clinical data. Cost inputs were obtained from administrative databases and the published literature. The most common shortcomings were as follows: 11 studies used data inputs from sources that might potentially incorporate bias and did not provide explicit discussion about the direction and magnitude of potential biases; 6 studies did not specify the source of funding for the study, which is recommended to add transparency; and 5 studies either failed to specify the time horizon for analysis, conducted analysis using a short time horizon that is not expected to fully capture clinically significant costs and effects, or did not implement adequate discounting for both future costs and outcomes. Table 4 reports the quality scores assigned to each cost-effectiveness study using the best-case scoring method where full points were assigned to inapplicable criteria. Scores assigned using the worst-case approach, where zero points were assigned to inapplicable criteria, are reported in parentheses.

	We conducted a systematic literature review to identify and assess clinical and economic evidence for 47 oncology orphan drugs marketed in the United States and applied 2 independent grading frameworks to the selected studies to critically assess the quality of each body of evidence. The supporting bodies of evidence available for marketed oncology orphan drugs vary in quality, with limited evidence demonstrating their economic value.

	Although the GRADE methodology is relatively explicit, reviewer judgment is required. In particular, we found that it was necessary to grade certain criteria subjectively in order to better accommodate oncology orphan drug characteristics. For example, the majority of RCTs for oncology orphan drugs had relatively small sample sizes that could lead to substantial imprecision (eg, lack of power, wide confidence intervals). We assigned lower ratings for studies that did not meet predetermined enrollment criteria or appeared underpowered to detect differences in their primary end point, but we did not consider a smaller sample size in itself to be a limitation. We critically assessed each study and lowered ratings when we identified instances of bias that were not explicitly addressed. We also lowered ratings when studies were poorly reported, or when bodies of evidence consisted of studies that reported inconsistent treatment effects.

	The number of RCTs identified in our study is comparable to the number in a study conducted by Tsimberidou and colleagues that evaluated the long-term marketing outcome of oncology drugs approved by the FDA without a randomized trial.

 In their study, 68 approved oncology drugs were identified and almost half (n = 31) were approved without an RCT. Kesselheim and colleagues recently evaluated pivotal trial characteristics of orphan and nonorphan drugs approved between 2004 and 2010 to treat cancer and concluded that pivotal trials for approved oncology orphan drugs were more likely to be smaller, nonrandomized, and unblinded, and to use surrogate end points to assess efficacy.

 The study is limited in that it only evaluated pivotal trials and did not account for other available clinical information. Also, the study was restricted to recently approved oncology orphan drugs and did not review the entire pharmaceutical category. We observed the same characteristics in the clinical trials that were included in our study. We also observed that there is a lack of published post-marketing studies and information about longer-term safety and efficacy.

	We hypothesized that the level of evidence available for oncology orphan drugs would have a strong correlation with disease prevalence in that larger patient populations enable better evidence generation. However, our results suggested a potentially weak correlation between the prevalence of rare cancers and the level of evidence available for drugs indicated to treat rare cancers. A weak relationship suggests that evidence development for oncology orphan drugs may not depend as much on the size of a particular patient population. Instead, evidence development in this pharmaceutical category may depend more on other factors, possibly regulatory requirements or reimbursement considerations.

	The overall dearth of cost-effectiveness studies in this pharmaceutical category, and in oncology as a whole, may reflect evidence limitations or publication bias, where studies of drugs with higher costs, greater benefit-risk uncertainty, or lower effectiveness are not published. In assessing the economic challenges of orphan drugs, Drummond and colleagues stated: “In short, if standard health technology assessment (HTA) procedures were to be applied to orphan drugs, virtually none of them would be ‘cost-effective’.”

 This conclusion was largely based on 2 factors: (1) high incremental cost per quality-adjusted life-year and (2) insufficient breadth and quality of clinical evidence for orphan drugs compared with drugs for more common diseases.

 Our results demonstrated that, contrary to these prior suggestions, it is feasible for some oncology orphan drugs to be considered cost-effective in specific healthcare settings using standard methods of health technology assessment. The clinical and economic value of each orphan drug should be assessed individually, on a case-by-case basis. We observed that all of the drugs considered cost-effective in their respective studies had moderate-quality to high-quality bodies of clinical evidence. This finding suggests there may be a relationship between the level and quality of clinical evidence and the likelihood that an oncology orphan drug has published estimates of cost-effectiveness.

	Evidence development is costly and challenging for all healthcare interventions, but particularly for orphan drugs, due to smaller patient populations and limited clinical knowledge of rare conditions. The trade-offs to generating more robust bodies of evidence may include delayed product accessibility, higher costs, or reduced availability of therapies. For conditions that have therapeutic alternatives, these trade-offs may be acceptable. However, for life-threatening conditions with limited therapy options, these trade-offs may not be considered acceptable. For rare diseases with only a single treatment option, one may question whether economic analysis should apply. Cost-effectiveness analysis is useful in that it provides information about the value of a health intervention compared with an alternative, which may be best supportive care or no therapy. From an equity perspective, McCabe and colleagues argue that there is no sustainable reason why the cost-effectiveness of orphan drugs should be evaluated differently from other drugs.

 However, healthcare systems may wish to consider additional factors when making decisions about orphan drugs, such as budget impact, disease severity, availability of alternative therapies, or societal preferences toward patients with rare diseases.

	Our study highlights 3 important policy questions. First, what types of study designs, incentives, or methods can be used to encourage better evidence development for oncology orphan drugs? Second, how much evidence is necessary or considered sufficient to healthcare decision makers, and what types of evidence should be generated prior to and after marketing approval? Finally, what are the process and decision-making criteria for evaluating comparative effectiveness data for oncology orphan drugs? Emerging private and public initiatives are attempting to address these questions to some extent with evolving methods such as coverage with evidence development

 but additional innovative analytic methods and policy approaches are necessary.

	We recognize several limitations to this study. The most important limitation is that the GRADE assessment framework provides information about the amount of confidence that can be associated with a body of evidence and does not relay any information about the magnitude of clinical benefit or safety of a drug. Certain oncology orphan drugs with low-grade published evidence may yield significant clinical benefits, and conversely, certain drugs with high-grade evidence may yield very little benefit. Currently, there is no established framework for quantifying the magnitude of benefits or risks from health interventions. Garrison and colleagues suggest pairing CER and benefit-risk analysis into 1 framework and describe how cost-effectiveness analysis models could be adapted to conduct quantitative benefit-risk assessments.

 It is important for manufacturers to engage in continuous evidence generation, even after a product is commercialized, so that the effectiveness and safety of their products can be accurately assessed. Comparative effectiveness research methods that capture and incorporate nonpublished clinician or patient experiences could also be useful to help better identify the real-world value of marketed oncology orphan drugs.

	Given the large number of drugs included in this review and the numerous keywords we could have used to search for potential clinical and economic studies, this review may not have captured all relevant drugs or studies. We used broad search terms in 2 major databases and also conducted manual searches of reference citations; this methodology allowed us to screen as many studies as possible. Only 1 author (MMC) reviewed potential studies, abstracted information, critically assessed articles, and assigned quality ratings. Although attempts were made to be accurate and consistent, it is possible that unintentional errors and inconsistencies could have reduced or improved the quality rating for a body of evidence. The QHES is limited in its ability to identify poorly analyzed studies and does not have a benchmark for total scores, which limits its ability to quantitatively categorize and distinguish high-quality studies from low-quality studies.

	The results of our study show that oncology orphan drugs marketed in the United States have varying levels and quality of clinical evidence and a shortage of evidence demonstrating economic value. It is uncertain whether the current evidence levels for oncology orphan drugs marketed in the United States are sufficient to support decision-making practices consistent with principles of evidence-based medicine and CER. This issue remains an open policy question that requires additional evaluation.

Articles

This study critically assessed the published clinical and economic evidence supporting oncology orphan drugs marketed in the United States.

Louis P. Garrison, PhD

Articles

Systematic Review of Comparative Effectiveness Data for Oncology Orphan Drugs