David L. Veenstra, PharmD, PhD

Continuing advances in genetic sequencing and large, ongoing studies have led to calls for periodic reinterpretation of patients’ sequencing results.

The development of transparent and systematic policies for coverage and reimbursement of genetic reinterpretation will be challenging.

We propose a framework consisting of 3 components: (1) types of genetic reinterpretation upon which to focus, (2) questions that should be asked of those offering reinterpretation, and (3) a process for requesting such information.

This framework will facilitate systematic and transparent policies for genetic reinterpretation in an efficient manner.

The advent of next-generation sequencing has enabled the generation and analysis of vast amounts of genomic data. Although technology has increased the diagnostic yield and efficiency of making genetic diagnoses, we also are presented with the challenge of not knowing what the clinical implications are for a significant portion of identified genetic variants, referred to as variants of uncertain significance (VUSs).

The good news is that with ongoing, very large genomic data sets and studies, every year we gain a better understanding of genetic variants. As a result, calls periodically arise for reinterpretation of available data to continuously identify new disease genes and disease-associated variants and more accurately assess genetic contributions to medical conditions.

 This process—reevaluating the clinical implications of previously identified genetic variants—is referred to as genetic reinterpretation.

Genetic reinterpretation is important for US payers because medical care induced by reinterpretation could be costly, but, at the same time, reinterpretation can lead to improvements in patient outcomes and progress in equity across diverse patient populations.

 However, the application of evidence-based processes for developing coverage and reimbursement policies for genetic reinterpretation will be challenging for several reasons. First, the determination of clinical relevance of a variant is based on multiple types of data and generally emerges over time based on scientific and clinical consensus. Second, the timing of reinterpretation will be highly dependent on the availability of new data, which will vary by clinical indication and patient population. Third, many payers likely do not have the expertise or resources needed to conduct such evaluations.

We propose a brief framework to facilitate the development of sound payer policies for genetic reinterpretation that identifies key issues and relies on organizations offering genetic reinterpretation services to provide evidence and rationale to support coverage of their services. The framework consists of 3 components: (1) types of genetic reinterpretation upon which to focus, (2) questions that should be asked of those offering reinterpretation, and (3) a process for requesting such information.

Component 1. What to Focus on: Reclassification of VUSs to Pathogenic or Benign

 shows the conceptual process of variant reinterpretation. Results from genetic tests fall into 3 major categories: (1) pathogenic/likely pathogenic (“pathogenic”); (2) benign/likely benign (“benign”); and (3) VUSs. By far, most variants are benign. VUSs, however, are not uncommon. The assessment of VUS arises when family, epidemiological, or functional studies provide insufficient evidence to determine if the variant is associated with disease risk.

Although there are many possible types of variant reclassifications, the most likely result of accumulating evidence is the most obvious: making uncertain variants more certain—that is, reclassifying them as benign or pathogenic. Although the specific number varies based on date of the study and type of testing, the vast majority of VUSs will be reclassified as benign but perhaps 15% of VUSs will be reclassified as pathogenic.

 From a pragmatic perspective, it thus makes the most sense to focus on the impact of the reinterpretation process on these 2 types of reinterpretation: VUS to benign and VUS to pathogenic.

Component 2. What Are the Key Data Elements? Five Questions to Ask

Next, we rely on a decision-analytic approach to conceptualize and identify key data and outcomes given the lack of direct evidence from sources such as randomized controlled trials.

 The intervention in this case—reinterpretation—would be compared with a scenario in which reinterpretation is not conducted. We are thus interested in incremental outcomes. Baseline risk and treatment effect size are the metrics we typically consider when evaluating health care interventions. In this case, baseline risk is analogous to the prevalence of pathogenic variants, and the effect size is driven by frequency of reclassification as well as the downstream clinical and economic impacts. The cost of the intervention is of course important. Lastly, uncertainty and heterogeneity of effects across patient populations must be considered.

Question 1. How often would sufficient evidence accumulate to reclassify VUSs across clinical applications?

 An estimate of the frequency of reclassifying VUSs is essential, as it will be one of the primary drivers of clinical and economic value. Given the diversity of genomic findings, it may make the most sense to focus on high-volume tests that are medically actionable, such as testing for hereditary cancer syndromes and cardiac disease.

 Conditions for which a molecular/targeted treatment is available, effective, and likely expensive also merit scrutiny.

Drivers for reclassification will be large releases of new reference data, new functional experiments, new publications of disease genes, and improvements in prediction algorithms.

Question 2. What are the clinical consequences of variant reclassification?

 Recommendations for clinicians encourage them not to make medical decisions based on a VUS because the clinical significance is uncertain. Yet clinicians may be tempted to act, both from a desire to help their patients and potentially because of liability concerns.

 VUSs also may lead to worry for patients.

Reclassification of a VUS to pathogenic can trigger medical decisions based on that genetic diagnosis, initiate cascade screening in family members, and influence family planning decisions. Medical actions should be supported by guidelines to increase clinical utility and likelihood of reimbursement.

Reclassification of a VUS to benign should not trigger additional medical decisions but will resolve the uncertain status of the VUS that was identified, potentially alleviating patient worry. Furthermore, any unnecessary actions that were undertaken that could have associated clinical risk (eg, more frequent colonoscopies) could be stopped.

Question 3. What are the economic consequences of variant reclassification?

 Reclassification of VUSs could lead to either an increase or a decrease in health care spending. Because reclassification to benign will be most common, the potential cost savings from avoiding nonindicated health care, including ongoing monitoring, in these patients may offset the cost of indicated health care in cases of reclassification to pathogenic.

Importantly, the costs saved by reducing downstream adverse health outcomes such as the development or progression of disease should be considered, as well as impacts on accurate risk stratification of family members with appropriate use of cascade genetic testing once a pathogenic variant is identified in the family.

Question 4. How much does reinterpretation cost, and what is the model for the service?

 The steps to providing reinterpretation include (1) building a genomic data warehouse, (2) adopting a pipeline for automated variant reassessment, (3) human review of any flagged variant, and (4) issuing a revised report. This process will have a large, fixed cost initially to build the infrastructure, with a small marginal cost for future reinterpretations. For example, reanalysis can be triggered by a laboratory seeing the same variant in another patient and classifying it differently. In this case, the primary cost is to recontact the referring provider.

Reimbursement for chronic disease management may provide a useful model for the structure of a reimbursement contract. A payer could go to multiple laboratories and contract for a certain number of reinterpretations. Another consideration is that with dropping sequencing costs, reinterpretation over a limited amount of time could be included in sequencing cost or provided as an ongoing subscription service.

Question 5. How heterogeneous is the yield of variant reinterpretation across diverse clinical populations?

 Ancestrally underrepresented groups are less likely to benefit from current genetic testing because a large amount of genetic reference data involves individuals of European ancestry, and these findings are not generalizable to minority groups.

 Thus, minority populations are more likely to receive VUS results. However, as reference data sets improve, minority populations are the most likely to benefit immediately from reinterpretation because common variants in these ancestral groups can quickly be discarded as causes of rare diseases as they will be present in large numbers of individuals.

Component 3. Payers Request Evidence and Rationale From Reinterpretation Providers

Finally, we propose that rather than undertaking the task of assessing genomic reinterpretation on their own, payers place the onus of providing information on those seeking reimbursement for such services, namely laboratories that perform genetic testing. This approach follows the conceptual process and format established by the Academy of Managed Care Pharmacy for the submission of evidence from drug and biologic manufacturers to support formulary listing.

 Although many differences exist between assessment of pharmaceuticals and variant reinterpretation, the concept is similar: Provide guidance for important evidence that improves communication between laboratories and payers and enables payers to make structured, informed decisions.

There are unresolved issues related to genetic reinterpretation, such as who should decide if and when reinterpretation is needed, who should perform the reinterpretation, and who should be informed about the results. It is not yet clear what the responsibilities and roles of laboratories, geneticists, clinicians, patients, payers, and policy makers are in handling these issues.

Health care payers can do their part to help address these challenges by using a systematic and transparent approach to coverage of genetic reinterpretation. Relying on the concepts outlined here and evidence and rationale from genetic sequencing providers will facilitate this goal in an efficient manner.


 The Comparative Health Outcomes, Policy & Economics (CHOICE) Institute, School of Pharmacy, University of Washington (DLV), Seattle, WA; Department of Health Policy and Management, Mailman School of Public Health, Columbia University (JWR), New York, NY; Department of Public Health Policy and Management, School of Global Public Health, New York University (JAP, AG), New York, NY; School of Public Health in Austin, The University of Texas Health Science Center at Houston (HSB), Austin, TX; Avalon Health Economics LLC (JES), Morristown, NJ; Division of Clinical Genetics, NewYork-Presbyterian Morgan Stanley Children’s Hospital, Columbia University Medical Center (SMB), New York, NY; Kennedy Family Professor of Pediatrics in Medicine, Division of Clinical Genetics, Department of Pediatrics, Columbia University (WKC), New York, NY; Department of Psychiatry, Columbia University Vagelos College of Physicians and Surgeons, NY State Psychiatric Institute (PSA), New York, NY.

 This work was supported by a grant (R01HG010365) from the National Human Genome Research Institute (NHGRI). Drs Appelbaum and Chung also received support from NHGRI grants RM1HG007257 and U01HG008680. Dr Veenstra also received support from NHGRI grant R01HG009694.

 Dr Veenstra has consulted for Foundation Medicine and received grants from Illumina. Dr Chung has consulted for Regeneron Genetics Center SAB. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (DLV, JWR, JAP, HSB, JES, AG, SMB, WKC, PSA); analysis and interpretation of data (AG, SMB); drafting of the manuscript (DLV, JWR, JAP, HSB, JES, SMB, WKC); critical revision of the manuscript for important intellectual content (DLV, JWR, HSB, JES, AG, SMB, WKC, PSA); obtaining funding (PSA); and administrative, technical, or logistic support (JAP, AG).

 David L. Veenstra, PharmD, PhD, Comparative Health Outcomes, Policy & Economics (CHOICE) Institute, School of Pharmacy, University of Washington, Box 357630, H375 Health Science Bldg, Seattle, WA 98195-7630. Email: veenstra@uw.edu.

1. El Mecky J, Johansson L, Plantinga M, et al. Reinterpretation, reclassification, and its downstream effects: challenges for clinical laboratory geneticists. 

. 2019;12(1):170. doi:10.1186/s12920-019-0612-6

2. Deignan JL, Chung WK, Kearney HM, Monaghan KG, Rehder CW, Chao EC; ACMG Laboratory Quality Assurance Committee. Points to consider in the reevaluation and reanalysis of genomic test results: a statement of the American College of Medical Genetics and Genomics (ACMG). 

. 2019;21(6):1267-1270. doi:10.1038/s41436-019-0478-1

3. Harrison SM, Rehm HL. Is ‘likely pathogenic’ really 90% likely? reclassification data in ClinVar. 

. 2019;11(1):72. doi:10.1186/s13073-019-0688-9

4. Macklin S, Durand N, Atwal P, Hines S. Observed frequency and challenges of variant reclassification in a hereditary cancer clinic. 

. 2018;20(3):346-350. doi:10.1038/gim.2017.207

5. Mersch J, Brown N, Pirzadeh-Miller S, et al. Prevalence of variant reclassification following hereditary cancer genetic testing. 

. 2018;320(12):1266-1274. doi:10.1001/jama.2018.13152

6. Xiang J, Yang J, Chen L, et al. Reinterpretation of common pathogenic variants in ClinVar revealed a high proportion of downgrades. 

. 2020;10(1):331. doi:10.1038/s41598-019-57335-5

7. Veenstra DL, Piper M, Haddow JE, et al. Improving the efficiency and relevance of evidence-based recommendations in the era of whole-genome sequencing: an EGAPP methods update. 

. 2013;15(1):14-24. doi:10.1038/gim.2012.106

8. Turner SA, Rao SK, Morgan RH, Vnencak-Jones CL, Wiesner GL. The impact of variant classification on the clinical management of hereditary cancer syndromes. 

. 2019;21(2):426-430. doi:10.1038/s41436-018-0063-z

9. Kurian AW, Li Y, Hamilton AS, et al. Gaps in incorporating germline genetic testing into treatment decision-making for early-stage breast cancer. 

. 2017;35(20):2232-2239. doi:10.1200/JCO.2016.71.6480

10. Makhnoon S, Shirts BH, Bowen DJ. Patients’ perspectives of variants of uncertain significance and strategies for uncertainty management. 

. 2019;28(2):313-325. doi:10.1002/jgc4.1075

11. Berg JS, Foreman AKM, O’Daniel JM, et al. A semiquantitative metric for evaluating clinical actionability of incidental or secondary findings from genome-scale sequencing. 

. 2016;18(5):467-475. doi:10.1038/gim.2015.104

12. Invitae introduces routine exome reanalysis to help patients receive diagnoses faster. News release. Invitae; December 17, 2020. Accessed December 18, 2020. https://ir.invitae.com/news-and-events/press-releases/press-release-details/2020/Invitae-Introduces-Routine-Exome-Reanalysis-to-Help-Patients-Receive-Diagnoses-Faster/default.aspx

13. Ben-Eghan C, Sun R, Hleap JS, et al. Don’t ignore genetic data from minority populations. 

. 2020;585(7824):184-186. doi:10.1038/d41586-020-02547-3

14. Oni-Orisan A, Mavura Y, Banda Y, Thornton TA, Sebro R. Embracing genetic diversity to improve Black health. 

. 2021;384(12):1163-1167. doi:10.1056/NEJMms2031080

15. Smith CE, Fullerton SM, Dookeran KA, et al. Using genetic technologies to reduce, rather than widen, health disparities. 

. 2016;35(8):1367-1373. doi:10.1377/hlthaff.2015.1476

16. West KM, Blacksher E, Burke W. Genomics, health disparities, and missed opportunities for the nation’s research agenda. 

. 2017;317(18):1831-1832. doi:10.1001/jama.2017.3096

17. Pannier A, Dunn JD. AMCP format for formulary submissions, version 4.0. 

. 2016;22(5):448. doi:10.18553/jmcp.2016.16092

Articles

Periodic reinterpretation of genetic sequencing results presents a challenge for developing transparent and systematic coverage and reimbursement policies.

Reimbursement for Genetic Variant Reinterpretation: Five Questions Payers Should Ask

 To systematically assess clinical and economic evidence for oncology orphan drugs marketed in the United States and to highlight the challenges and opportunities for evidence development within this pharmaceutical category.

 We conducted systematic literature searches of the Medline and Embase databases for clinical and cost-effectiveness studies published before June 2010 for all oncology orphan drugs marketed in the United States. We used the Grading of Recommendations Assessment, Development and Evaluation method and the Quality of Health Economic Studies criteria to assess the quality of the selected studies.

We identified 60 randomized controlled trials and 21 cost-effectiveness analyses to support 47 oncology orphan drugs. A total of 21 drugs had moderate or high-quality bodies of clinical evidence, 11 had low-quality or very low quality clinical evidence, and 15 drugs could not be evaluated because we were unable to identify clinical evidence that met our inclusion criteria. The Spearman rank correlation coefficient for the level of evidence for oncology orphan drugs and disease prevalence was 0.3 (95% confidence interval, 0.0-0.5). The cost-effectiveness analyses received quality scores between 72 and 100 (range 0-100), with a mean score of 85.

The results of our study show that oncology orphan drugs marketed in the United States have varying levels and quality of clinical evidence and a paucity of evidence regarding economic value. Innovative analytic and policy approaches are needed to develop and implement a decision-making framework for this pharmaceutical category that is consistent with evidence-based medicine and comparative effectiveness research.

	We conducted a systematic literature review to critically assess the clinical and economic evidence supporting oncology orphan drugs marketed in the United States.

		Oncology orphan drugs marketed in the United States have varying levels and quality of clinical evidence, and a paucity of evidence demonstrating their economic value.

		The current levels of clinical and economic evidence present challenges for decision making about oncology orphan drug availability and accessibility.

		Innovative analytic and policy approaches are necessary to develop and implement a decision-making framework for oncology orphan drugs that is consistent with evidence-based medicine and comparative effectiveness research.

	Comparative effectiveness research (CER) can be described as the assessment of a medical intervention against alternative interventions with the intent of identifying treatment strategies that are likely to have preferable benefit-risk profiles or are considered cost-effective in real-world clinical settings. The purpose of CER is to assist healthcare providers, payers, patients, and decision makers in making informed healthcare decisions that will improve individual and population health.

	Definitions of a rare disease differ around the world and the prevalence threshold varies between countries.

 In the European Union, a disease is considered rare if it affects fewer than 215,000 individuals, while in the United States, a rare disease is described as a condition with prevalence of less than 200,000 or a disease with distinct subpopulations consisting of fewer than 200,000 individuals nationwide.

 As of 2010, 362 drugs indicated for rare diseases (orphan drugs) have received US Food and Drug Administration (FDA) market approval, and oncology therapies comprise the largest clinical subcategory.

 In general, orphan drugs have relatively higher costs than other drugs because manufacturers must rely on smaller patient populations to recoup development investments. In the United States, 15 orphan drugs were commercialized between 2006 and 2008; 6 of these cost more than $100,000 per patient per year.

	Previous studies have suggested that orphan drugs have less robust bodies of evidence because smaller patient populations and limited knowledge of rare conditions constrain the design, conduct, analysis, and interpretation of clinical trials.

 Due to the rapid rate of oncology orphan drug development and the significant financial burden associated with cancer treatments, there have been increasing pressures on drug manufacturers to demonstrate the value of their products. Less robust bodies of evidence will present particular challenges to CER and will make decision making about orphan drug accessibility difficult. The comparative effectiveness data supporting oncology orphan drugs marketed in the United States have not been well studied. The primary objective of this study was to systematically and critically assess the level and quality of clinical and economic evidence currently available for all oncology orphan drugs marketed in the United States. A secondary objective was to highlight the challenges and opportunities for evidence development within this pharmaceutical category.

	The Cumulative List of Designated Approved Orphan Products (www.fda.gov/orphan/designate/allap.rtf) describes drug products that have received orphan designation and have ever received marketing approval from the FDA.

	We used this list, updated May 5, 2009, by the FDA, to identify products indicated to treat rare cancers. Orphan products indicated for diagnosis, palliative care, or treatment of secondary conditions associated with cancer, such as neutropenia, were not included in this study. Any product withdrawn or discontinued from the US market as of June 2010 was also excluded.

	For each oncology orphan drug included in this study, we conducted a literature search in the Medline and Embase databases for randomized controlled trials (RCTs) and cost-effectiveness analyses published prior to June 2010, using search terms that included the drug’s US brand name, generic name, disease indication, and the terms “randomized,” “efficacy,” “cost-effectiveness,” and “economic.” Our priority was to identify RCTs, but we also used the literature search to identify observational studies (prospective or retrospective) or other studies that described the treatment effect of the drug. We also identified published articles through information provided on the FDA Web site for new drug approvals and manual searches of article references. We defined cost-effectiveness analysis using the definition established by the Panel on Cost-Effectiveness in Health and Medicine as “An analytic tool in which costs and effects of a program and at least one alternative are calculated and presented in a ratio of incremental cost to incremental effect.”

	For each literature search, 1 author (MMC) reviewed all of the titles and abstracts of studies likely to meet the following inclusion criteria: (1) original RCT or cost-effectiveness analysis evaluating the orphan drug used for its approved orphan indication, or observational study or published article describing the treatment effect of the orphan drug used for its approved orphan indication; (2) comparative RCTs in which the orphan drug is compared with placebo or a clinically relevant comparator; (3) if the orphan drug (eg, drug A) is used as part of a combination therapy regimen, the trial is designed so that the clinical effect of the orphan drug can be isolated and directly assessed (eg, drugs A, B, and C vs drugs B and C); (4) article is published in the English language; and (5) entire article is available for review. 

 presents a flow diagram that describes the literature search methods and the restrictions applied to our search.

	For each clinical study that met the inclusion criteria, we abstracted information about the comparator, patient characteristics, study design and treatment allocation, primary outcome measure, statistical analytic method, reporting of treatment effect with uncertainty, and study sponsor. For each cost-effectiveness analysis that met the inclusion criteria, we abstracted information about the comparator, study perspective, methods, data sources, primary outcome measure, base-case and sensitivity analysis results, and study sponsor (if reported).

	We used the Grading of Recommendations Assessment, Development and Evaluation (GRADE) method, an evaluation system created by a diverse group of international guideline developers, to assess the quality of clinical bodies of evidence. The GRADE system assesses the quality of a body of evidence by focusing on 4 main components: study design, quality, consistency of evidence, and directness of comparator, population, and intervention. The method also evaluates limitations, potential biases, and uncertainty to assign 1 of 4 possible grades: high, moderate, low, and very low.

 We adapted the grading system to our study by including a fifth grade category, “not able to assess,” to describe circumstances where we could not identify published studies that met our inclusion criteria. 

 details the GRADE methodology and how it was implemented in this study, and defines each quality grade.

	We used the Quality of Health Economic Studies (QHES) grading criteria, a validated quantitative instrument developed by health economists, to assess the quality of cost-effectiveness studies. The QHES grading system consists of 16 criteria that are described in 

 Each criterion is associated with a weighted score that was\ assigned in its entirety for each criterion perceived by the lead author to be satisfactory. Zero points were assigned to each criterion that was perceived to be unsatisfactory. Certain criteria, such as subgroup analysis, were not applicable for all studies. In these circumstances, we used both a best-case and worst-case scoring method where full points and zero points were assigned to each inapplicable criterion. The weighted scores were summed across all criteria for a total of 100 possible points for each study.

	We graphically explored whether disease prevalence was associated with the level and quality of evidence for oncology orphan drugs and calculated the Spearman rank correlation coefficient to estimate the strength of the relationship. We obtained estimates of disease prevalence from Orphanet, an electronic reference for rare disease information in Europe (www.orpha.net).

 This resource was the only one we identified that provided a collection of prevalence estimates for rare diseases. Although the data were derived in Europe, we believe the estimates are of similar magnitude in the United States, assuming similar risk factors for rare cancers. The assessment of correlation excluded 5 drugs: aldesleukin for metastatic melanoma, valrubicin for urinary bladder carcinoma, doxorubicin liposome for metastatic ovarian cancer, and toremifene citrate and exemestane for breast cancer. These cancers are not considered rare. The drugs are indicated for smaller subpopulations of patients with specific disease, physical, or genetic characteristics. Prevalence estimates for patients with these specific characteristics were not reported. We also explored whether the level and quality of evidence for oncology orphan drugs is associated with receipt of FDA accelerated approval.

	We initially identified 48 oncology orphan drugs for inclusion in this study from the Cumulative List of Designated Approved Orphan Products. Of these drugs, 3 (Idamycin, Vesanoid, and Mylotarg) have been discontinued or withdrawn from the US market, and 5 drugs (Gleevec, Sprycel, Tasigna, Velcade, and Temodar) were indicated to treat more than 1 orphan indication or patient population; each indication was independently evaluated for these drugs. We did not include expanded or supplemental approvals for nonorphan indications. In total, 47 orphan drugs and indications were included in this study and are listed in 

 by their brand and generic names. The majority of oncology orphan drugs are indicated for rare blood cancers (n = 30). Table 3 also describes the date of full and/or accelerated regulatory approval, and lists the number of RCTs identified in the published literature that met the inclusion criteria with citations of all published articles that were used to inform the body of evidence.

	We identified a total of 60 RCTs that met our inclusion criteria. The greatest number of RCTs were conducted for interferon alpha-2a indicated for chronic myelogenous leukemia (CML) (n = 7), aldesleukin indicated for metastatic renal cell carcinoma (n = 7), and rituximab indicated for non-Hodgkin B-cell lymphoma (n = 6). The majority of trials we identified comprised the evidence base used for regulatory approval.

 summarizes the clinical evidence grades assigned in this study. Twelve drugs had moderate-quality evidence, including cladribine for hairy cell leukemia and arsenic trioxide for acute promyelocytic leukemia. Although we did not identify any RCTs for these drugs that met our inclusion criteria, their clinical bodies of evidence consisted of observational studies that consistently reported similar treatment effects or a dose-response association with little threat to validity. Individual GRADE ratings assigned to each drug and their clinical bodies of evidence are listed in Table 3.

 shows that there are drugs indicated for cancers with lower prevalence that had high-quality evidence and drugs indicated for cancers with higher prevalence that had low-quality evidence or lacked a sufficient body of evidence for review. The Spearman rank correlation coefficient between disease prevalence and the level of evidence for oncology orphan drugs was 0.3 (95% confidence interval [CI], 0.0-0.5). This indicates that there may be a weak correlation between the prevalence of rare cancers and the number of RCTs conducted for drugs indicated to treat rare cancers. A total of 14 drugs in our study received accelerated approval for the orphan indication. Of these, zero had high-quality bodies of evidence and 5 had moderate-quality bodies of evidence. Although our sample size was small, these results provided some indication that the accelerated approval provision does not optimize high-quality evidence generation for oncology orphan drugs.

	We identified 21 cost-effectiveness studies that met our inclusion criteria. Of those studies, 10 evaluated the use of either interferon alpha or imatinib mesylate, or compared both for CML. 

 summarizes the cost-effectiveness studies included in this assessment and describes comparators, study perspectives, base-case incremental cost-effectiveness ratios, and results of sensitivity analyses.

	The cost-effectiveness analyses received QHES scores ranging from 72 to 100, with a mean score of 85. The clinical data used to inform the studies were largely obtained from either randomized trials or patient medical records. The majority of studies implemented modeling techniques to project longer-term health outcomes that could not be obtained from limited clinical data. Cost inputs were obtained from administrative databases and the published literature. The most common shortcomings were as follows: 11 studies used data inputs from sources that might potentially incorporate bias and did not provide explicit discussion about the direction and magnitude of potential biases; 6 studies did not specify the source of funding for the study, which is recommended to add transparency; and 5 studies either failed to specify the time horizon for analysis, conducted analysis using a short time horizon that is not expected to fully capture clinically significant costs and effects, or did not implement adequate discounting for both future costs and outcomes. Table 4 reports the quality scores assigned to each cost-effectiveness study using the best-case scoring method where full points were assigned to inapplicable criteria. Scores assigned using the worst-case approach, where zero points were assigned to inapplicable criteria, are reported in parentheses.

	We conducted a systematic literature review to identify and assess clinical and economic evidence for 47 oncology orphan drugs marketed in the United States and applied 2 independent grading frameworks to the selected studies to critically assess the quality of each body of evidence. The supporting bodies of evidence available for marketed oncology orphan drugs vary in quality, with limited evidence demonstrating their economic value.

	Although the GRADE methodology is relatively explicit, reviewer judgment is required. In particular, we found that it was necessary to grade certain criteria subjectively in order to better accommodate oncology orphan drug characteristics. For example, the majority of RCTs for oncology orphan drugs had relatively small sample sizes that could lead to substantial imprecision (eg, lack of power, wide confidence intervals). We assigned lower ratings for studies that did not meet predetermined enrollment criteria or appeared underpowered to detect differences in their primary end point, but we did not consider a smaller sample size in itself to be a limitation. We critically assessed each study and lowered ratings when we identified instances of bias that were not explicitly addressed. We also lowered ratings when studies were poorly reported, or when bodies of evidence consisted of studies that reported inconsistent treatment effects.

	The number of RCTs identified in our study is comparable to the number in a study conducted by Tsimberidou and colleagues that evaluated the long-term marketing outcome of oncology drugs approved by the FDA without a randomized trial.

 In their study, 68 approved oncology drugs were identified and almost half (n = 31) were approved without an RCT. Kesselheim and colleagues recently evaluated pivotal trial characteristics of orphan and nonorphan drugs approved between 2004 and 2010 to treat cancer and concluded that pivotal trials for approved oncology orphan drugs were more likely to be smaller, nonrandomized, and unblinded, and to use surrogate end points to assess efficacy.

 The study is limited in that it only evaluated pivotal trials and did not account for other available clinical information. Also, the study was restricted to recently approved oncology orphan drugs and did not review the entire pharmaceutical category. We observed the same characteristics in the clinical trials that were included in our study. We also observed that there is a lack of published post-marketing studies and information about longer-term safety and efficacy.

	We hypothesized that the level of evidence available for oncology orphan drugs would have a strong correlation with disease prevalence in that larger patient populations enable better evidence generation. However, our results suggested a potentially weak correlation between the prevalence of rare cancers and the level of evidence available for drugs indicated to treat rare cancers. A weak relationship suggests that evidence development for oncology orphan drugs may not depend as much on the size of a particular patient population. Instead, evidence development in this pharmaceutical category may depend more on other factors, possibly regulatory requirements or reimbursement considerations.

	The overall dearth of cost-effectiveness studies in this pharmaceutical category, and in oncology as a whole, may reflect evidence limitations or publication bias, where studies of drugs with higher costs, greater benefit-risk uncertainty, or lower effectiveness are not published. In assessing the economic challenges of orphan drugs, Drummond and colleagues stated: “In short, if standard health technology assessment (HTA) procedures were to be applied to orphan drugs, virtually none of them would be ‘cost-effective’.”

 This conclusion was largely based on 2 factors: (1) high incremental cost per quality-adjusted life-year and (2) insufficient breadth and quality of clinical evidence for orphan drugs compared with drugs for more common diseases.

 Our results demonstrated that, contrary to these prior suggestions, it is feasible for some oncology orphan drugs to be considered cost-effective in specific healthcare settings using standard methods of health technology assessment. The clinical and economic value of each orphan drug should be assessed individually, on a case-by-case basis. We observed that all of the drugs considered cost-effective in their respective studies had moderate-quality to high-quality bodies of clinical evidence. This finding suggests there may be a relationship between the level and quality of clinical evidence and the likelihood that an oncology orphan drug has published estimates of cost-effectiveness.

	Evidence development is costly and challenging for all healthcare interventions, but particularly for orphan drugs, due to smaller patient populations and limited clinical knowledge of rare conditions. The trade-offs to generating more robust bodies of evidence may include delayed product accessibility, higher costs, or reduced availability of therapies. For conditions that have therapeutic alternatives, these trade-offs may be acceptable. However, for life-threatening conditions with limited therapy options, these trade-offs may not be considered acceptable. For rare diseases with only a single treatment option, one may question whether economic analysis should apply. Cost-effectiveness analysis is useful in that it provides information about the value of a health intervention compared with an alternative, which may be best supportive care or no therapy. From an equity perspective, McCabe and colleagues argue that there is no sustainable reason why the cost-effectiveness of orphan drugs should be evaluated differently from other drugs.

 However, healthcare systems may wish to consider additional factors when making decisions about orphan drugs, such as budget impact, disease severity, availability of alternative therapies, or societal preferences toward patients with rare diseases.

	Our study highlights 3 important policy questions. First, what types of study designs, incentives, or methods can be used to encourage better evidence development for oncology orphan drugs? Second, how much evidence is necessary or considered sufficient to healthcare decision makers, and what types of evidence should be generated prior to and after marketing approval? Finally, what are the process and decision-making criteria for evaluating comparative effectiveness data for oncology orphan drugs? Emerging private and public initiatives are attempting to address these questions to some extent with evolving methods such as coverage with evidence development

 but additional innovative analytic methods and policy approaches are necessary.

	We recognize several limitations to this study. The most important limitation is that the GRADE assessment framework provides information about the amount of confidence that can be associated with a body of evidence and does not relay any information about the magnitude of clinical benefit or safety of a drug. Certain oncology orphan drugs with low-grade published evidence may yield significant clinical benefits, and conversely, certain drugs with high-grade evidence may yield very little benefit. Currently, there is no established framework for quantifying the magnitude of benefits or risks from health interventions. Garrison and colleagues suggest pairing CER and benefit-risk analysis into 1 framework and describe how cost-effectiveness analysis models could be adapted to conduct quantitative benefit-risk assessments.

 It is important for manufacturers to engage in continuous evidence generation, even after a product is commercialized, so that the effectiveness and safety of their products can be accurately assessed. Comparative effectiveness research methods that capture and incorporate nonpublished clinician or patient experiences could also be useful to help better identify the real-world value of marketed oncology orphan drugs.

	Given the large number of drugs included in this review and the numerous keywords we could have used to search for potential clinical and economic studies, this review may not have captured all relevant drugs or studies. We used broad search terms in 2 major databases and also conducted manual searches of reference citations; this methodology allowed us to screen as many studies as possible. Only 1 author (MMC) reviewed potential studies, abstracted information, critically assessed articles, and assigned quality ratings. Although attempts were made to be accurate and consistent, it is possible that unintentional errors and inconsistencies could have reduced or improved the quality rating for a body of evidence. The QHES is limited in its ability to identify poorly analyzed studies and does not have a benchmark for total scores, which limits its ability to quantitatively categorize and distinguish high-quality studies from low-quality studies.

	The results of our study show that oncology orphan drugs marketed in the United States have varying levels and quality of clinical evidence and a shortage of evidence demonstrating economic value. It is uncertain whether the current evidence levels for oncology orphan drugs marketed in the United States are sufficient to support decision-making practices consistent with principles of evidence-based medicine and CER. This issue remains an open policy question that requires additional evaluation.

This study critically assessed the published clinical and economic evidence supporting oncology orphan drugs marketed in the United States.

Systematic Review of Comparative Effectiveness Data for Oncology Orphan Drugs

Despite the pervasiveness of laboratory tests and their importance to medicine, evidence for their appropriate use often is very limited. In this article, we argue for a fundamental restructuring of the process by which laboratory tests are evaluated and reimbursed. We present an approach that would promote more evidence-based appraisals for laboratory tests. In addition, we urge that coverage and reimbursement for laboratory tests move toward an evidence- and value-based approach, using the tools that largely have been adopted for pharmaceuticals by many US healthcare payers. To address this information gap for laboratory tests, we note several potential strategies to encourage manufacturers, laboratory service providers, and payers to collect outcome and cost data that will better support effective use of new laboratory tests. Integral to increasing appropriate use and reimbursement will be the development of a common language and format for dialogueâ€”facilitating the development, review, and delivery of evidence-based tests by manufacturers, clinical laboratories, and healthcare payers. 

Laboratory tests are an important yet often underappreciated component of healthcare and the health economy.

It is estimated that as much as $56 billion will have been spent on laboratory diagnostic services in 2005.

More important, laboratory tests by their very nature are designed to initiate a cascade of decisions regarding further testing, prevention, or treatment–decisions that ultimately determine the course of illness and cost of healthcare for patients who receive them. A recent report estimated that although diagnostics account for 1.6% of the Medicare total costs, they influence 60% to 70% of downstream treatment decisions.

Despite their pervasiveness and importance to medicine, the evidence for the application and utility of laboratory tests often is limited.

Tests being brought to market do not require the same data as therapeutics do, and technology assessments of laboratory tests have not undergone the revolutionary changes in evidence review that have occurred for drugs.

Instead, payment for tests is linked to an antiquated coding system.

 Finally, innovative insurance coverage and payment policies that are common for drugs are rare for laboratory tests.

In this article, we argue for a fundamental restructuring of the way laboratory tests are evaluated and reimbursed. We propose that coverage and reimbursement for laboratory tests should move toward an evidence- and value-based approach, using some of the tools that have been adopted for pharmaceutical assessment by many US healthcare payers. Furthermore, reevaluating the methods used to assess and pay for tests is a timely endeavor, as diagnostic tests are on the forefront of the trend toward personalized medicine.

Public and private health insurers pay for tests performed in laboratories that meet standards certified under the Clinical Laboratory Improvement Amendments (CLIA) of 1988, according to 

code does not exist, health insurers sometimes reimburse the new test by using a 

code for an existing test, a process known as "mapping."

code is established, test payments are established by "cross-walking" or "gap-filling." In cross-walking, a new test is determined to be similar to an existing test, multiple existing test codes, or a portion of an existing test code. Payment is set at an "appropriate" percentage of the payment for the existing test.

In gap-filling, insurers are left to determine an appropriate payment amount for the new code. The Centers for Medicare & Medicaid Services uses local carrier decisions as a basis for the next year's clinical laboratory fee schedule.

A recent report by the Institute of Medicine suggested that the process of establishing reimbursement levels for novel laboratory tests is out-of-date and not equipped to handle emerging diagnostic and genomic tests: 

[Medicare] payments for some individual tests likely do not reflect the cost of providing services, and anticipated advances in laboratory technology will exacerbate the flaws in the current system. Problems with the outdated payment system could threaten beneficiary access to care and the use of enhanced testing methodologies in the future.

More broadly, some argue that reimbursement must more accurately reflect clinical and economic value.

Public and private payers have not developed uniform methods for requesting information from laboratory test service providers about novel tests. There are several likely reasons. The first reason is historical: most health plans do not have standing test evaluation groups similar to pharmacy and therapeutics (P&T) committees, possibly because laboratory tests have not been viewed as clinically or economically significant enough to justify the expense associated with these groups. Second, in contrast to pharmaceutical manufacturers, neither test manufacturers nor laboratory service providers generally have large, sophisticated marketing teams targeting physicians, health plans, and patients. (A relevant exception may be the marketing of the ThinPrep test.

) Third, as noted above, federal regulatory requirements do not compel manufacturers or distributors to provide the type of evidence (eg, randomized clinical trials) that lends itself to systematic review.

Finally, the way laboratory tests are used in practice makes it difficult to apply the pharmaceutical evaluation model. Unlike most pharmaceuticals, laboratory tests are used in a wide range of clinical settings. For example, a complete blood count is used in hundreds of clinical situations, while antihypertensive drugs are used in a limited number of conditions. 

In lieu of an evidence-driven, value-based approach, the process of adopting new laboratory tests is essentially what economists call an "administering pricing" system. Pricing is set through a negotiation process that may be based on historical comparators in the case of cross-walking or perceived levels of analytic complexity in the case of gap-filling. As a result, there is little reward for creating additional value (either in a clinical or an economic sense) and hence little incentive to create the evidence to support value creation (L. P. Garrison, PhD, and M. J. F. Austin, PhD. The economics of personalized medicine: a model of incentives for value creation and capture. Unpublished observations, January 2006). We describe a process that is designed (1) to improve the quality of the evidence that is available when an insurance coverage decision is being made and (2) to increase the transparency and uniformity of the process by which payers request information from test manufacturers or laboratory service providers. 

Developing a Language for Describing Benefit

Test manufacturers, laboratory service providers, and health insurance plans will benefit from standardizing the way evidence supporting new laboratory tests is presented. Methodological standards for the evaluation of diagnostic tests have been published.

In addition, several domains relevant to health insurers, clinicians, and patients are considered in a published framework to evaluate diagnostic technologies (

Although many agree about the value of using these domains to evaluate tests, there is less agreement on how much evidence is necessary for an insurance coverage decision. 

In addition to considering the relevant domains from the Table, it is important to evaluate the incremental impact of a test; that is, the improvement that the new test provides over current diagnostic strategies (L. P. Garrison, PhD, and M. J. F. Austin, PhD. The economics of personalized medicine: a model of incentives for value creation and capture. Unpublished observations, January 2006). Because prospective trials directly comparing new laboratory tests with established diagnostic strategies are uncommon (particularly those evaluating the impact of the tests on patient outcomes), decision-analytic modeling techniques often are necessary to conduct quantitative evaluations.

Modeling is an underappreciated approach to evaluating new tests. Models help frame questions, provide transparent mechanisms for stating hypotheses about cause and effect, highlight deficiencies in clinical data, and force decision makers to make explicit judgments about values for data that are used to inform the model. Although some clinicians and health insurance plan executives remain skeptical of decision models, quality standards for models have been published and are readily accessible to decision makers who wish to assess their quality.

We define value for laboratory tests as it is defined for other health technologies: The intervention provides an overall benefit to the patient at an acceptable cost (ie, it is cost-effective). Petitti notes that there are 4 well-recognized criteria for identifying an intervention as cost-effective

More effective and more costly, with the added benefit worth the added cost. 

Cost saving with an outcome equal to or better than that of the alternative. 

Assessing value for tests can be difficult because tests are intermediate steps in the treatment pathway. The advantage of Petitti's framework

is that it allows flexibility, because value for tests can be defined narrowly (eg, the least expensive way to make a diagnosis) or broadly (improvement in survival at an acceptable added cost). 

Format for Dialogue Among Test Manufacturers,

Manufacturers and laboratory service providers cannot expect, and payers cannot promise, coverage of and appropriate reimbursement for every new test that comes to market. Similarly, payers cannot expect, and manufacturers and their delivery partners cannot promise, the same level of evidence for all tests. Nevertheless, the process should be transparent, and adopting a standard format would help clarify expectations and improve the decision-making process. We propose a process for information content and interaction that is intentionally similar to the format developed by the Academy of Managed Care Pharmacy (AMCP) for the evidence-based evaluation of drugs. More than 50 public and private health insurers covering more than 100 million lives have adopted the AMCP format.

 provides a template for manufacturers' reporting of clinical and economic information regarding laboratory tests. This template takes into account differences in evidence that are typically available for new laboratory tests compared with new pharmaceuticals. 

Some health plans might have a designated organizational unit that evaluates laboratory tests, providing a structure for soliciting and reviewing manufacturers' products. In the case of pharmaceutical products, that unit is the P&T committee. Health plan staff working with P&T committees usually are receptive to receiving information from manufacturers or service laboratories about new products. Indeed, the "unsolicited request" process pioneered by AMCP was designed to create a structure for dialogue between manufacturers and payers. A similar scheme would improve transparency and the flow of information for new laboratory tests. 

P&T committee members often are not health plan employees. Having such a quasi-independent group evaluate novel laboratory tests may improve the credibility of the decision process in the eyes of manufacturers, clinical laboratories, physicians, and patients. Still, maintaining committees is costly, and such maintenance may not be justified given the relatively low volume of tests that are introduced annually. One option is to fold the test evaluation process into the existing P&T structure. Alternatively, payers could hire consultants to evaluate select novel tests and make recommendations regarding coverage and reimbursement. 

Payers should be timely both in coverage decisions and in setting reimbursement levels, and when decisions are made, they should be supported with a rationale. In cases where requests for coverage are denied, such information allows manufacturers to design studies or collect other data that address concerns regarding the quality or content of the information supporting the product. 

To address known methodological weaknesses in studies evaluating diagnostic tests,

we describe in this section options for improving the quality of information. Some focus on market incentives; others are more regulatory in nature. At this early stage, we do not advocate one approach over another. Rather, we hope to foster a dialogue among stakeholders. 

Testing services in the United States are regulated by both federal and state laws and agencies, and are further reviewed by accrediting professional entities. All laboratory tests that meet the legal definition of in vitro diagnostic devices (IVDs; generally, diagnostic assays made for distribution outside a single provider laboratory) are evaluated by the US Food and Drug Administration's (FDA's) Center for Devices and Radiological Health. However, most tests offered by laboratory service providers are not IVDs; the manufacture of analytes or other components of such tests (called analyte-specific reagents) may need to meet FDA specifications for legal distribution. IVD approval, which typically is based on analytical performance, does not require other types of evidence that the FDA often mandates for devices that carry significant risk (eg, prospective controlled trials for implantable defibrillators). Although laboratory tests usually pose little direct risk of injury or death, they often lead to a cascade of clinical decisions, sometimes involving procedures that do carry significant risk for patients. For example, a positive prostate-specific antigen test often leads to prostate biopsy, which carries risk of bleeding and infection. As an extreme example, women whose genetic tests show BRCA mutations often choose to have prophylactic mastectomies and oophorectomies.

Genomic tests that are used to guide therapy choices also could represent a class of laboratory assays that pose unique patient risks. 

To address potential downstream indirect risk, the FDA could require test manufacturers or service providers distributing IVDs to submit clinical algorithms with recommended follow-up plans based on test results. In cases where the downstream consequences are significant, the FDA could require specific post-marketing studies. Another area of concern is laboratory-developed or "home brew" tests–those that are supplied by a single laboratory or laboratory company and therefore are not subject to federal regulatory review beyond CLIA. Many tests, including genetic assays, are brought to market as home brew tests.

 Strengthening regulatory requirements for some of these tests might enhance the ability of payers and clinicians to gauge their value. With a goal of more uniform quality standards for testing, federal, state, and professional oversight measures should be considered in toto. 

To increase funding of high-quality clinical studies focused on patient outcomes,

some advocate levies on medical services to fund prospective and retrospective clinical and economic studies for new and existing technologies.

Such research could be directed through existing agencies within the National Institutes of Health, the Agency for Healthcare Research and Quality, or freestanding private clinical research facilities. Such a system would be expensive and difficult to administer and manage only for laboratory tests, however, given the inconsistencies in reimbursement for tests across the settings where they are ordered. Levies also may create a significant disincentive for test development. In this context, levies could not be directed solely at laboratory testing; rather, they should include all reimbursable health services. 

-based reimbursement, if novel tests are not promptly and specifically assigned codes, reimbursement and hence use are hindered. And because 

codes are updated only annually, opportunities for gathering clinical information are lost. To avoid these problems, novel tests could be assigned unique temporary 

codes that would allow billing and tracking of the test's use in delivery systems, and its effect on clinical decision making and patient outcomes. 

These unique codes would allow manufacturers or service laboratories to obtain reimbursement for novel tests, but such a strategy would shift a large part of the cost of test development and evaluation to payers. To address this problem, a risk-sharing approach could be implemented: payers could agree to preliminarily cover new tests on the contingency that manufacturers and service laboratory providers have a certain amount of time to establish clinical utility and value. Although these types of agreements are just beginning to be used for pharmaceuticals, they may offer promise for laboratory tests, given the fact that the evidence available at the time of market launch is different from the evidence that will be available after clinical use.

Finally, and perhaps most important, consideration should be given to rewarding innovative and novel tests with greater reimbursement based on an explicit measure of the additional value they create. Health economists have developed techniques for measuring the incremental value of new technologies in terms of cost savings, morbidity reduction, and life extension, and for integrating these impacts into models that provide a measure of overall incremental value. Furthermore, higher expected levels of reimbursement can be used to justify greater investments in measuring and demonstrating ultimate clinical utility and economic value. 

Laboratory tests are developed under a different regulatory structure than pharmaceuticals and have a unique and complex function in medical care. Novel tests may come to market with little information supporting their role in clinical decision making or evidence regarding their impact on patient outcomes. To address these issues and to improve patient care and outcomes, we have outlined a structure and process for information sharing among laboratory test manufacturers, clinical laboratories that provide tests, and payers that make coverage and reimbursement decisions about these tests. Although it is unlikely and perhaps unnecessary that the evaluation process for laboratory tests will equal that required for drugs, we can move much further toward a system that supports better gathering and sharing of high-quality evidence. Test manufacturers, clinical laboratories, payers, and clinicians all must play a role in this process. 

From the Fred Hutchinson Cancer Research Center, Seattle, Wash (SDR); the Department of Medicine (SDR), the Department of Pharmacy (DLV, LPG, SDS), and the School of Public Health (RC, JC), University of Washington, Seattle; and the Laboratory Corporation of America, Research Triangle Park, NC (PB). 

This research was supported by an unrestricted grant to the University of Washington from Laboratory Corporation of America. 

Address correspondence to: Scott D. Ramsey, MD, PhD, Fred Hutchinson Cancer Research Center, 1100 Fairview Ave N, M2-B230, PO Box 19024, Seattle, WA 98109. E-mail: sramsey@fhcrc.org. 

Toward Evidence-based Assessment for Coverage and Reimbursement of Laboratory-based Diagnostic and Genetic Tests

 To determine the potential impact of genetic testingand pharmacogenomics on healthcare delivery and costs.

 We examined 3 examples: (1) BRCA1/2 testing forbreast cancer risk, (2) HER2/neu overexpression testing to guidedrug treatment in women with breast cancer, and (3) CYP2C9 testingbefore the use of the anticoagulant warfarin. We discussedeach genetic testing example from the perspective of the patient,provider, insurer, industry, government, and society.

 The expanded use of genetic information offers manypotential clinical benefits, but also many economic challenges.One of those challenges will be managing the impact of genetictesting on healthcare delivery and costs.

 Systematic, evidence-based technology assessmentsand economic evaluations will have to be used to guide theincorporation of genomics into clinical practice. More researchalso will be needed to assess patient preferences and willingness topay for genomic technologies; how providers can assess and usegenomic technologies; and how the industry, insurers, and governmentcan best balance the relevant costs and benefits.

Many have asserted that the wealth of knowledgeoffered by the human genome portends a timeof rapid change in medicine.

 However, genomictechnologies will influence not only health outcomesbut also the delivery and cost of healthcare; and in thisera of increasing concern about healthcare costs, it willbe impossible to consider the implications of genomicmedicine without also considering the economic implications.

 Harold Varmus, MD, former director of theNational Institutes of Health, noted in an editorial in 

David L. Veenstra, PharmD, PhD

Articles

Reimbursement for Genetic Variant Reinterpretation: Five Questions Payers Should Ask

Systematic Review of Comparative Effectiveness Data for Oncology Orphan Drugs

Toward Evidence-based Assessment for Coverage and Reimbursement of Laboratory-based Diagnostic and Genetic Tests

Genetic Testing and Pharmacogenomics: Issues for Determining the Impact to Healthcare Delivery and Costs

Managed Care in the Genomics Era: Assessing the Cost Effectiveness of Genetic Tests