This publication was supported by Pfizer, Inc.

 How would you define real-world evidence (RWE)?

 The FDA defines RWE as the clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of real-world data (RWD). This is very consistent with how a manufacturer would define RWE. In the broadest sense, RWE is any data that is developed outside of a clinical trial setting. RWE can play a pivotal role in helping us assess the efficacy and safety of our medicines at a population level, outside of the confines of controlled clinical trial. It is also an incredibly valuable tool to communicate the benefits and risk associated with a medicine to patients, payers, regulators, and providers.

 It depends on the need. Patient-reported outcomes (PROs) can be a source of RWD. It could be useful to understand the experience of the patient as they are taking the medication, for instance, adverse effects. That data sometimes is captured in more meaningful ways in phase 4 assessments post launch as a product comes to market and is used in a larger population. The second domain can be claims data extracts; how is the product being used? What do we see within our claims systems in terms of diagnosis, population, [and] ages? After they use the product, what additional diagnosis are they experiencing? What other procedures are they experiencing while on the medication? Are the services rendered suggesting that a product is or isn’t working based on its intended use? Another domain of that is moving away from claims and into the transactional information of care, such as the electronic medical records (EMRs). From EMRs, structured data can be collected and insights can be gleaned. These data fields might indicate someone’s illness or other medications they are using. That data is most useful data when codified in some way, but [there is] also free text information within the EMRs—which can be useful through reading over notes and what’s happening post–care delivery. Machine learning has been used to mine these free-text notes for meaningful insights. We can understand unexpected consequences of therapy, and can create new data sets and instruments. Those are some main areas of RWE. Then metadata can be established, which is a combination of those pieces of information-gleaned insight. What are we understanding about a product launch as it is used in a practical setting? What is the confluence of those things telling us?

 How do you utilize RWE in decision-making?

 One way to understand whether or not a product is being used in a way differently than intended, either from the labeling of the product or from the policy that we create establishing the criteria for use. We can understand if those are being followed or if there is drifting of utilization. We also can learn about discontinuation of therapy. Are we investing in therapies that are not seeming to reach the clinical end points and not being continued? That’s very important information. [For example, in] clinical studies, a product may perform at, say, a 70% efficacy level, reaching some end point in a controlled, observed environment in which patients are taking their medication, and we are making sure that they stay in the trial. Outside of that trial, maybe people have a high discontinuation rate, and, therefore, the therapy is only 50% effective. If we make decisions around the coverage of that product and the value proposition of it based on it being 70% effective, then it is only 50% effective. Those assumptions tend to be faulty in terms of economic impact. It would be less cost effective than it seems. That data can be cycled in, not only from the perspective of contracting for what we pay for it, but it also could take the form of supporting new types of innovative contracting (eg, risk sharing). If someone fails the therapy or discontinues the therapy, there might be some type of remuneration back to the health plan sponsor to make them whole for the investment, or at least part of the investment, in a therapy that didn’t reach its end point.

 Does this differ in oncology versus other therapeutic areas? If yes, how?

 The interesting thing about oncology RWE is that it tends to be a condition for which there are very defined end points that we can observe. So, whether they are adverse events or failures of therapy, there is a good way to know that information. If someone has recurrence of cancer, we can detect that. There are other disease states [for which] the picture is not quite as clear. Let’s say someone with diabetes has a cardiovascular event; sometimes, it is hard to know if that cardiovascular event was due to their underlying diabetes or some other cardiovascular factor. As you get further away from that kind of evidence, it’s harder to know their association. Oncology, because of the end points as well as [the possible speed of recurrence], is a very attractive disease state for looking at that data.

 What do you believe are the best sources of RWD (eg, EMRs, claims data, registries)?

 The research question that you want to address should drive the real-world data source that you use to generate RWE. Each of the data sources listed above [has its] own advantages and disadvantages. For example, EMRs often have rich data on clinical variables that might not be captured in claims data. So, if your research question is assessing clinical outcomes associated with a particular tumor type, you will give preference to EMR data or registry data over claims data. On the other hand, if you want to look at a large patient population, since a payer is making decisions at the population level, you would prefer claims data with a large number of patients included. With technological advances in EMRs and data curation, we now have the ability to integrate data from both data sources so you get the best of both worlds.

The interesting thing about oncology RWE is that it tends to be a condition for which there are very defined end points that we can observe. So, whether they are adverse events or failures of therapy, there is a good way to know that information.


 From a manufacturer’s perspective, historically, RWE has been used to understand unmet need, treatment patterns, cost of illness, prevalence, and incidence of different diseases in the target patient population for a drug. The availability of larger data sets have also enabled researchers to generate comparative effectiveness data once a medicine has been on the market for a duration of time or when newer competitor medicines are introduced. With the passing of the 21st Century Cures Act, there has been a push toward the use of RWE to supplement data from clinical trials, either to contextualize or to compare clinical trial evidence (ie, synthetic control–arm studies). In addition, we have also seen label expansions resulting from the use of RWD.

 How do you think that RWE can address knowledge gaps in oncology?

 Oncology has its own nuances vs other therapeutic areas when it comes to RWE. Recently, a lot of new drugs that are being developed are [for treatment of] rare tumors, biomarker-driven patient populations, or patients [who have certain cancers with] a poor prognosis [for which] the only treatment option available is palliative care. Moreover, there have been products approved based on data from early-phase, single-arm clinical trials, all of which leads to uncertainty in terms of outcomes in a [real-world] setting and broad patient population. These knowledge gaps can be very important for payers to address, and RWE can play an important role in addressing these knowledge gaps. For example, findings from external control, [real-world] studies complement efficacy data from single-arm trials in successful oncology product approvals.

 What type of RWE do you find most beneficial?

 With new product approvals, the treatment landscape changes from the time that the clinical trials are designed. Therefore, the standard of care evolves, and it also impacts treatment guidelines. Moreover, payers don’t have head-to-head comparative data to inform their formulary decision-making when there are multiple products approved in the same class. Under these circumstances, RWE-based comparative effectiveness studies can be very beneficial for payers and physicians in helping them with formulary or treatment decisions. In addition, with the introduction of newer therapies, RWE studies assessing adherence, adverse events, and resource utilization can also be very informative. Finally, clinical trials use restrictive inclusion and exclusion criteria in patient selection, thereby limiting the generalizability its findings. Therefore, RWE studies validating findings from trials can further provide value to payers.

 Please discuss findings from the recent survey conducted that assessed payer perceptions of the use of RWE in oncology decision-making.

 Please discuss study design, results, and key takeaways.

 This was a survey of US payers to understand their perceptions of the use of RWE in informing oncology formulary decisions.

 A multidisciplinary steering committee developed a web-based survey and piloted the survey with 5 US payers. Based on comments received from the pilot survey, we revised the questionnaire and fielded the survey to more than 200 US payers who were part of the Academy of Managed Care Pharmacy (AMCP) Market Insight Program. This panel included pharmacy leaders involved in medication product evaluation and/or utilization management who agreed to participate in payer-based research. After the survey was completed, we convened a virtual panel discussion with 10 US payers to discuss the survey findings. Findings from this survey showed that most US payers use RWE to help inform oncology formulary decision-making. In particular, US payers thought RWE was useful to assess comparative effectiveness where head-to-head clinical trials are not available and to help inform clinical guidelines. Based on the survey, almost half of US payers reported [using RWE] to help inform off-label coverage decisions.

RWE can be very helpful in contextualizing clinical trial data under the following circumstances—[in the] niche patient population with rare tumor types or diseases or in the case of single-arm clinical trials in which it might be unfeasible or unethical to conduct a randomized trial.

 Do you generate your own RWE? If yes, why? How is this accomplished?

 Yes, we generate and publish RWE across all phases of drug development. Prelaunch, we generate RWE to understand disease epidemiology, standard of care, treatment patterns, and disease burden. In the postlaunch setting, we develop RWE to understand real-world utilization outcomes beyond those studied within the clinical trial (eg, adherence, costs, resource utilization, real-world outcomes), validate clinical trial findings, and [accomplish] comparative differentiation. Manufacturers have access to several data sources (eg, claims data, EMRs, registries) along with in-house analytic capabilities to generate robust RWE. Smaller health plans often don’t have these capabilities and rely on the manufacturer to generate and disseminate these findings.

 Your colleagues and you published a review concerning RWE inclusion in FDA submissions.

 Please discuss review methodology, findings, and key takeaways. Why do you think that RWE is so infrequently included in submissions?

, is the first of its kind to systematically aggregate detailed regulatory feedback to provide practical insights to manufacturers.

 Whereas existing FDA guidance documents provide a theoretical framework for conducting regulatory RWE studies, the details of how to actually design and analyze adequate RWE studies remain largely unaddressed. To meet this critical knowledge gap, a team of researchers from Pfizer, Analysis Group, and the Dana-Farber Cancer Institute analyzed 133 original and 573 supplemental oncology New Drug Application and Biologics License Application approvals to identify the attributes of a successful RWE study that contribute to an accelerated or full drug approval. The key takeaways from our review were as follows:

to confirm appropriate data sources and whether the RWE study should be designed as a natural history study for contextualization or as an external control study for comparison with the pivotal trial. A hybrid study design to combine trial with external control data through Bayesian or frequentist methods and ambidirectional RWE data collection (both prospective and retrospective) are study designs worth considering.

 to ensure that RWD are of high quality and fit for purpose. Although chart review was the most common source for RWE, the FDA also commented that data from such studies could have limited generalizability and [be] subject to selection bias.

Align the RWE and pivotal trial populations

 by matching on-trial inclusion and exclusion criteria to the extent possible and adjusting for the remaining imbalance in baseline characteristics with propensity score weighting methodology, such as inverse probability treatment weighting. Critically, the study protocol needs to be developed 

Describe methods to minimize residual confounding and unmeasured confounding

, including appropriate index date and reduction in missing values. If imputation methods are used to address missing values, then validation of the imputation algorithms is recommended. The impact of unmeasured confounding should be evaluated through quantitative bias analysis.

Whereas these findings might seem straightforward, they are often challenging to implement in practice. For example, each data source has its own strengths and limitations, so getting the right data source, understanding the limitations, [and] curating it to develop regulatory-grade data can be a challenge in itself. Identifying and mirroring a clinical trial can also be difficult, especially for newer, biomarker-driven products [for which] the biomarker might not routinely be tested and captured in RWD sources. Also, patients who are prescribed products in a real-world setting can be difficult to match with the patients who are enrolled in clinical trials using strict inclusion and exclusion criteria. For all these reasons, RWE that is generated is not frequently used in supporting regulatory submissions or, even if submitted, might not be accepted by regulatory agencies [for decision-making].

 In which cases do you think that RWE can be most helpful as part of an FDA submission?

 RWE can be very helpful in contextualizing clinical trial data under the following circumstances—[in the] niche patient population with rare tumor types or diseases or in the case of single-arm clinical trials in which it might be unfeasible or unethical to conduct a randomized trial.

 Would you look more favorably upon an oncology drug that included RWE as part of the submission?

 That’s a difficult question to answer. Yes, sometimes. There is a baseline set of overall study information that we need, and all things being equal among, let’s say, 2 different agents, if 1 agent had RWE and the other one did not, I would look more favorably on that [first] agent. On the other hand, if you look at differences of performance within the clinical trial, and 1 product lacks RWE but has stronger efficacy in clinical trials, we would look at the magnitude of the difference to determine which has the stronger performance. It’s a difficult trade-off to consider but necessary to evaluate.

 Are you utilizing outcomes-based contracting at your organization? Why or why not?

 Yes, but in a very limited fashion. There wasn’t anything needed in our market for some time just based on the clinical management strategy. We have recently launched into OBCs that with a small number of contracts, and we are contemplating a much larger footprint of those contracts as we think about vendors such as pharmaceutical manufacturers sharing in the accountability of delivering good outcomes between the plan, the pharmaceutical supplier, the provider, and the member and aligning that. It’s a key strategy as we move forward and make sure that we are all pulling in the same direction in terms of our incentives to drive value overall.

 What educational opportunities exist to increase RWE understanding and evidence generation for payers? How would this best be accomplished? What tools are needed?

 We are at a time of an inflection point around how information is used and what we consider is part of our reviews and our policy. For instance, composite end points have been shunned for many years by people in evidence circles who are considering coverage, and now they are accepted by the FDA as a primary end point for some studies for some drugs to be approved. In a similar way, RWE is becoming a very meaningful, important, and impactful set of measurements for understanding how products perform, and we need a paradigm shift [regarding] using those end points. Traditionally, people in the payer space who are making policy decisions have only thought of using information from controlled clinical trials in terms of their policy decision. We need to bring these new data sets in to learn more than we know today and accept those as part of our evidence package for policy consideration. I think it is going to take some time for that migration of thinking ... As we begin to teach that in our colleges of medicine and colleges of pharmacy and bring that into the evidence-based medicine community (eg, some of the thought leadership in that space, such as at Oxford, and other evidence-based medicine centers), we will begin to grow in terms of our use of that data and the acceptance of those data sets.

 How do you use PRO data in oncology decision-making, if at all? What types do you find most impactful?

 We look at several things; the most prominent are PROs around discontinuation and adverse effects. Understanding the experience folks have in terms of tolerability of therapy is key to knowing how to manage those adverse effects. How do we help a member or patient understand what they might experience? What types of treatment might they use, or what differences in their routine could help lower the instance of those adverse effects? Lastly, knowing when to discontinue the therapy. These are key opportunities for us to help improve the experience, improve quality of life, but also to know the likelihood of therapy to deliver on its promise.

 What do you see as the value of PRO data in informing policies and formulary decisions? What challenges have you experienced?

 Many people who do value assessments have a negative perception around patient-reported information, because it is subjective. People can report experiencing things that may or may not be related to the treatments, or they may not know how to describe [their experience] in a way that reflects what is happening with the product that is getting co-opted. It is something that is worth [being cautious about]. We think about how much credibility we give to [patient-reported data]. On the inverse of that is the opportunity of seeing through the lens of someone who is taking the therapy, and we can listen to [their] unique challenges or successes—that they were able to go to work more days than they had before. If a global assessment measures productivity and/or activity, those are very meaningful end points [that indicate] someone’s likelihood to stay engaged in their care. There are opportunities and challenges.

When making formulary decisions, US payers desire additional evidence beyond traditional safety and efficacy data. This may be more pronounced in oncology due to single-arm trial designs, niche patient populations, and accelerated regulatory approvals. Economic models help project long-term clinical and economic benefits of such therapies that payers can use to inform access decisions for medicines.


 In what other ways do you use PRO data or think it could be helpful?

 Other ways we use that information is to understand physician experience or physician preference for a certain therapy, so we can understand some of the information, some of the experiences, with care. We also utilize that information to appreciate member satisfaction. It’s a way to understand how satisfied someone is with their care, and, in particular, with their treatment. As we move into this world of Medicare quality, stars ratings, and [Consumer Assessment of Healthcare Providers & Systems] scores, those perceptions are needed for monetizing our plans and keeping members in our rolls. We are providing the therapies that people are feeling best on and are most likely to stay on. Those members are more likely to renew with our plan so that we can help to recoup the investment we made in those therapies.

 Please discuss findings from a recent survey by Oderda et al that assessed payer perceptions of the use of PROs in oncology decision-making.

Please discuss study design, results, and key takeaways.

 This was a survey of US payers to better understand their perceptions of how PROs impact formulary decision-making in oncology. A multidisciplinary steering committee developed a web-based survey and piloted the survey with 5 US payers. Based on comments received from the pilot survey, we revised the questionnaire and fielded the survey to more than 200 US payers who were part of the AMCP Market Insight Program. This panel included pharmacy leaders involved in medication product evaluation and/or utilization management who agreed to participate in payer-based research. After the survey was completed, we convened a virtual panel discussion with 10 US payers to discuss the survey findings. The survey showed that [the] majority of US payers, both small payers (< 1 million lives), and large payers (

 1 million lives), believed that PRO evidence was useful in informing oncology formulary decision-making. This was true for PRO data coming from clinical trials as well as [for real-world] PRO data. Almost half of the payers reported that lack of PRO data would influence oncology formulary review. Overall, these findings validated manufacturers’ efforts in collecting patient-centric evidence collected through PROs to enable patient access.

 How do you feel that PRO data could influence a value-based agreement, if at all? Do you see the use of PRO data increasing for your organization in the future?

 It could have a big influence on those agreements with information [on] potential problems in therapy, such as how likely [it is for] someone to discontinue because of adverse effects or how likely is it to improve quality-of-life aspects. Being able to frame that with a likelihood of the event in value-based agreements is a very powerful tool.

 How do you utilize economic models in oncology decision-making? What types of models do you find most and least beneficial? When do you find them most beneficial (eg, when evaluating products with similar safety/efficacy)?

 Models are at inflection points where [they] are becoming used, but folks still misunderstand them and misuse them. The models most commonly used for oncology in our area are budget-impact modeling and cost-effectiveness modeling. I find both helpful. I was training at the time, and I had a postdoctoral training that taught me the value of those tools, so I’ve employed those throughout my career. There is a way to contextualize and compare the value of treatments with other known interventions. It helps us understand things that are hard to wrap our heads around in terms of investments in high-cost therapies that have very high-cost morbidities associated with the disease.

 Please discuss findings from a recent survey by Biskupiak et al that assessed payer use of economic models in oncology decision-making.

This was a survey of US payers to better understand their perceptions of how economic models impact formulary decision-making in oncology.

 A multidisciplinary steering committee developed a web-based survey and piloted the survey with 5 US payers. Based on comments received from the pilot survey, we revised the questionnaire and fielded the survey to more than 200 US payers who were part of the AMCP Market Insight Program. This panel included pharmacy leaders involved in medication product evaluation and/or utilization management who agreed to participate in payer-based research. After the survey was completed, we convened a virtual panel discussion with 10 US payers to discuss the survey findings. The survey showed that [a] majority of US payers indicated moderate/most interest in cost-effectiveness and budget-impact models. Large plans (≥ 1 million lives) were more likely to have expertise in reviewing oncology models than [were] small plans (< 1 million lives). Common reasons for not reviewing economic models were that the models weren’t available at the time of review and [that there was a] perception of bias. More than two-thirds of payers reported that economic models were useful when comparing therapies with similar efficacy/safety. Overall, this study highlights the relevance of economic models in informing oncology formulary decision-making. Additionally, [the study results shed] light [on] opportunities that exist to improve the utility of oncology economic models.

 What do you see as the benefits and drawbacks of economic models? How could the drawbacks be overcome?

 First, the benefits. When making formulary decisions, US payers desire additional evidence beyond traditional safety and efficacy data. This may be more pronounced in oncology due to single-arm trial designs, niche patient populations, and accelerated regulatory approvals. Economic models help project long-term clinical and economic benefits of such therapies that payers can use to inform access decisions for medicines.

Economic models may have some drawbacks, as well:

. Payers often criticize models that are perceived to be too complicated and not transparent enough.

. When clinical trial [results are published], payers may still have some unanswered questions. As such, assumptions based on RWD or expert opinion are made. Whereas efforts are made to use assumptions based on national estimates, those assumptions may be subject to change over time and may not be reflective of a particular health plan.

Some economic models are cumbersome to use

. Most models have many tabs that may not be easy to navigate. As such, end users may get frustrated when reviewing models.

These limitations can be overcome in a few ways. First, to improve transparency, developers should reference all assumptions included in building the model. Additionally, highlight all assumptions that were included in the model, as it may help increase transparency. Ensure applicability of the model to the health plan. To ensure that the model is relevant to the health plan, models should allow end users to modify assumptions to better reflect patient demographics and clinical characteristics observed within a particular health plan. Additionally, sensitivity analyses can be included that allow end users to see the effect that certain variables have on the results. Creating user-friendly interfaces may also improve adoption and utility of the model.

Articles

A Q&A With Bhakti Arondekar, PhD, MBA; Alexander Niyazov, PharmD, MPH; and Jay Weaver, PharmD, MPH

Expert Perspectives on the Role of Real-World Evidence, Patient-Reported Outcomes, and Economic Models in Oncology Formulary Decision and Regulatory Submissions for FDA Approval

 Clinical Brief summarizes key findings from studies regarding health plan use of real-world evidence, patient-reported outcomes, and economic models in oncology formulary decision-making and regulatory submissions to the FDA. These studies were originally presented at the Academy of Managed Care Pharmacy Nexus 2020 Virtual meeting, held from October 19 to 23, 2020. The results from these studies were later published as 4 separate articles.

REAL-WORLD EVIDENCE, PATIENT-REPORTED OUTCOMES, AND ECONOMIC MODELS IN ONCOLOGY FORMULARY DECISION-MAKING

Randomized controlled trials (RCTs) are considered the gold standard for demonstrating safety and efficacy of a therapy. However, they may provide inadequate information for formulary decision-making, because the patient population in RCTs often does not reflect that in the real world.

 This may be particularly relevant in oncology decision-making because of accelerated regulatory approvals and rare tumor types.

 The implementation of the US FDA Safety and Innovation Act in 2012 allowed for accelerated approval of select drugs that treat serious or life-threatening disease if 1 or more clinically significant end point is considerably better than that of existing drugs.

 The US FDA Safety and Innovation Act, enacted in 2012, facilitates the development and review of new drugs for serious or life-threatening diseases, allowing promising new drugs to receive accelerated approval based on surrogate markers of disease, while requiring that efficacy testing continues post-launch to confirm the initial results.

 Results from an analysis of oncology drugs approved from 2012 to 2017 showed that drugs with a breakthrough designation obtained their first FDA approval significantly sooner after their investigational new drug application (NDA) than drugs without breakthrough designation status (median, 5.2 years vs 7.1 years; 

 Although accelerated approvals make innovative therapies available to patients faster, payers are challenged with making decisions about coverage of breakthrough oncology drugs based on limited clinical data.

Therefore, real-world evidence (RWE), defined by the Federal Food, Drug, and Cosmetic Act as “data regarding the usage, or the potential benefits or risks, of a drug derived from sources other than traditional clinical trials,” is often used to fill these knowledge gaps and guide oncology formulary decision-making.

 Sources of real-world data (RWD) include electronic health records, medical claims and billing data, product and disease registries, and patient sources such as information stored on mobile devices.

However, information on how RWE is used by payers to make oncology formulary decisions is unclear.

 Recent studies investigated how US payers use RWE to guide health care decision-making in oncology, which may help payers identify pertinent RWE and help manufacturers design relevant RWE studies.

Payer Perceptions of the Use of Real-World Evidence in Oncology-Based Decision-Making

An interdisciplinary steering committee of payers, academic experts, and experts in health economics and outcomes research developed a survey to assess payer understanding about the use of RWE in decision-making for oncology products coverage.

 The content of the survey was based on literature reviews and current knowledge of RWE use in oncology decision-making. A pilot survey was sent to 5 payers, then reviewed and revised based on their feedback.

The survey used in the study addressed 4 major areas on the use of RWE in decision-making for oncology coverage and was composed of 18 questions. Of these, 7 explored the types of decisions in oncology that can be guided by RWE and ways RWE is used by organizations in formulary decision-making, 5 covered internal and external sources of RWE, 2 addressed barriers to performing real-world analyses, and 4 questioned the experience and interest of payers in using RWE to guide outcomes-based contracting.

 The survey was distributed to a panel of 221 payers, which included Academy of Managed Care Pharmacy (AMCP)members who were pharmacy leaders involved in medication product evaluation or utilization management.

 The survey was also distributed to an additional 10 US payers who were not affiliated with this panel and agreed to participate in a live panel discussion in March 2020 about the results.

Survey results were primarily presented as response frequencies, and dummy variables (yes/no) were used for responses on the multiple response questions.

 test or Fisher’s exact tests (large [at least 1 million members] or small [less than 1 million members]), plan type, and scope of service (regional or national) were used to evaluate categorical responses according to plan size. Statistical significance was set at 2-tailed 

Of the 231 survey recipients (221 AMCP plus 10 payer panelists), 106 completed the survey (response rate, 45.9%).

 Of the 99 respondents who provided demographic information, the largest proportions represented managed care organizations (MCOs; 47.5%) or pharmacy benefit managers (PBMs; 37.4%), with a similar proportion of regional (49.5%) and national (47.5%) plans represented.

 As for payer type, 64.6% of respondents represented commercial and government organizations, 19.2% represented government only, and 16.2% represented commercial only.

 The majority (89.9%) of respondents were pharmacists, with the most commonly reported roles being pharmacy administrators (55.6%) and community or clinical pharmacists (30.3%).

Of the respondents, 84.9% use RWE for making formulary decisions in oncology.

 According to these payers’ responses, RWE was most useful in comparing the effectiveness of interventions when no head-to-head trials were available and in providing supportive evidence for guidelines (eg, National Comprehensive Cancer Network [NCCN] guidelines).

 In the 10-payer panel discussion of the survey findings, the panelists concluded that interest in the use of RWE for decision-making continues to grow, particularly for comparing effectiveness between drugs, determining formulary placement, and guiding financial decisions.

 Along with RCT data and feedback from experts, RWE is also used by the Pharmacy and Therapeutics Committee to determine initial placement and coverage. Following product launch, RWE associated with a product can help to refine related decisions and therapy sequence.

A majority of respondents reported using RWE from claims data (79.2%), medical records data (68.9%), and data from prospective cohort studies (60.4%).

 PBMs and MCOs reported using similar types of data, although MCOs were more likely than PBMs to use prospective cohort studies (66.0% vs 56.8%) and patient registries (42.6% vs 27.0%).

 Nearly half of respondents stated that their plans perform their own real-world studies to guide decisions for oncology coverage (48.1% for large plans; 40.0% for small plans). Results are often used to garner data for budget-impact models or cost-effectiveness models to confirm RCT results within a payer’s own patient populations, and to support off-label use.

The panelists identified high availability as the reason most payers use pharmacy and medical claims for RWE, although panelists expect the use of electronic health records to continue to increase.

 They noted that PBMs, with limited access to medical data, often need to rely on pharmacy claims data, which typically provide insight into only dose strength and therapy duration.

Limited resources and personnel was reported as the driving for plans that had difficulty conducting their own real-world studies; this was a greater concern for small plans than for large plans (91.1% vs 70.4%, respectively; 

 Other barriers commonly reported were insufficient analytic capabilities (55.6% of large plans and 57.8% of small plans) and a lack of appropriate in-house data (46.3% of large plans and 48.9% of small plans).

 Panelists noted that in-house production of RWE may be too costly for some payers. Some payers were concerned about whether data produced from in-house studies would be meaningful because of the small number of eligible patients for some of the innovative oncology drugs.

Interest in using RWE to guide outcomes-based contracting was reported by the majority of respondents—76.5% were moderately, very, or extremely interested; 17.6% were somewhat interested.

 However, only 17.2% of organizations stated that they had experience in outcomes-based contracting in oncology, with the most commonly reported barriers including challenges around the definition of outcomes to be measured and monitored over time (76.5%), complexity of outcomes-based agreements (74.5%), costs of implementation (48.0%), small size of the target population (48.0%), and lack of personnel (34.3%).

 Panelists cited uncertainty about how to identify and use data for contracts; they noted that outcomes and end points can be affected by factors including adherence, proxy measures, comorbidities, hospitalizations, and relapses.

 Furthermore, data access may be limited in some health plans, making measurement and tracking of the necessary data difficult.

US payers perceive that RWE plays an important role in oncology decision-making, especially for innovative drugs indicated for niche populations where evidence from RCTs is often limited by accelerated approvals.

 However, the associated expense and limited resources make it difficult to conduct internal RWE studies, and many payers gain insights from RWE published in NCCN and other guidelines.

 Additionally, the timing of RWE availability is an issue for payers when making oncology formulary and contracting decisions because RWE that compares therapy effectiveness is often not available prior to product launch, which is when it would help payers assess the role of new oncology therapies for the patients who are most likely to respond.

 Additional types of RWE generated through clinical trials, such as cost and prevalence of a disease, approaches to treatment, and adverse events, may help guide optimal placement of a new product; however, the panelists stated that they rarely see this information. Finally, the increased use of RWE highlights the need to educate payers on the appropriate use of RWE for making oncology formulary decisions.

Payer Perceptions on the Use of Economic Models in Oncology Decision-Making

An interdisciplinary steering committee of payers, academics, and industry health economics and outcomes research experts created a survey based on literature reviews and current knowledge of the use of economic models in oncology decision-making.

 The survey was composed of 13 questions about payer perceptions and use of economic models, including questions about payer interest in using different types of economic models (n = 1), experience of respondent organizations with using economic models in oncology decision-making (n = 8), barriers to using economic models (n = 2), and interest in partnering with manufacturers to develop or validate an economic model in oncology using plan data from the organization (n = 2).

The survey was distributed to a panel of 221 payers, which included AMCP members who were pharmacy leaders involved in medication product evaluation or utilization management.

 The survey was also distributed to an additional 10 US payers who were not affiliated with this panel and agreed to participate in a live panel discussion about the results.

Survey results were presented as frequencies of responses, coded as “yes” or “no” for multiple-response questions.

 Categorical responses were evaluated using Pearson’s χ

 test or Fisher’s exact test according to plan size, plan type, and scope of service (regional or national). Statistical significance was set at 2-tailed 

Of the 231 survey recipients (221 AMCP plus 10 payer panelists), 106 completed the survey (response rate, 45.9%).6 MCOs (47.5%) and PBMs (37.4%) were most commonly represented, with relatively similar proportions of regional (49.5%) and national (47.5%) plans.

 Commercial and government payers represented the majority (64.6%) of respondents.

 The majority (89.9%) of respondents were pharmacists, with pharmacy administrators (55.6%) and community or clinical pharmacists (30.3%) as the most commonly reported roles.

Current Payer Perceptions and the Use of Economic Models

Compared with small plans (< 1 million lives covered), a higher proportion of large plans (≥ 1 million lives covered) reported reviewing economic models related to oncology products within the last 2 years (57.0% vs 35.6%; 

 = .043) and having individuals with expertise in evaluating economic models in oncology (55.6% vs 31.1%; 

 Reasons given for not reviewing economic models included lack of available models at the time of product review (44.1%), potential bias (38.2%), lack of personnel or training (33.3%), lack of transparency (31.4%), and excessively complicated models (28.4%).

Payers ranked cost-effectiveness models (CEMs; 85.3% moderate/most interest) and budget-impact models (BIMs; 80.4% moderate/most interest) higher than cost calculators (34.3% moderate/most interest), and the majority of respondents reported sometimes, often, or always using CEMs (66.7%) and BIMs (62.7%) to inform oncology formulary decisions.

 Approximately two-thirds of payers (68.6%) stated that economic models can help inform decisions at least a moderate amount when considering therapies with a similar efficacy and safety profile.

After reviewing a manufacturer-sponsored economic model, 43.1% of payers reported using internal data to perform additional analyses for oncology decision-making, 9.8% reported using the manufacturer-sponsored model without their own internal data to validate the model, and 31.4% reported 

 using a manufacturer-sponsored model to make coverage decisions.

In the panel discussion, panelists confirmed their interest in economic models and stated that they valued data on standard of care, the anticipated role of the drug in therapy, estimated target population, costs of adverse events and hospitalizations in clinical trials, and total cost of care.

 Although they noted that data are often limited at the time of initial formulary decision-making (particularly for drugs with accelerated approval) and can change when the product enters the market, the panelists said that early economic data available from clinical trials are valuable, particularly if the data factor in total cost of care.

 Additionally, the panelists said that progression-free survival and overall survival are highly relevant end points to include in economic models, may be helpful for outcomes-based contracting, and should be included in follow-up RWE studies.

 The panelists added that real-world economic data are useful during the postlaunch stage to verify initial decisions and inform potential changes in coverage decisions or treatment pathways.

 However, the panelists noted concerns around prevalence in trial populations vs the plan’s population, lack of accurate and inclusive estimates for all components of total care costs, and treatment patterns in a trial that do not reflect those of physicians in the real world.

Economic models can provide valuable supplemental information to inform formulary decision-making in oncology, but the models are often lacking at the time of product review.

Results from the survey and panel discussion showed that payers consider economic models to provide valuable information to add to traditional trial data when making oncology coverage decisions.

 The Institute for Clinical and Economic Review (ICER) provides information on the economic impact of formulary decisions that is generally available near the approval date for a product, which is often too late for consideration by payers.

 Therefore, access to economic data that contribute to the estimated total cost of care as early as possible enables payers to evaluate predicted costs with assumptions about patient population, rates of adverse events and hospitalizations, and medical costs.

 However, information needed for initial formulary review is often not available in the public domain or requires extensive searching by payers, and liaisons with manufacturers can improve understanding of early information on the product and preliminary economic data through guidance on preapproval information exchange.

However, only about 50% of payers surveyed reported having experience reviewing economic models or performing their own economic studies in oncology, highlighting an opportunity to improve education on optimizing creation, validation, and use of the models for decision-making, particularly for small health plans.

 Possible strategies to address payer concerns about assumptions and transparency in economic models include involving payers in the development to ensure that the parameters chosen by manufacturers are relevant, providing references for the selected assumptions, includingsensitivity analyses, and explaining the origins of, rationale for, and changes in, assumptions over time.

Key limitations of the study included the underrepresentation of integrated delivery systems in the survey and decision-makers other than pharmacy directors of health plans.

 Additionally, the limited length of the survey did not allow for collection of additional information to explain answers, and respondents were not required to answer all questions, which may have introduced a bias.

 Furthermore, the study may have included multiple respondents from the same organization, and specific information on the respondents was unavailable because the results were deidentified.

 Requests for preapproval RWE from manufacturers may help to bridge this gap, and payer education about the applications and limitations of economic models in oncology decision-making is beneficial to optimize the use of these models.

Payer Perceptions on the Use of Patient-Reported Outcomes in Oncology Decision-Making

An interdisciplinary steering committee that included payers, experts in academic and industry health economics and outcomes research, and a measurement specialist created a survey to assess understanding on the value and use of patient-reported outcomes (PRO) evidence in oncology decision-making.

 The survey included 10 questions that addressed payer perceptions and use of PRO evidence: questions about the usefulness of PRO evidence from clinical trials and real-world studies (n = 2), the effect on formulary decisions of PRO evidence from clinical oncology trials or RWE (n = 4), the PRO version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) (n = 2), the use of PRO evidence from clinical trials to provide context for safety data (n = 1), and the perceived role of PRO evidence for measuring value in innovative or value-based agreements (n = 1).

The survey was distributed online through the AMCP Market Insights program to a panel of 221 payers, which included AMCP members who were pharmacy leaders involved in evaluation or utilization management of medication products, as well as 10 additional US payers who agreed to participate in a live panel discussion about the survey results.

Results were presented primarily as response frequencies, with multiple-response questions coded as “yes” or “no.”7 Categorical responses were evaluated using Pearson’s χ

 test or Fisher exact test according to plan size, plan type, and scope of service (regional vs national), and statistical significance was set at 2-tailed 

 MCOs (47.5%) and PBMs (37.4%) were most commonly represented, and a similar proportion of regional (49.5%) and national (47.5%) plans were represented.

 The majority (64.6%) of respondents’ organizations were private payer plans covering commercial and government lives, and all health plan sizes were well represented.

 Most respondents (89.9%) were pharmacists, with pharmacy administrators (55.6%) and pharmacists/clinical pharmacists (30.3%) being the most commonly reported roles.

Current Payer Perception and Use of PRO Evidence

The majority of respondents stated that PRO evidence is useful for formulary decision-making, with similar responses between small (< 1 million covered lives) and large health plans (≥ 1 million covered lives; 60.0% vs 57.4%; 

 = .794). Respondents also stated that PRO evidence from real-world studies is useful (small plans, 57.8% vs large plans, 51.9%; 

 Nearly half of payers stated that a lack of PRO evidence from oncology clinical trials would negatively affect formulary review at least somewhat (48.2% of large plans vs 42.2% of small plans; 

 Respondents considered PRO evidence to be useful for providing additional safety-related context (78.2%), oncology formulary decision-making when selecting between therapies (34.3%), and measured value for innovative or value-based agreements (51.5%).

Panelists in the 10-member panel discussion agreed with the survey findings and added that PRO may help with development of treatment pathways to address health-related quality of life (HRQOL)in oncology, shared decision support, care management designs, and dialogues between provider and patient.

 They added that PRO evidence may address patient functionality during progression-free survival and development of post-progression clinical pathways that identify adverse effects affecting concepts measured in PROs.

The panelists said that PRO evidence from real-world studies was more generalizable and realistic than data from clinical trials, but noted that RWD are often unavailable at the time of initial formulary decision-making, particularly with accelerated drug approvals.

 They added that PRO evidence from real-world, postlaunch studies is valuable but may be difficult for payers to find and incorporate into future decision reviews because they may have insufficient resources to thoroughly review literature post launch.

 PRO evidence in the package insert is also more likely to be easy to access and accepted by payers than PRO evidence from other sources.

Total cost of care was also identified as a concern, and the panelists stated that improved knowledge of PRO evidence may affect treatment decisions and models of care delivery used by network providers, which ultimately influence the total cost of care.

 Although interest in using PRO evidence for outcomes-based or value-based contracts was expressed by many panelists, they preferred more objective measures and could not identify specific examples in which PRO data were used for outcomes contracts.

Although payers may have a bias against PRO evidence based on a perception of its subjectivity, the effect of PRO on total cost of care and utilization of health care resources is increasingly recognized by the FDA and value frameworks from multiple organizations, including ICER and the American Society of Clinical Oncology.

 PRO evidence related to HRQOL may be particularly relevant in symptomatic late-stage or metastatic oncology; therefore, real-world studies that include HRQOL-related PROs may be of interest to payers and providers.

 However, better methods are needed for collecting and reporting PRO evidence that is meaningful and actionable for payer decision-making, as providers are less likely to record patient-reported measures (eg, pain or fatigue) in the patient’s medical chart than objective measures that can be tracked over time (eg, blood cell count).

 Technologic advances for gathering and measuring PRO data (eg, from member digital apps and online portals) have been developed, and further developments should focus on increasing the usefulness and actionability of gathering and reporting PRO data.

 The PRO-CTCAE measurement system, designed to evaluate patient-reported toxicity symptoms for participants in oncology clinical trials, is 1 example of a tool for tracking PROs, but 81.1% of payers in the current study stated that their level of awareness of PRO-CTCAE was low or very low, and 67.7% stated that they did not have sufficient information to answer questions about its usefulness.

 Therefore, education of payers about the value of PRO-CTCAE and other PRO measurement systems is needed.

Although the panelists were unfamiliar with contracts that were based on PRO measures, PRO evidence also has the potential to be used more fully in value-based contracting with manufacturers.

 CMS and some commercial payers have been investigating ways to include PRO evidence in payment and accountability programs (although these are still not widely used in oncology), and widespread utility of PRO evidence in outcomes- or value-based contracts requires continued development and enhancement of standardized and relevant PRO measures.

Although MCOs, PBMs, and large payers were well represented, integrated delivery systems accounted for only 10.1% of respondents, and decision-makers other than health plan pharmacy directors (eg, medical directors) were not surveyed and may have different perspectives.

 Additionally, perspectives from these US-based payers may not be generalizable to those in other countries, and the opinions of participants in the panel discussions may not represent those of all respondents.

PRO evidence is a complementary addition to traditional efficacy and safety data from clinical trials used by payers when making formulary decisions, particularly in oncology where HRQOL carries relatively high importance and traditional evidence is limited because of the large number of accelerated approvals.

 PRO evidence has expanded from that limitation of patient-physician interactions to a valuable information source that payers can use to assess patient experience and HRQOL in their plan members undergoing oncology treatment.

Real-World Evidence in Support of Oncology Product Registration: a Systematic Review of New Drug Application and Biologics License Application Approvals From 2015-2020

A systematic review of NDA and biologics license application (BLA) approvals was performed to identify oncology products from the Center for Drug Evaluation and Research (CDER) calendar year approvals, CDER Drug approvals by month, and the Center for Biologics Evaluation and Research biological approvals by year; the corresponding FDA review documents were assessed for RWE.

 Supplemental NDA and BLA approvals were also identified in a similar manner, with CDER efficacy supplement approvals and hematology/oncology approvals and safety notifications used as additional sources.

 Automated text extraction was used to extract text into a consolidated database and to search for keywords related to RWE. After drug approvals with RWE were identified, pivotal information related to the trial, application, and RWE were extracted from the FDA review documents.

Eleven of the 133 approvals for oncology NDAs and BLAs included RWE supporting efficacy.

 The median time from investigational new drug submission to approval was 5.5 years (range, 2.8-8.4).

 The NDAs or BLAs with RWE were for avelumab, axicabtagene ciloleucel, entrectinib, erdafitinib, polatuzumab vedotin-piiq, selinexor, avapritinib, capmatinib, tafasitamab, and tazemetostat.

 Of 573 oncology supplemental NDAs and BLAs identified, 249 were for new oncology indications, and 2 (0.8%) of these included RWE in support of efficacy (for blinatumomab and palbociclib).

All 13 products with RWE for efficacy were FDA approved after enactment of the 21st Century Cures Act, with 11 applications that received orphan drug designation, 12 that received priority review, 9 that have been designated as breakthrough therapies, 6 that received fast-track designation, and 3 that received all of these designations.

 The 4 products that received full approvals had designations of orphan disease, breakthrough therapy, priority review, no advisory committee meeting called by the FDA, and high level of primary efficacy.

RWE was used for contextualization in 8 applications, in comparison with clinical trial results in 3 applications, and both purposes in 2 applications.

 Chart review from clinical sites was the most common source of RWD, and overall response rate (ORR) was the most common end point.

 RWE for 3 therapies used a real-world population that matched the eligibility criteria to the trial, and of the 11 new approvals, RWE sample sizes were more than 50% larger than in the pivotal trial in 4 (36%), similar in 3 (27%), and more than 50% smaller in 4 (36%) cases.

FDA-identified sources of bias related to the use of historical controls and RWD included selection bias, residual or unobserved confounding, different methods of outcome assessment and frequency of measures compared with the pivotal trials, lack of comparability between trial participants and external controls, incorrect classification of outcomes, and inadequate statistical adjustment of differences between comparator groups.

RWE may help accelerate the drug development process, but it has inherent limitations, which prompted the FDA to make several recommendations for improving its quality. These recommendations include early engagement to ensure appropriateness of the data source and decide the purpose of this RWE study, development of a prespecified study protocol to improve transparency, and selection of the RWE population that is comparable with that of the pivotal trial.8 Additionally, the FDA encouraged the use of concordance analysis to address the discrepancy between using real-world tumor assessment and Response Evaluation Criteria in Solid Tumors [RECIST] criteria to evaluate ORR, descriptions of ways to minimize confounding (eg, appropriate index date and reduction in missing values), and quantitative bias analysis to describe effects of unmeasured confounding.

The FDA tended to have fewer critiques on RWE quality when the pivotal trials had a high ORR and the RWE had a low ORR; however, the magnitude of primary efficacy in the pivotal trial was associated with receiving full approval (vs accelerated approval), with ORRs of more than 70% and median duration of response (DOR) of at least 9 months observed in these trials.

 Accelerated approvals had a wide range of ORRs (13% to ≤ 70%) and median DOR (4 months to not reached).

 However, an ORR of greater than 70% should not be interpreted as the benchmark for full approval.

The Role of Real-World Evidence, Patient-Reported Outcomes, and Economic Models in Oncology Formulary Decisions and the Role of Real-World Evidence in Oncology Product Registration