Donna M. Fick, RN, PhD

 To provide recommendations that will improve approaches to measuring the value of new medical technologies to patients.

 Informed discussion by experts after literature review.

 A working group was formed, and participants discussed how value frameworks should incorporate key features important to patients in evaluating new medical technologies, particularly for chronic diseases.

 The working group suggests that new value frameworks should integrate real-world evidence to complement randomized controlled trials, incorporate the ways in which real-world behavior mediates outcomes, and explicitly discuss how therapies affect real-world equity and disparities in care.

 Collective stakeholders that include key decision makers within our healthcare system need to recognize the importance of implementing real-world evidence and devote resources to further research into the chronic disease areas in which the impact of human behavior is amplified by the duration of disease and treatment.

Steps have been taken to develop critical evaluations of drugs with the creation of value frameworks from organizations such as the Institute for Clinical and Economic Review, American Society of Clinical Oncology, and Memorial Sloan Kettering, but more work can and should be done to further refine these approaches. The Value FORWARD (Frameworks, Outcomes, and Real-World Adherence in Chronic Disease) Committee puts forth the following recommendations:

Value frameworks should incorporate real-world effectiveness to complement randomized controlled trial efficacy.

Value assessments must explicitly incorporate the ways in which real-world behavior mediates outcomes.

Value assessments should explicitly discuss how therapies affect real-world equity and disparities in care.

Policy makers, healthcare payers, patients, employers, and other stakeholders continue to fret about the rising cost of new medical technologies. Many scientific organizations have responded by developing new tools to assess the value of these technologies, commonly called “value frameworks.” For example, the American Society of Clinical Oncology (ASCO),

 and the Institute for Clinical and Economic Review (ICER) have offered alternative approaches to assessing the value of technologies. In principle, better information about value can help moderate growth in the cost of technologies that do not provide sufficient benefit to patients.

Yet there remains a gap between the goal of measuring value conferred to patients and the available evidence for new medical technologies. Randomized controlled trial (RCT) evidence generated for regulatory approval often incorrectly measures or fails to measure aspects of value that are key to patients. Some of these gaps and limitations are widely understood,

 but others have received too little attention, such as the role of patient behavior in mediating clinical outcomes.

 For example, evidence generated by RCTs often reflects outcomes achieved with motivated patients treated in environments in which individual patient behaviors that are critical to driving real-world outcomes, including adherence to medications, are controlled. As a result, interventions that improve adherence may not demonstrate large efficacy benefits in RCTs but may demonstrate larger benefits in real-world settings in which adherence is typically lower.

A working group was formed in partnership with Precision Health Economics and Intarcia Therapeutics in 2017 to make recommendations on how value frameworks should close the gap between evidence and patient behavior when evaluating new medical technologies, particularly for chronic diseases. Our working group recommended that these value frameworks enhance attention to (1) real-world evidence (RWE), (2) the role of adherence, and (3) nonclinical components of the patient experience.

Unlike healthcare systems in countries such as England, Germany, and Australia, where health technology assessment processes are well established,

 the US healthcare system does not have a formalized approach for defining or measuring value. Private organizations have stepped in to fill this gap but offer varying perspectives and approaches. For example, ASCO has released a value framework meant to aid physician communication of cancer treatment options with patients.

 ICER evaluates the cost-effectiveness of drug therapies from a payer and health system perspective.

 Memorial Sloan Kettering’s drug evaluation framework, the Drug Abacus, which evaluates cancer drugs, enables users to input their own preferences on different dimensions of value.

 These value frameworks differ in the perspectives that they use and in the attributes of value that they include in their evaluations. As a result, different value frameworks evaluating the same drug may have different conclusions about what may be considered reasonable value.

Recommendation 1: Value Frameworks Should Incorporate Real-World Effectiveness to Complement RCT Efficacy

Value frameworks have tended to emphasize evidence generated in RCTs, excluding or downplaying the use of RWE. Clinical trial evidence is emphasized as the gold standard for health benefit, although recently, there has been a growing awareness and movement to improve databases and infrastructure for RWE.

Clinical trial populations often fail to mirror real-world populations. This external validity problem is already well documented.

 Clinical trials are conducted in controlled environments where the structure of the environment lends itself to optimal rates of treatment adherence.

 More recently, some prominent researchers have argued that RCTs with smaller samples, often seen in breakthrough treatments with accelerated market approval, suffer from internal validity problems such as misalignment between the characteristics of subjects in the treatment and comparator groups.

RWE can help address these limitations. To be sure, it also suffers from limitations related to internal validity (eg, persons using novel therapies in the real world are often systematically different from their peers using older drugs at the same time). Methodological solutions exist to control for some of these confounders, and more generally, care must be taken to ensure the highest degree of internal validity in RWE. However, RWE is more than just a second-best alternative to RCT evidence; it provides inferences that are necessarily applicable to real-world populations. One major area of behavior around treatment that can be captured more realistically with RWE is adherence to treatment regimens.

 Our committee recommends explicit evaluation of RWE as a potential input into value assessments.

Recommendation 2: Value Assessments Must Explicitly Incorporate the Way Real-World Behavior Mediates Outcomes

As noted earlier, a critical moderator of health outcomes outside of clinical trials is adherence. At present, adherence is not explicitly included in any value framework or in the wider body of literature of cost-effectiveness models and budget impact models. Full adherence is typically “included” as an underlying assumption in these models and frameworks, as it is implicitly a part of clinical efficacy.

The need to do this is intuitive, particularly when comparing treatments in the same therapeutic area that possess different delivery mechanisms (eg, daily oral tablets vs weekly injections) or different adverse event profiles. An evaluation of a treatment that has a positive effect on behavior and encourages a higher rate of adherence than its comparator ought to consider the clinical consequences of better adherence (eg, an improvement in patient outcomes or achievement of treatment goals).

Adherence can also be considered an element for differentiating high-value and low-value care. Treatments that encourage improved adherence will result in less waste, because prescribed and dispensed treatments will actually be used. The importance of reducing this waste through improved adherence cannot be underestimated. Past estimates have found that more than $100 billion annually of avoidable costs are a result of medication nonadherence.

 Explicitly incorporating adherence would enable value frameworks to better identify high-value compared with low-value care.

The magnitude of this underuse has not been well understood, in part because our current adherence measures are not sufficiently nuanced to capture the full spectrum of behavior. Current measures of adherence do not infer medication consumption and are typically used as a binary measure of “good adherence” at or above 80% or “bad adherence” below 80%.

 This threshold implies that 80% is sufficient to achieve optimal outcomes, which has not been demonstrated across all diseases. Additionally, these measures primarily capture prescription behavior (ie, whether the prescription has been submitted/picked up), not whether the person has taken the treatment.

 Further, these measures do not capture primary nonadherence for persons who do not fill their initial prescription.

Starting with chronic diseases, we should seek to map adherence patterns and link those adherence patterns to outcomes. Work that has already started in this area—for example, in individuals with schizophrenia

—has demonstrated that adherence is not binary, with nuanced adherence patterns having real impacts on patient outcomes. We should seek to build on this work and develop studies that capture the value of adherence for health and quality-of-life outcomes so that adherence can be explicitly included as a separate value component in economic valuations of treatment.

Recommendation 3: Value Assessments Should Explicitly Discuss How Therapies Affect Real-World Equity and Disparities in Care

A critical aspect of treatment in the real world largely neglected by existing value frameworks is the impact on disparities in care. Previous research has demonstrated that nonclinical characteristics, such as education level, can affect adherence behaviors, which ultimately affect outcomes.

 Socially disadvantaged individuals are more likely to experience disease, at greater severity and with a higher likelihood of adverse effects.

 Combined with poorer adherence behaviors, disparities in healthcare can be compounded in chronic disease, where long-term consistent adherence to treatment is necessary for achieving positive outcomes. Explicitly evaluating how a new treatment can affect adherence and otherwise improve existing disparities in care would be an important element in determining the social impacts of therapeutic use.

A relatively strong body of literature already exists that articulates inequity in healthcare and some of the causes of those disparities. However, more research is required to develop methodologies to quantify and incorporate equity impacts into value assessments. It will be necessary to extend these equity concerns to additional nonclinical aspects of care, such as productivity and caregiver burden. The increased burden on productivity and caregivers also disproportionately affects disadvantaged individuals. Patients and providers have emphasized the importance of these aspects of care in making treatment decisions, but insufficient research exists to enable these aspects to play a large decision-making role. Further research and studies that take innovative approaches to, first, identifying the relevant nonclinical aspects of the patient experience and, second, measuring and quantifying these aspects are highly important to furthering our evidence base for treatment innovations and overall improving healthcare decision making.

A wide range of stakeholders are demanding better evidence of value. Value assessments and frameworks must adapt by making better use of real-world data and considering how individual behavior causes real-world outcomes to diverge from those seen in clinical trials. Our recommendations can help align value frameworks with value to individuals in the real world. Although we have already taken good steps forward to develop critical evaluations of drugs, more work can and should be done to further refine these approaches. Minimally, further efforts should be made to incorporate RWE to more accurately reflect the performance of new innovations by including adherence and measuring patient outcomes in the usual settings of patient care. To do so, the collective stakeholders that comprise key decision makers of our healthcare system need to devote resources to further research in chronic diseases where the impact of human behavior is amplified by the duration of disease and treatment.

The authors would like to thank Tiffany Shih for administrative and editorial support.

 Harvard Medical School (ABJ), Boston, MA; Precision Health Economics (JWC, LY), Los Angeles, CA; University of California, San Francisco (WMA), San Francisco, CA; Health Intelligence Partners (JB), Chicago, IL; University of Illinois, Chicago (WB), Chicago, IL; University of Michigan (AMF), Ann Arbor, MI; Pennsylvania State University (DMF), University Park, PA; Intarcia Therapeutics (DF, JY, BS, KY-I), Boston, MA; Dentons US LLP (RK), Washington, DC; University of Southern California (DNL), Los Angeles, CA; Tufts Medical Center (PJN), Boston, MA; The Brookings Institution (KP), Washington, DC.

 Funding for the Value FORWARD Committee was provided by Intarcia Therapeutics. Support was provided by the Office of the Director, National Institutes of Health (1DP5OD017897-01, Dr Jena). The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.

 Dr Jena reports receiving consulting fees unrelated to this work from Amgen, Analysis Group, AstraZeneca, Biogen, Bristol Myers Squibb, Celgene, Eli Lilly, Hill Rom Services, Novartis, Pfizer, Precision Health Economics, Sanofi Aventis, Tesaro, and Vertex Pharmaceuticals. Ms Chou is an employee of Precision Health Economics, a healthcare consultancy with clients in the life sciences industry, and holds equity in Precision Medicine Group, parent company of Precision Health Economics. Ms Yoon is a previous employee of Precision Health Economics. Dr Aubry attended paid advisory boards and meetings for Intarcia on April 8, 2017, and May 16, 2017, and serves as a consultant to Precision Health Economics. Dr Burton has consulted for and received lecture fees from Amgen. Dr Fendrick has consulted for AbbVie, Amgen, Bayer, Centivo, Community Oncology Association, EmblemHealth, Exact Sciences, Freedman Health, Health at Scale Technologies, Lilly, Mallinckrodt, MedZed, Merck, Risalto, Sempre Health, State of Minnesota, Takeda, Wellth, and Zansors; has performed research for the Agency for Healthcare Research and Quality, Boehringer-Ingelheim, Gary and Mary West Health Policy Center, Laura & John Arnold Foundation, National Pharmaceutical Council, Patient-Centered Outcomes Research Institute, PhRMA, Robert Wood Johnson Foundation, and State of Michigan/CMS; and holds outside positions as co-editor-in-chief of 

, member of the Medicare Evidence Development & Coverage Advisory Committee, and partner in V-BID Health, LLC. Mr Franklin is employed by Intarcia Therapeutics, Inc. Dr Lakdawalla is a consultant to Precision Health Economics and holds equity in its parent company, Precision Medicine Group. Dr Neumann reports participating on advisory boards or consulting for AbbVie, Avexis, Bayer, Bluebird, Celgene, DePuy Synthes, GSK, Merck, Novartis, Novo Nordisk, Pacira, Paratek, Precision Health Economics, Vertex, Boston Health Economics, and Congressional Budget Office. Drs Yee, Sakurada, and Yu-Isenberg are former employees of Intarcia Therapeutics. Drs Sakurada and Yu-Isenberg have attended Academy of Managed Care Pharmacy, American Diabetes Association, Asembia, International Society for Pharmacoeconomics and Outcomes Research, and National Association of Managed Care Physicians conferences when under employment of Intarcia Therapeutics. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (ABJ, JWC, WMA, JB, WB, DMF, RK, DNL, KP, JY, BS, KY-I); acquisition of data (LY, DF, KY-I); analysis and interpretation of data (ABJ, WMA, JB, AMF, DMF, RK, DNL, PJN, KP, BS, KY-I); drafting of the manuscript (ABJ, JWC, LY, WB, AMF, PJN, KP, KY-I); critical revision of the manuscript for important intellectual content (JWC, WMA, JB, WB, AMF, DMF, DF, RK, DNL, PJN, KP, JY, BS, KY-I); provision of patients or study materials (KY-I); obtaining funding (DF, KY-I); administrative, technical, or logistic support (JWC, LY); and supervision (DNL, KY-I).

 Anupam B. Jena, MD, PhD, 180 Longwood Ave, Boston, MA 20115. Email: jena@hcp.med.harvard.edu.

1. Schnipper LE, Davidson NE, Wollins DS, et al. Updating the American Society of Clinical Oncology value framework: revisions and reflections in response to comments received. 

. 2016;34(24):2925-2934. doi: 10.1200/JCO.2016.68.2518.

2. Drug Abacus. Drug Pricing Lab website. drugpricinglab.org/tools/drug-abacus. Accessed October 1, 2017.

Best Care at Lower Cost: The Path to Continuously Learning Health Care in America

. Washington, DC: National Academies Press; 2013.

4. Nordon C, Karcher H, Groenwold RH, et al. The “efficacy-effectiveness gap”: historical background and current conceptualization. 

. 2016;19(1):75-81. doi: 10.1016/j.jval.2015.09.2938.

5. Maclean JR, Pfister M, Zhou Z, Roy A, Tuomari VA, Heifets M. Quantifying the impact of nonadherence patterns on exposure to oral immunosuppressants. 

. 2011;7:149-156. doi: 10.2147/TCRM.S16870.

6. Sun M, Larcher A, Karakiewicz PI. Optimal first-line and second-line treatments for metastatic renal cell carcinoma: current evidence. 

. 2014;7:401-407. doi: 10.2147/IJNRD.S48496.

7. Oortwijn W, Broos P, Vondeling H, Banta D, Todorova L. Mapping of health technology assessment in selected countries. 

. 2013;29(4):424-434. doi: 10.1017/S0266462313000469.

8. ICER Value Assessment Framework. Institute for Clinical and Economic Review website. icer-review.org/methodology/icers-methods/icer-value-assessment-framework. Accessed October 1, 2017.

9. Bothwell LE, Greene JA, Podolsky SH, Jones DS. Assessing the gold standard—lessons from the history of RCTs. 

. 2016;374(22):2175-2181. doi: 10.1056/NEJMms1604593.

10. Frieden TR. Evidence for health decision making—beyond randomized, controlled trials. 

. 2017;377(5):465-475. doi: 10.1056/NEJMra1614394.

11. Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidence—what is it and what can it tell us? 

. 2016;375(23):2293-2297. doi: 10.1056/NEJMsb1609216.

12. van Onzenoort HA, Menger FE, Neef C, et al. Participation in a clinical trial enhances adherence and persistence to treatment: a retrospective cohort study. 

. 2011;58(4):573-578. doi: 10.1161/HYPERTENSIONAHA.111.171074.

13. Deaton A, Cartwright N. Understanding and misunderstanding randomized controlled trials [NBER working paper no. 22595]

 National Bureau of Economic Research website. nber.org/papers/w22595. Published September 2016. Updated October 2017. Accessed October 1, 2017.

14. Cohen AT, Goto S, Schreiber K, Torp-Pedersen C. Why do we need observational studies of everyday patients in the real-life setting? 

. 2015;17(suppl D):D2-D8. doi: 10.1093/eurheartj/suv035.

15. Burton WN, Chen CY, Schultz AB, Edington DW. The association between achieving low-density lipoprotein cholesterol (LDL-C) goal and statin treatment in an employee population. 

. 2010;13(1):1-8. doi: 10.1089/pop.2009.0020.

Articles

New value frameworks should incorporate real-world evidence that reflects patient treatment behavior, adherence to medication, and equity concerns arising from disparities in care.

Donna M. Fick, RN, PhD

Articles

Understanding and Improving Value Frameworks With Real-World Patient Outcomes

Comparison of Two Systems for Clinical Practice Profiling in Diabetic Care: Medical Records Versus Claims and Administrative Data