Jonathan Ratner, PhD

Randomized controlled trials (RCTs) reflect priorities established by regulators. Recently, pragmatic clinical trials (PCTs) have begun to attract interest. Unlike RCTs, PCTs aim to better inform post-regulatory decision making by using head-to-head comparisons of alternative treatments, diverse patient populations, and outcomes meaningful to patients, prescribers, and payers.

 To describe how US insurers and public payers perceive the value of PCTs for assessment of new prescription drugs.

 Criterion-based sample of US insurers and public payers.

 We gathered qualitative evidence from intensive interviews with formulary decision makers at 15 payers, representing 10 major types of US payers. Prior literature and exploratory interviews informed our question selection.

Payers viewed PCTs favorably despite wariness of drug company—sponsored trials. Payers would accept results from PCTs as part of payers’ synthesis of multiple sources of evidence. Payers were enthusiastic about 2 PCT features—a diverse population (compared with the more homogeneous populations typical of RCTs) and an active comparator drug (not placebo). Payers did not anticipate that PCTs would displace their own analyses of internal data. Pharmaceutical companies’ financial interest in obtaining trial results that favor their own drugs reduces PCTs’ perceived value and dampens their appeal to payers; nonetheless, payers would seek PCT results and review them carefully, as they do other evidence.

 Recommendations to trial designers based on payers’ views include tailoring different types of PCTs to different disease conditions, building in head-to-head comparisons in phase IIIb PCTs, and designing phase IV PCTs to include broader populations.

US payers desire both scientific rigor in clinical trial design and real-world applicability of trial results. Payers find pragmatic clinical trials (PCTs) appealing because PCTs provide:

		Information that allows evaluation of the comparative value of drugs against active comparators.

		Opportunities to triangulate PCTs’ evidence of effectiveness from real-world settings with data from regulatory RCTs and internal claims data sets.

		Health outcomes for patients who more closely reflect the diversity of patients they enroll (eg, patients with comorbidities, minorities), as well as meaningful data on subpopulationsfor which a drug works.

Interest in comparative-effectiveness research (CER) suggests a need to address both regulatory requirements and requests from post-regulatory decision makers (eg, payers, patients) for evidence of the relative value of comparator treatments. Pragmatic clinical trials (PCTs) have been proposed as potential means for satisfying both regulators and post-regulatory decision makers.

 Although the definition of PCTs is emerging, a recent definition is indicative:

The RCTs whose explicit purpose is to be most informative to decision makers are called pragmatic or practical clinical trials (PCTs). . . . Common elements of such trials include clinicallyeffective comparators, study patients with common comorbid conditions and diverse demographic characteristics, and providers from community settings. Primary and secondary outcomes are patient-centered, chosen to reflect what matters most to patients and clinicians.

Examples of PCTs include head-to-head comparisons of different pharmacologic agents for treatment of various conditions such as hypercholesterolemia

Randomized controlled trials (RCTs) have long been considered the highest level of evidence on clinical efficacy.

 However, regulatory trials historically have focused on narrow patient populations and used placebo comparisons. In contrast, PCTs aim to answer meaningful scientific questions about investigational drugs via study designs with (1) broad patient populations, (2)  active comparators, or both. Traits of PCTs may also include (3) outcomes of interest to patients, prescribers, and payers

; (4) usual care (in contrast to a strict clinical protocol); and (5) fewer data collection elements than RCTs have (to reduce burden of  data collection). Our study defines a PCT as a randomized trial with at least 2 of these 5 “pragmatic” traits; we used this working  definition to create a shared frame of reference with our interviewees. Others, including the pragmatic-explanatory continuum indicator summary (PRECIS) group, have further discussed the extent of pragmatism in PCTs.

Pragmatic clinical trials are seen as a bridge between RCTs (randomized) and observational studies (using real-world data).7 However, according to some observers, payer decision makers often perceive several factors as barriers to their accepting PCTs (

Moreover, because payers’ awareness and acceptance of PCTs have been understudied, uncertainty exists about whether and how US payers, key stakeholders for trials, would use results from PCTs to guide coverage and formulary decision making.

We gathered qualitative evidence to address our primary study question: 

Assuming that PCTs are randomized, how would payers view a PCT’s strength of evidence, compared with an RCT on the one hand and an observational study on the other?

 We pursued specific questions that explored which facets of PCTs payers find more compelling or more problematic than others (See 

 for interview questions). We conducted structured, in-depth interviews of formulary decision makers and senior staff  members at private health insurers and public payers. This study was funded by Eli Lilly and Company.

We selected a criterion-based (purposive) sample

 of 15 payers and related organizations drawn from 10 categories of payers  located throughout the United States (

). In selecting payers, our main criterion was that the sample reflect the diversity of US payer types—from managed care organizations offering comprehensive medical services and prescription drugs to pharmacy benefit managers; from national insurers to regional insurers; and from commercial insurers with nonelderly enrollees to public payers serving seniors.These organizations reflect most major types of US healthcare payers.

Our use of a nonprobability sample restricts the results’ generalizability. But unlike a conventional survey, the study’s purposive, interview-based approach offers a feasible method of obtaining reliable, credible, and detailed information about the use, perceived merits, and perceived difficulties of PCT evidence within US payers. Our method ensured that respondents were formulary decision makers (not junior staff assigned to fill out a website survey). Moreover, we believe our interviewing method reduced miscommunication.

Interview Format and Protocol Development 

The 1-hour interviews typically included 1 to 3 formularydecision makers, drawn from the following: vice president for pharmacy, director of pharmacy, chief medical officer, other physician, vice president for drug evaluation and pharmacoeconomics, and pharmacy and therapeutics committee chair. The interviews followed a protocol of questions, although interviewees’ own logic dictated occasional changes in the order of topics. The protocol permitted interviewees to convey both interest in and concerns about PCTs. After exploratory interviews at 5 payers, we sharpened the protocol’s focus on important, under-researched topics. Moreover, early in each interview, we defined a PCT as a randomized, prospective controlled trial with the traits listed above. This established a common terminology for interviewees and interviewers.

The study protocol posed questions that led interviewees to explore a decision tree (

). To elicit preferences between complex alternatives, we generally separated each alternative into constituent elements. Decision makers were asked to compare element A with element B, and then to explain the considerations that informed their choice. This process was repeated for various features of a PCT and for comparisons between a PCT and other types of studies. Our approach overcame 1 difficulty decision makers displayed in the initial interviews: describing their preferences regarding alternatives, each of which could involve multiple features of a trial or other type of evidence.

In addition, the protocol elicited each payer’s global assessment of PCTs as a type of evidence. This question guarded against misinterpreting a payer’s whole view of PCTs as being the sum of responses to piecemeal questions. Payers answered our questions in terms of clinical trials that met our definition of a PCT and typically did not refer to a particular PCT that they had reviewed, because their own experience with PCTs was limited.

To develop findings, the 2 investigators who conducted the interviews independently reviewed their own notes, identified interviewees’ main points, and classified them as “consensus” or as representing 2 or more interviewee groupings. These  investigators (who were not pharmaceutical company employees) then reviewed each other’s readings of the main points and  their classification, resolved differences by reexamination of their notes, and produced findings that synthesized the interviews.

Theme 1: Payers’ Views of Place of Pragmatic Clinical Trials in Chain of Evidence

. All formulary decision makers consider PCTs to be rare (to date). Some decision makers did cite a head-to-head trial funded by a drugcompany as an example of a PCT, but most struggled to recall PCTs they had used in decision making. Even “classic” PCTs, the Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial (ALLHAT) and Clinical Antipsychotic Trials in Intervention Effectiveness (CATIE),

 did not come to mind for many. We could not determine whether the issue was faulty memory (decision makers could not recall these trials at all) or imperfect labeling (decision makers did not link these trials to the term “PCT” because their practical features did not stand out). Segal

 explains why these 2 trials can be considered pragmatic. Payers’ impression that PCTs have been rare is consistent with a review that found only 22 PCTs out of the many clinical trials with reports published from January 1996 through September 2010.

Once comfortable with the PCT concept as we defined it, payer decision makers stated that PCTs would be part of the continuum of evidence they consider in assessing new drugs. As our interviewees explained, when gathering evidence about a new drug, their default mode is triangulation. Rather than focus on a single type of evidence, decision makers assemble several types—for example, RCTs, analyses of their own claims database, and observational studies of postlaunch data on populations other than their own enrollees. After weighing the different sources’ conclusions, limitations, and relevance to their own enrollees, decision makers triangulate (ie, draw inferences from multiple evidence sources about the relative effectiveness of a new drug). Accordingly, decision makers stated that PCTs would become one more link in the chain of evidence they consider.

Overall, among interviewees, the concept of triangulation across RCTs, PCTs, registry data, internal claims-based studies, and observational studies was widely shared. In an alternative analogy, evidence forms a multidimensional crossword puzzle: each dimension (type of evidence) provides a clue to the answer.

 Formulary decision makers find PCTs appealing because they improve the external validity of the evidence base. Payers generally view explanatory RCTs as exemplars of internal validity. Nonetheless, decision makers stated they want evidence of external validity: that the results apply not just to the RCT’s population but also to the payer’s own enrollees. Decision makers noted that patients excluded from many RCTs often resemble payers’ own patient populations more than the   RCTs’ restricted populations. Consequently, payer decision makers generally favor PCTs over RCTs because PCTs’ broad  population holds greater relevance to payers. Furthermore, payers recognize that more external validity (a wider array of subgroups studied) often entails less internal validity (small subgroup samples and less statistical power to detect a treatment effect). Among decision makers interviewed, the consensus was that external validity trumps internal validity—perhaps with caveats; for example, when a drug is the first in a therapeutic class or when evidence of effectiveness of comparators is weak.

Ranking Randomized Controlled Trials and Pragmatic Clinical Trials

. The decision makers interviewed differed in their views of where in their hierarchy of evidence PCTs would fit, especially when the evidence available included traditional RCTs. Some decision makers rejected the idea of ranking because RCTs and PCTs are incommensurate: “RCTs are the gold standard for demonstrating whether a drug works” while “PCTs provide evidence on the types of patients for which a drug works.” Other insurers expressed a related perspective: RCTs demonstrate whether a drug works under ideal conditions; in contrast, PCTs demonstrate whether a drug that works in an RCT also provides benefit under real-world conditions (imperfect patient adherence as well as physicians’ uneven conformity with practice guidelines). Most payers stated that traditional RCTs should be conducted to establish efficacy against a placebo in most (not all) therapeutic areas. In contrast, 1 payer wanted PCTs to supplant all RCTs. Nonetheless, most payers want traditional RCTs as well as PCTs and suggest that traditional RCTs should precede the PCTs.

Theme 2: Payers’ Views of Pragmatic Clinical Trial Design

Formulary decision makers emphasized that they hold nuanced views regarding PCT design. Five aspects stand out:

. As seen by decision makers generally, different medical conditions call for PCTs with different features. For example, condition X might call for a PCT with a broader patient population than most RCTs, while condition Y might call for a PCT with a head-to-head comparison or with particular outcomes measures.

 Although decision makers noted that diagnoses and clinical factors often affect which subgroups they want included in a trial, they singled out several subgroups frequently as key for inclusion in PCTs. Although designers of a PCT can select from an array of population subgroups, payers most often mentioned women and seniors. Most decision makers stated they want study populations to include more women. A smaller but sizable segment of interviewees wanted study populations to include seniors.

 Payers typically urge PCT designers to include “their” ethnic group in the study population. Ethnic groups that are most prominent among health plans’ enrollees differ by region. For example, it is possible that African Americans are the salient minority group in Northeast plans, Hispanics in Southwest plans, and Asians and Hispanics in West Coast plans.

Articles

Payers like key traits of pragmatic clinical trials, but are wary of pharmaceutical companies and plan to carefully scrutinize this new, appealing type of evidence.

Jonathan Ratner, PhD

Articles

Pragmatic Clinical Trials: US Payers' Views on Their Value