Known as a “methods wiz” among her collaborators, Jessie Franklin, Ph.D. dedicates her ingenuity to developing and applying statistical methods to the study of comparative effectiveness and adverse effects of drugs, consequences of drug policy, and drug utilization. She is an assistant professor of medicine at Harvard Medical School and biostatistician in the Division of Pharmacoepidemiology and Pharmacoeconomics at Brigham and Women’s Hospital. There, she leads the RCT DUPLICATE Project, a landmark study to demonstrate study design, methods, and value of real-world evidence in the approval and regulation of drugs. We ask her about the purpose and the promise of the project.
Q: What is the RCT DUPLICATE Project, and what are its goals?
A: The project launched in late 2017 when the Food and Drug Administration contracted with us to replicate published randomized controlled trials using real-world data sets. Their intent was to find out if the use of real-world evidence would have led to the same regulatory decisions.
They wanted to know: If we had evidence from a real-world data study instead of an RCT, would we have come to the same regulatory conclusion?
So, we’ve selected 30 published randomized controlled trials that contributed to a regulatory decision to replicate with non-randomized, real-world data. We spent the first year setting up the processes for replication and figuring out how we will judge agreement once we've completed the replication. We’re now working on implementing the replications. The results will demonstrate whether RWE can be used to supplement or even replace some clinical trials for regulatory decision-making.
Q: What are the FDA’s and others’ concerns about working with real-world data?
A: We’re working huge transactional data sets—three health insurance claims databases—that have tons of useful information about patients taking medications in the real world. But the validity of studies using these data sets has been in doubt due to concerns about unmeasured confounders, data quality and accuracy, a lack of study transparency, and so on.
We hope that the RCT DUPLICATE project can address those concerns by providing an empirical evidence base for building confidence in real-world data studies and a process for how we can create valid studies.
Q: Why does replicating randomized trials establish the validity of real-world data studies?
A: We use the treatment effect estimate from the RCT as the reference standard against which we can compare the effect estimate from the nonrandomized replication. Replication is the only way we have to assess the success of the entire research process—from identifying the clinical question of interest, selecting the appropriate data source to answer that question to epidemiologic study design and selecting the analyses that determine our final result. We are practiced at studying each one of those parts in isolation of the others. But confidence in an analytic result requires assessing the entire research process.
RCT DUPLICATE will help us learn which methods work best in real data and real clinical questions. It will tell us which questions can and which questions cannot be answered with real-world data—and if in fact we do need to run an RCT to answer validly.
Q: How do you select randomized control trials to replicate in the RCT DUPLICATE project?
A: The purpose of RCT DUPLICATE is to design the non-randomized real-world data studies—across many clinical questions—with high methodological quality. So, our process begins with selecting a set of candidate randomized controlled trials that we believe are replicable in our data sources.
Trials may be replicable in some data sources, but not in others. For example, we're not able to replicate any trials that have rheumatological primary outcomes because we don't have those outcomes measured in our health insurance claims datasets.
What we do have in these data are hard clinical endpoints. These are the events that send patients to the hospital: a hip fracture, stroke, heart failure or asthma exacerbation, and so on. If one of these is the outcome in an RCT, then that's a trial that we can probably replicate.
Therapeutic areas of the trials we selected fall into four buckets: diabetic medications, cardiovascular medications (anti-coagulants, anti-platelets, anti-hypertensives, heart failure medications, statins, anti-arrhythmics), anti-osteoporosis medications, and asthma and COPD. All have clinical outcomes that we can measure well in our claims databases.
Q: What ensures that your RWD analyses are truly regulatory grade?
A: The FDA has more than 50 years of experience of relying on the results of randomized trials to inform their regulatory decision-making. Now they have a mandate from Congress with the 21st Century Cures Act to come up with guidance for how real-world data can be used for regulatory decision-making. While they've been using real-world data for safety through the Sentinel program, they have less experience in using these data to support the effectiveness of a medication.
So, just as this project is providing empirical data on how well real-world data can match the results of RCTs, it’s also providing FDA with a process model for accepting real-world data evidence to support regulatory decisions.
For that reason, we're meticulous about the process of implementation. For example, before we ever look at outcomes or comparative results, we’ve finalized the design, specified our primary analysis, and registered our full protocol on ClinicalTrials.gov, where it’s given a time stamp.
Only after the protocol has been registered do we implement our study, do our analyses, and document all findings. This process ensures that the decision to move forward with a specific design and analysis plan depends only on its scientific validity, not on whether or not we like the end results.
The question then is how do we share our study and results with FDA? When a sponsor submits randomized trial evidence to support a regulatory approval, they submit their entire randomized controlled trial dataset to FDA. Then FDA statisticians and reviewers work with the data to adjudicate every outcome, evaluate the details of the analyses, and produce alternative analyses. FDA is accustomed to working with the data at that level, and we need to enable them to work with real-world data at the same level.
That’s one of the reasons we’re using the Aetion Evidence Platform in this project. The platform will allow FDA to work with the data without needing to transfer the data directly to FDA. When we share a replication with FDA, they can go in to the platform and look around. They can see what we did, what we didn't do, and change whatever they want: the follow-up window, the definition of the outcome, and so on. Perhaps they disagree with an aspect of the design, like the follow-up time. They can go in and change that to see if the result changes in a meaningful way.
This is important because if the FDA can change things, they can see if the results hold up. And if the results hold up, then FDA can trust the results. If they change a small detail and the result completely changes, they know this result is not robust.
Q: How do you define success for the RCT DUPLICATE project?
A: We fully expect we're not going to be able to successfully replicate every single trial that we are working on. But if we can delineate the characteristics that make a clinical question answerable in real-world data, then we've learned a lot.
We also want to confirm which design and analytic choices make our database studies interpretable for decision-making. What designs and analyses can we use that will provide valid causal conclusions? And that will provide results that match the results of the randomized controlled trial? If we can answer those questions, the project will have been a success.