US FDA comments on Unlearn’s PROCOVA methodology

By Charles Fisher, Founder and CEO, Unlearn.AI

PROCOVA™ is a statistical methodology developed by Unlearn.AI for incorporating prognostic scores derived from trial participants’ digital twins into the design and analysis of phase 2 and 3 clinical trials. This methodology was previously qualified by the European Medicines Agency (EMA) for use as the primary analysis in phase 2 and 3 clinical trials with continuous outcomes. Unlearn recently received comments from the Center for Drug Evaluation and Research (CDER) at the US Food and Drug Administration (FDA) stating that they concur with EMA and that PROCOVA does not deviate from current guidance. This confirms our belief that PROCOVA is an acceptable statistical methodology under current guidance from both EMA and FDA.

In the context of a clinical trial, a trial participant’s digital twin provides a comprehensive, probabilistic forecast of what their future clinical outcomes may be if they were to get assigned to the control group. A participant’s digital twin is used to compute a prognostic score by taking the expected value of this probability distribution. That is, the prognostic score describes what a trial participant’s future clinical outcomes would be, on average, if they got assigned to the control group. Adjusting for participants’ prognostic scores in the analysis of a clinical trial increases power, which can be leveraged to improve decision making or to gain efficiency by decreasing the number of participants who need to be randomized to the control group.

When we introduced PROCOVA, we described it as a special case of ANCOVA—a widely used statistical methodology in clinical trials—in which one estimates a treatment effect while adjusting for a covariate that obtains the maximum gain in power over an unadjusted analysis [Schuler et al (2021) International Journal of Biostatistics]. We showed that, theoretically, this optimal covariate is the prognostic score.

Typically, trials that use ANCOVA adjust for simple baseline covariates such as each participant’s age. In PROCOVA, by contrast, a pre-specified model based on artificial intelligence trained on historical patient data is used to construct a prognostic score for each participant collected at their first visit in a trial. This way, historical patient data is used to learn to construct an approximation to the optimal covariate that maximizes power in a future study, thereby leveraging rapidly improving machine learning technologies and increasingly vast quantities of individual participant data to improve clinical trials.

Being a special case of ANCOVA, PROCOVA inherits its statistical properties; for example, it produces unbiased estimates of treatment effects and controls the type-I error rate of hypothesis tests. And, since ANCOVA is an acceptable and widely used methodology for the analysis of clinical trials, we argued that PROCOVA should also be an acceptable methodology under current guidance from both EMA and FDA.

Unlearn received a qualification opinion from the EMA through the novel methodologies qualification opinion program in September 2022 stating “CHMP qualifies PROCOVA as prognostic score adjustment and the proposed procedures, as described in a handbook for trial statisticians, could enable increases in power or precision of treatment effect estimates in controlled randomised clinical trials with continuous outcomes”.

Previously, Unlearn had also submitted a letter of intent (LOI) to the CDER/CBER Innovative Science and Technology Approaches to New Drugs (ISTAND) Pilot Program at FDA asking for a review of PROCOVA for use in phase 2 and 3 clinical trials with continuous outcomes. In a letter recently sent to Unlearn, the FDA determined that this LOI would not be accepted into the ISTAND Pilot Program because our PROCOVA methodology is already covered by FDA’s current guidance on Adjusting for Covariates in Randomized Clinical Trials for Drugs and Biological Products.

The comments from FDA are reproduced verbatim below.

1. We appreciate your interest in statistical methodology intended to improve the efficiency of clinical trials by using trial subjects’ predicted outcomes on placebo (prognostic scores) in covariate adjustment with linear models in order to increase the precision of treatment effect estimation and statistical power.

2. Based on our review of your LOI, PROCOVA is in principle a special case of an Analysis of Covariance (ANCOVA). This type of model is expected to improve the precision of statistical analysis in clinical trials.

3. If a prognostic covariate is derived using independent external data, then it is valid to adjust for this pre-specified covariate in an ANCOVA in a prospective trial. PROCOVA may or may not lead to a more precise treatment effect estimate than more traditional ANCOVA modeling. This likely will depend on the prognostic value of the PROCOVA covariate in its intended study population.

4. A power calculation may be optimistic if the correlation between a derived covariate and an outcome of interest differs between external data and the prospective trial in which the covariate will be used for adjustment. An unadjusted sample size calculation will often be conservative and help prevent underpowered trials. This is usually a good practice, and conservatism is often warranted in sample size planning.

5. While preparing the guidance Adjusting for Covariates in Randomized Clinical Trials for Drugs and Biological Products, we worked with comments from the public docket. We concur with EMA that PROCOVA is a special case of ANCOVA. As such, CDER’s current feedback is that PROCOVA does not appear to deviate from our guidance. Any future requests related to its use should be product or trial specific and directed by a sponsor to the appropriate review division.