- - The MOVe-OUT trial (NCT04575597) sponsored by Merck showed a large treatment effect of molnupiravir on reducing hospitalization or death due to COVID-19 (about 50%).
- - The trial was stopped early due to such high efficacy, which along with the fact that the number of outcome events was relatively small (N = 81) signifies a probability of overestimation of the treatment effect of molnupiravir against COVID-19.
- - Currently, there is little information to evaluate the risk of overestimation. Transparency regarding all aspects of results is needed. Most importantly, the trial data monitoring committee and regulatory bodies should carefully and thoroughly examine the evidence before any recommendation is made.
Over the past few days, there has been much excitement around the globe about the potential antiviral role of molnupiravir against COVID-19. Interim analyses of the trial results (also known as the MOVe-OUT trial) about the efficacy and safety of molnupiravir in patients with COVID-19, released by Merck and Ridgeback on October 1, looked very promising.
Such seemingly promising results even triggered a global sell-off of stocks of vaccines against COVID-19 (Forbes). Currently, Merck is planning to submit an application for Emergency Use Authorization (EUA) to the United States Food and Drug Administration (US FDA) as well as submit marketing applications to other regulatory bodies around the world.
Between 1963 and early 2016, only about 90 drugs were formally approved to treat 9 human diseases caused by virus infection, including antivirals against HIV (Human Immunodeficiency Virus), HBV (Hepatitis B Virus), HCV (Hepatitis C Virus), HSV (Herpes Simplex Viruses), HCMV (Human Cytomegalovirus), HPV (Human Papillomavirus), RSV (Respiratory Syncytial Virus), VZV (Varicella-Zoster Virus), and Influenza virus (De Clercq et al., 2016). It would be a remarkable achievement and of great benefit to COVID-19 patients that molnupiravir for use against COVID-19 is formally approved and eventually added to the short list of approved antivirals.
However, evidence must be thoroughly examined before any recommendation or decision is made even in the currently urgent need for a drug which could effectively fight against COVID-19. When we looked at the available information on the molnupiravir trial (Merck and Ridgeback), some features of the trial, including the facts that the trial was stopped early for large positive treatment effects and the number of outcome events was relatively small, garnered our attention.
From a methodological perspective, a trial with a small number of events that is stopped early for apparent treatment effects might be prone to the risk of overestimation. In this OE Original, we discuss the potential impacts of stopping-a-trial-early-for-benefits.
1. What have we known so far about the molnupiravir trial?
The summary from the interim analyses of the molnupiravir trial was summarized in Table1.
The molnupiravir trial (NCT04575597) is a double-blind (i.e., participants and investigator) randomized controlled trial (RCT), involving participants who had confirmed mild-to-moderate COVID-19 with symptom onset within 5 days of randomization.
The trial efficacy results, represented by the composite outcome -- hospitalization or death due to COVID-19, are very positive. Molnupiravir was found to reduce the risk of hospitalization or death in COVID-19 patients by about 50% (28/385) at Day 29 after randomization, compared to placebo (53/377). In terms of safety, molnupiravir seems to be safe, as both the incidence of any adverse events and the incidence of drug-related adverse events were comparable between molnupiravir and placebo.
Table 1: Available information from the molnupiravir trial interim analysis
The Molnupiravir Trial
Double-blind randomized controlled trial (RCT)*
Patients who had laboratory-confirmed mild-to-moderate COVID-19, with symptom onset within 5 days of randomization.
The interim analysis evaluated data from 775 patients; At the time that the trial was stopped early, the sample size was 1550
Molnupiravir (dose and duration not specified in the press release)
Main Findings (quotes)
1. Molnupiravir reduced the risk of hospitalization or death by approximately 50%; 7.3% of patients who received molnupiravir were either hospitalized or died through Day 29 following randomization (28/385), compared with 14.1% of placebo-treated patients (53/377); p=0.0012.
2. Through Day 29, no deaths were reported in patients who received molnupiravir, as compared to 8 deaths in patients who received placebo.
3. The incidence of any adverse event was comparable in the molnupiravir and placebo groups (35% and 40%, respectively). Similarly, the incidence of drug-related adverse events was also comparable (12% and 11%, respectively). Fewer subjects discontinued study therapy due to an adverse event in the molnupiravir group (1.3%) compared to the placebo group (3.4%).
* Information was from the clinicaltrials.gov, NCT04575597
Although not reported in the press release, relative risk (RR) and 95% confidence interval (CI), relative risk reduction (RRR) and 95% CI, absolute risk reduction (ARR) and 95% CI, as well as the number needed to treat (NNT) along with its 95% CI for the composite outcome -- hospitalization or death due to COVID-19 through Day 29 after randomization, were calculated in the OE Original.
- RR: 0.52 (95% CI: 0.33 to 0.80), indicating that molnupiravir reduces around 50% of the risk of being hospitalized or dead due to COVID-19, compared to placebo.
- RRR: 48% (95% CI: 20% to 67%), suggesting that for those who would have been hospitalized or dead due to COVID-19, had they been in the placebo group, 48% would survive if they receive molnupiravir.
- ARR: 6.8% (95% CI: 2.42% to 11.15%), indicating that if 100 COVID-19 patients were treated with molnupiravir, about 7 would be prevented from hospitalization or death.
- NNT: ~15 (95% CI: 9 to 42), meaning clinicians need to treat 15 patients to avoid 1 hospitalization or death due to COVID-19.
However, it came to our attention that due to such impressive treatment effects the molnupiravir trial stopped early. It has been recognized that stopping a trial early for benefits might lead to the risk of overestimating the treatment effects (e.g., Bassler et al., 2007; Ioannidis, 2005; Montori et al., 2005), and therefore worth an in-depth discussion.
“The reason for early stopping [of a clinical trial] that may have the most effect on clinical practice, however, is that investigators note treatment effects that appear to be unlikely by chance -- and that usually large -- that persuade them that the experimental intervention is beneficial.”
Users' Guides to the Medical Literature:
A Manual for Evidence-Based Clinical Practice, 3rd ed, pp. 136
2. The issue of stopping-a-trial-early-for-benefits
The clinical trials which are stopped early for benefits are often referred to as the truncated trials (Guyatt et al., 2015). It is often tempting to stop a clinical trial early when it shows “dramatic superiority at interim analysis [in order to] save money and lives by hastening the availability of an effective intervention” (The Lancet, 2008).
However, by terminating clinical trials early for apparent benefits, especially those also having a small number of outcome events, there is a probability that the treatment effects are overestimated. According to Guyatt et al. (2015), results from clinical trials on the same research question fluctuate around the true underlying effect. In other words, there should be a random distribution of results from trials investigating the same research question around the true underlying effect. The trials stopped early for benefits could be those at the high end of the random distribution, which might cause the overestimation of treatment effects.
A systematic review, including 91 RCTs stopped early for benefits (truncated RCTs) and 424 matching non-truncated RCTs, found that the pooled ratio of RRs in truncated RCTs vs matching non-truncated RCTs was 0.71 (95% CI: 0.65 to 0.77) (Bassler et al., 2010). The meta-regression further showed that the overestimation of the truncated RCTs was independent of either the presence of a statistical stopping rule or the methodological quality of the RCTs (i.e., allocation concealment and blinding), but associated with the number of outcome events (Bassler et al., 2010). When there were fewer than 500 outcome events, large differences in treatment effect size (ratio of RRs < 0.75) existed between the truncated and non-truncated RCTs (Bassler et al., 2010).
What does the 0.71 (95% CI: 0.65 to 0.77) mean? We illustrate this using the truncated molnupiravir trial, which reported only 81 outcome events (i.e., hospitalization or death due to COVID-19), as an example. When fitting 0.71 (95% CI: 0.65 to 0.77) to the truncated molnupiravir trial which had a RR of 0.52 (95% CI: 0.33 to 0.80), we could infer that the average RR from the non-truncated molnupiravir trials would be about 0.73 (95% CI: 0.46 to 1.13) had such trials on molnupiravir against COVID-19 matched the pattern seen in Bassler et al. (2010). As we can see now, the statistical significance of the efficacy of molnupiravir against COVID-19 no longer exists.
Some researchers, however, criticized studies like Bassler et al. (2010) which compared truncated RCTs vs. matching non-truncated RCTs. Goodman, (2009) argued that such comparisons were like comparing “apples to oranges… Trials stopped early for efficacy are by definition statistically significant, whereas results from a trial stopped because a target size was achieved can fall anywhere. If one compares statistically significant results to all possible results, the point estimates from the significant studies will be higher, a well-known fact that is the basis of publication bias.”
Goodman (2009) argued that it is more appropriate to compare the truncated trials with a comparable set of positive trials that used fixed sample sizes with no interim monitoring, as what Freidlin et al. (2009) did. Freidlin et al. (2009) concluded that truncated RCTs might have a relatively small effect on efficacy estimates when properly implemented, analyzed, and reported. Specifically, Freidlin et al. (2009) found that the overestimation of treatment effects might be negligible for truncated trials when they had a well-designed interim-monitoring plan and were stopped at at least 50% information. However, the study also acknowledged that deviating substantially from the true effect would occur if the true effect was actually small, the trials were stopped at earliest stopping points (i.e., =< 25% of the sample size), or when the highest early monitoring boundaries were used for the early termination (Freidlin et al., 2009; Goodman, 2009).
Based on Freidlin’s et al. (2009) findings, the truncated molnupiravir trial might be at a low risk of overestimation, if the trial was carried out appropriately. Unfortunately, the press release of the truncated molnupiravir trial did not provide adequate information for us. It only mentioned that “At the time of the decision to stop recruitment based on the compelling interim efficacy results, the trial was approaching full recruitment of the Phase 3 sample size of 1,550 patients, with more than 90% of the intended sample size already enrolled.”
The press release did not provide details about the early monitoring boundaries or whether the true treatment effect of molnupiravir against COVID-19 was large or not (or we should say the data monitoring committee’s belief about the magnitude of the true treatment effect because one does not know the true effect).
The belief of the data monitoring committee about the relationship between these above 2 pieces of information is critical, as Goodman (2009) pointed out, “if the true effect is below the stopping boundary, estimates beyond the stopping boundary must deviate from the truth. On the other hand, if the true effect is large, the possibility of which motivates the use of a stopping rule in the first place, the measured effect will be unbiased.” In this sense, Goodman (2009) stressed that “the best protection against injudicious stopping is a wise and well functioning data monitoring committee, aided but not ruled by statistical stopping guidelines.”
Take away message
We should feel excited about the positive results of the efficacy of monulpiravir against COVID-19 for it may save thousands of lives. However, we should also acknowledge the risk of overestimating the treatment effect of monulpiravir because the trial was stopped early for benefits and the number of outcome events reported in this trial is relatively small (N = 81).
Transparent information is needed to evaluate the risk. Most importantly, the data monitoring committee and regulatory bodies should carefully and thoroughly examine the evidence before any recommendation is made. No matter how urgent it is to fight the COVID-19 pandemic, rushing to any decision that is not supported by high-quality evidence will do harm to the patients, hurt the credibility of regulatory bodies, and cause doubts about medical science among the public.
Bassler, D., et al. (2010). Stopping Randomized Trials Early for Benefit and Estimation of Treatment Effects: Systematic Review and Meta-regression Analysis. JAMA, 303(12), 1180-1187. doi:10.1001/jama.2010.310
Bassler, D., et al. (2007). Systematic reviewers neglect bias that results from trials stopped early for benefit. J Clin Epidemiol, 60(9), 869-873. doi:10.1016/j.jclinepi.2006.12.006
De Clercq, E., et al. (2016). Approved Antiviral Drugs over the Past 50 Years. Clinical microbiology reviews, 29(3), 695-747. doi:10.1128/CMR.00102-15
Freidlin, B., et al. (2009). Stopping clinical trials early for benefit: impact on estimation. Clinical Trials, 6(2), 119-125. doi:10.1177/1740774509102310
Goodman, S. N. (2009). Stopping trials for efficacy: an almost unbiased view. Clinical Trials, 6(2), 133-135. doi:10.1177/1740774509103609
Guyatt G, & Rennie D, & Meade M.O., & Cook D.J.(Eds.), (2015). Users' Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice, 3rd ed. McGraw Hill. https://jamaevidence.mhmedical.com/content.aspx?bookid=847§ionid=69030714
Ioannidis, J. P. (2005). Contradicted and initially stronger effects in highly cited clinical research. Jama, 294(2), 218-228. doi:10.1001/jama.294.2.218
Montori, V. M., et al. (2005). Randomized trials stopped early for benefit: a systematic review. Jama, 294(17), 2203-2209. doi:10.1001/jama.294.17.2203
The Lancet. (2008). Stopping trials early for benefit: too good to be true. The Lancet, 371(9621), 1310. doi:10.1016/S0140-6736(08)60569-3