Loading...
Visit our Evidence-Based Covid-19 Website and Stay Up to Date with the latest Research.
Volume:4 Issue:9 Number:3 ISSN#:2563-559X
OE Original

Meta-analyses Are Only As Good As The Studies They Combine: A Lesson Learned From Meta-analyses On Ivermectin Against COVID-19

Authored By: OrthoEvidence

September 20, 2021

How to Cite

OrthoEvidence. Meta-analyses Are Only As Good As The Studies They Combine: A Lesson Learned From Meta-analyses On Ivermectin Against COVID-19. OE Original. 2021;4(9):3. Available from: https://myorthoevidence.com/Blog/Show/149

Highlights


 - Meta-analyses (MAs) are analytical methods combining data one puts in. The validity of MAs results depends on the individual studies included. Here we discuss the issue of combining-appropriate-studies-in-an-MA using existing MAs regarding the efficacy of ivermectin on all-cause mortality in patients with COVID-19.


  •  - Our aim to use ivermectin meta-analyses as examples is not to criticize any published studies but to raise the awareness and caution among clinicians about combining-appropriate-studies-in-an-MA issue when interpreting MA results and applying them in their decision-making process.


  •  - Two published MAs, conducted around similar periods in time, made two different conclusions on the efficacy of ivermectin on all-cause mortality in patients with COVID-19. Popp et al. (2021) found no significant difference in all-cause mortality between ivermectin and control (i.e., placebo/standard of care) in hospitalized patients with confirmed COVID-19, whereas Bryant et al. (2021) identified significant differences.



  •  - We found that Bryant et al. (2021) combined 15 trials on the all-cause mortality outcome, while Popp et al. (2021) included only 2 studies. This signifies that the difference in conclusions between the two MAs was likely from the differences in the studies included.


  •  - We believe the MA conducted by Popp et al. (2021) was more likely to reflect the true efficacy of ivermectin on reducing mortality in patients with COVID-19 because it I) excluded studies involving control groups in which drugs with unproven efficacy was used (i.e., lopinavir/ritonavir, hydroxychloroquine, chloroquine), II) excluded RCTs involving participants with unknown or negative SARS-CoV-2 infection status, III) excluded RCTs in which ivermectin was administered in combination with doxycycline, a drug with unproven efficacy on COVID-19, and IV) no clear timeframe for the mortality outcome. The MA conducted by Bryant et al. (2021) included all above-mentioned studies.



A systematic review (SR) is defined as “a summary of research that addresses a focus clinical question in a systematic, reproducible manner” (Murad et al., 2015). Often, SR is accompanied by a so-called meta-analysis (MA), which refers to statistical pooling or combining of the results from different studies on the same topic (Lee, 2018; Murad et al., 2015).


The advantages of MAs over the narrative summary of evidence are well-known. For example, MAs are able to yield a single best estimate of effect which could facilitate clinical decision-making (Murad et al., 2015). In addition, MAs narrow the confidence interval (CIs) (increases precision) by bring together individual primary studies (Lee, 2018). The scientific value and importance of MAs have been widely recognized and accepted among researchers and clinicians, and therefore the conduct of MAs is increasingly popular.


The high popularity of MAs can be seen in the synthesis of evidence regarding COVID-19. As of September 15, 2021, a search in PubMed with the keywords “COVID-19” and ‘meta-analysis” in the title field with the filter “meta-analysis” yielded over 95 thousand records (Search strategy: "covid-19"[Title] OR "meta-analysis"[Title]) AND (meta-analysis[Filter]).


However, MAs are just analytical methods processing the data one puts in. Selecting and combining of appropriate studies are critical for conducting MAs as the validity of MA results depends on the individual studies included.


In the present OE Original, we discuss the issue of combining-appropriate-studies using existing MAs with regard to the efficacy of ivermectin on all-cause mortality in patients with COVID-19. Using ivermectin MAs as examples here is not to criticize any published studies but to raise the awareness among clinicians in terms of interpreting MA results and applying them in their decision-making process.


1. Two ivermectin MAs


Ivermectin, which is used for the treatment of parasite infestations, has gained substantial attention among both the medical professionals and the general public as one of the potential drugs that can be repurposed for treating SARS-CoV-2 infection. On one hand, major public health agencies, such as Health Canada and United States Food and Drug Administration (US FDA), are currently advising against the use of ivermectin for the treatment of COVID-19 due to lack of certain evidence. On other hand, MAs, quantitatively synthesizing evidence from randomized controlled trials (RCTs) regarding the use of ivermectin, have been emerging in the literature.


Among several, two MAs were discussed here (Bryant et al., 2021; Popp et al., 2021). Table 1 presents the information on PICO (Patient, Intervention, Comparator, Outcome) and literature search of the two MAs. We also appraised the two SRs using AMSTAR 2 -- a critical appraisal tool for SRs. Both SRs were rated as high overall confidence in the review results (0 or 1 non-critical weakness).


We selected these two MAs because they both examined RCT evidence regarding the efficacy of ivermectin on all-cause mortality in COVID-19 patients but made different conclusions.


The MA conducted by Bryant et al. (2021) found that ivermectin significantly reduces risk of all-cause mortality in hospitalized patients with COVID-19 [average risk ratio (RR): 0.38, 95% CI: 0.19 to 0.73), compared with no ivermectin. The SR suggests that “Health professionals should strongly consider its [ivermectin] use” (Bryant et al., 2021).


The MA, a Cochrane review published by Popp et al. (2021), however, concluded that uncertainty remains about whether ivermectin reduces all-cause mortality in hospitalized patients with confirmed COVID-19 (RR: 0.60; 95% CI: 0.14 to 2.51), compared to placebo or standard of care. To be noticed, Popp et al. (2021) counted 3 deaths in the placebo arm of the RCT conducted by Ravikirti et al. (2021), while Bryant et al. (2021) and we found the number reported by Ravikirti et al. (2021) was 4. Popp et al. (2021) did not specify the reason why 3 deaths rather than 4 were included in the MA. We recalculated RR with 4 deaths, and the RR was 0.48 (95% CI: 0.07 to 3.12), which did not change the conclusion made by Popp et al. (2021).


Over the results of the two MAs, a question inevitably arises: Why did two MAs, conducted around similar periods in time, give two different conclusions on the role of ivermectin in reducing all-cause mortality in patients with COVID-19? 


Table 1.  Information on PICO and literature search of the two MAs conducted by Bryant et al., 2021 and Popp et al. (2021)

 

Bryant et al. (2021)

Popp et al. (2021)

Population

People with mild, moderate, severe or critical COVID-19

Participants with confirmed SARS-CoV-2 infection (RT-PCR or antigen testing)

Intervention

Oral ivermectin, a minimum single dose of 6 mg. Studies assessing ivermectin in combination with doxycycline or other medicines or supplements will be included. Studies comparing different formulations, doses, and schedules of ivermectin will also be included.

All doses and regimens of ivermectin, co-interventions must have been comparable between the study arms, i.e. ivermectin plus standard of care versus standard of care

Comparator

No treatment, standard of care, placebo, or active pharmacological comparator

No treatment, standard of care, placebo, or active pharmacological comparator with proven efficacy such as dexamethasone and remdesivir

Outcome

All-cause mortality

All-cause mortality up to 28 days

Study Design

RCTs, Quasi-RCTs, Cluster RCTs

RCTs (Non-standard RCT designs, such as cluster RCTs and cross-over trials, were not eligible. These designs are considered inappropriate in this context, since the underlying cause of COVID-19 is an infection with the SARS-CoV-2 virus and the medical condition evolves over time.)

Databases Searched

1. CCSR, including CENTRAL, MEDLINE, EMBASE

2. Chinese databases for RCTs

1. CCSR, including CENTRAL, MEDLINE, EMBASE, ClinicalTrials.gov, WHO ICTRP

2. Web of Science Core Collection, including Science Citation Index Expanded and Emerging Sources Citation Index

3. Preprint servers, including MedRxiv and Research Square

Other Literature Search Information

1. Searched conducted on April 25, 2021;

2. No language restriction;

3. The reference lists of included studies were also searched

1. Searched conducted on May 26, 2021;

2. No language restriction;

3. The reference lists of included studies, SRs, and MAs were also searched

Included Studies

1. 21 trials were eligible;

2. 15 trials were meta-analyzed for all-cause mortality

1. 14 trials were eligible;

2. 2 studies were meta-analyzed for all-cause mortality

CCSR: Cochrane COVID-19 Study Register; CENTRAL: Cochrane Central Register of Controlled Trials; WHO: World Health Organization; ICTRP:  International Clinical Trials Registry Platform




2. The data put in an MA matters


To find possible answers to the question above, we need to take a close look at Table 1 again. It is obvious that Bryant et al. (2021) included 15 trials in the MA on the all-cause mortality outcome, while Popp et al. (2021) included only two studies. This signifies the issue of choosing-studies-to-be-included-in-an-MA.


In fact, choosing-appropriate-studies-to-be-included-in-an-MA is a critical issue for conducting MAs, which may cause selection bias if not properly carried out. Walker et al. (2008) pointed out that choosing-appropriate-studies-to-be-included-in-an-MA aimed to “reduce differences among studies, eliminate replication of data or studies, and improve data quality, and thus enhance the validity of the results.”


Table 2 shows the differences between RCTs included in the two MAs for the all-cause mortality outcome. Apparently, the MA conducted by Popp et al. (2021) was more likely to reflect the true efficacy of ivermectin on reducing mortality in patients with COVID-19 because it I) excluded RCTs involving control groups in which drugs with unproven efficacy was used (i.e., lopinavir/ritonavir, hydroxychloroquine, chloroquine), II) excluded RCTs involving participants with unknown or negative SARS-CoV-2 infection status, III) excluded RCTs in which ivermectin was administered in combination with doxycycline, a drug with unproven efficacy on COVID-19, and IV) no clear timeframe for the mortality outcome (Table 2).


Popp et al. (2021) did not include RCTs reporting zero events in both arms, whereas Bryant et al. (2021) did. The Cochrane Handbook suggests that “The standard practice in meta-analysis of odds ratios and risk ratios is to exclude studies from the meta-analysis where there are no events in both arms. This is because such studies do not provide any indication of either the direction or magnitude of the relative treatment effect.” However, Friedrich et al. (2007) recommended that review conductors should include 0 event trials when calculating odds ratio or relative risk to “provide a more conservative estimate of effect size … and to provide analytic consistency and include the same number of trials in the meta-analysis…



Table 2. Studies included in the two MAs for all-cause mortality

Study ID

Bryant et al. (2021)

Popp et al. (2021)

Ahmed et al. (2020)

Included, Bryant et al. (2021) considered 0 and 0 deaths in both groups

Not Included, Ahmed et al. (2020) claimed all-cause mortality as a secondary outcome, but not explicitly reported the values in results.

Babalola et al. (2020)

Included

Not Included, ivermectin compared to a control (lopinavir/ritonavir) with unproven efficacy on COVID-19

Chaccour et al. (2020)

Included, Bryant et al. (2021) considered 0 and 0 deaths in both groups

Not Included, Chaccour et al. (2020) listed the proportion of patients progressing to severe disease or death during the trial as an outcome, but only implied no deaths by claiming "No patient from either group progressed to severe disease."

Elgazzar et al. (2020)

Included

Not Included, ivermectin was compared to a control (hydroxychloroquine) with unproven efficacy on COVID-19. The study was retracted due to ethical concerns on 14 July 2021.

Fonseca et al. (2021), same study as Galan et al. (2021)

Included

Not Included, ivermectin compared to control arms (hydroxychloroquine/chloroquine) with unproven efficacy on COVID-19

Gonzalez et al. (2021)

Included

Included

Hashim et al. (2020)

Included

Not Included, ivermectin administered in combination with another active drug (doxycycline) with unproven efficacy on COVID-19

Lopez-Medina et al. (2021)

Included

Not Included, the primary analysis was per-protocol due to a labelling error that resulted in 16% of participants receiving the wrong intervention

Mahmud et al. (2020), now published as Mahmud et al. (2021)

Included

Not Included, ivermectin administered in combination with another active drug (doxycycline) with unproven efficacy on COVID-19

Mohan et al. (2021)

Included

Not Included, Mohan et al. (2021) reported mortality at 14 days, which was considered too short for mortality up to 28 days

Niaee et al. (2020), now published as Shakhsi Niaee et al. (2021)

Included

Not Included, study included around 30% of SARS-CoV-2-negative participants

Okumus et al. (2021)

Included

Not Included, Okumus et al. (2021) provided an unclear time frame of follow-up (an average of 3 months)

Petkov et al. (2021)

Included

Not Included, no scientific publication of results except a press release on the manufacturer's website

Ravikirti et al. (2021)

Included

Included

Rezai et al. (2020), now published as Shahbaznejad et al. (2021)

Included

Not Included, study included 76.8% participants with unknown or negative SARS-CoV-2 status




Here we examined the pooled effect estimates when including different studies in the MA:


Scenario #1: Including all relevant studies regardless whether the studies were appropriate or not to be included as Bryant et al. (2021) did;


Scenario #2: Including RCTs with most restrictive criteria as Popp et al. (2021) did;


Scenario #3: On the basis of MA conducted by Popp et al. (2021), including additional studies from Bryant et al. (2021) which reported 0 events in either arm;


Scenario #4: On the basis of MA conducted by Popp et al. (2021), including additional studies from Bryant et al. (2021) which included mortality outcomes with no clear timeframe;


Scenario #5: On the basis of MA conducted by Popp et al. (2021), including additional studies from Bryant et al. (2021) which included mortality outcomes with no clear timeframe as well as those which reported 0 events in either arm.


The effect estimates were shown in Table 3. We found that only scenario #1, in which all relevant studies were included regardless whether the studies were appropriate or not to be included, yielded a statistically significant RR. The rest scenarios all suggested no significant differences in mortality reduction between ivermectin and control (i.e., placebo/standard care) among patients with COVID-19.


Table 3. Effect estimates obtained from MAs including different studies

Scenario

Effect Estimate

#1. Including all relevant studies regardless whether the studies were appropriate or not to be included as Bryant et al. (2021) did

RR: 0.38; 95% CI: 0.19 to 0.73

#2. Including RCTs with most restrictive criteria as Popp et al. (2021) did

RR: 0.48; 95% CI: 0.07 to 3.12*

#3. On the basis of MA conducted by Popp et al. (2021), including additional studies from Bryant et al. (2021) which reported 0 events in either arm

RR: 0.48; 95% CI: 0.07 to 3.12

#4. On the basis of MA conducted by Popp et al. (2021), including additional studies from Bryant et al. (2021) which included mortality outcomes with no clear timeframe

RR: 0.60; 95% CI: 0.30 to 1.19

#5. On the basis of MA conducted by Popp et al. (2021), including additional studies from Bryant et al. (2021) which included mortality outcomes with no clear timeframe as well as those which reported 0 events in either arm.

RR: 0.60; 95% CI: 0.30 to 1.19

* The original RR given by Popp et al. (2021) was 0.60 (95% CI: 0.14 to 2.51). However, Popp et al. (2021) counted 3 deaths in the placebo arm of the RCT conducted by Ravikirti et al. (2021), while Bryant et al. (2021) and we found the number of deaths reported by Ravikirti et al. (2021) was 4. We recalculated RR with 4 deaths, and the RR was 0.48 (95% CI: 0.07 to 3.12), which did not change the conclusion made by Popp et al. (2021).


Closing Remark


In this OE Original, we illustrated the importance of including-appropriate-studies-in-an-MA by using two existing MA examples regarding the efficacy of ivermectin on all-cause mortality in patients with COVID-19. In this case, including inappropriate studies in the MA, such as those involving control groups in which drugs with unproven efficacy were used (i.e., lopinavir/ritonavir, hydroxychloroquine, chloroquine), might result in misleading conclusions -- which is that ivermectin was effective. When excluding such inappropriate studies, the statistical significance disappeared.


We concluded that current evidence was unable to ascertain the efficacy of ivermectin on all-cause mortality among patients with COVID-19. We also hope to raise the awareness and caution among our readers about the including-appropriate-studies-in-an-MA issue when interpreting MA results and applying them in their decision-making process. Just as our title suggests, meta-analyses are only as good as the data one puts in. The validity of MA results depends on the studies included.



Reference


Ahmed, S., et al. (2021). A five-day course of ivermectin for the treatment of COVID-19 may reduce the duration of illness. International journal of infectious diseases : IJID : official publication of the International Society for Infectious Diseases, 103, 214–216. https://doi.org/10.1016/j.ijid.2020.11.191

Babalola, O. E., et al. (2021). Ivermectin shows clinical benefits in mild to moderate COVID19: A randomised controlled double-blind, dose-response study in Lagos. QJM : monthly journal of the Association of Physicians, hcab035. Advance online publication. https://doi.org/10.1093/qjmed/hcab035

Bryant, A., et al. (2021). Ivermectin for Prevention and Treatment of COVID-19 Infection: A Systematic Review, Meta-analysis, and Trial Sequential Analysis to Inform Clinical Guidelines. American journal of therapeutics, 28(4), e434-e460. doi:10.1097/MJT.0000000000001402

Chaccour, C., et al. (2020). The effect of early treatment with ivermectin on viral load, symptoms and humoral response in patients with non-severe COVID-19: a pilot, double-blind, placebo-controlled, randomized clinical trial. EClinical Med. 2021;32:100720. doi: 10.1016/j.eclinm.2020.100720

Elgazzar, A., et al. (2020). Efficacy and safety of ivermectin for treatment and prophylaxis of covid-19 pandemic. Res Square. 2020. doi: 10.21203/rs.3.rs-100956/v2.Preprint. (Retracted)

Friedrich, J. O., et al. (2007). Inclusion of zero total event trials in meta-analyses maintains analytic consistency and incorporates all available data. BMC Medical Research Methodology, 7(1), 5. doi:10.1186/1471-2288-7-5

Galan, L. E. B., et al. (2021). Phase 2 randomized study on chloroquine, hydroxychloroquine or ivermectin in hospitalized patients with severe manifestations of SARS-CoV-2 infection. Pathogens and global health, 115(4), 235-242. doi:10.1080/20477724.2021.1890887

Gonzalez, J. L. B., et al. (2021). Efficacy and safety of Ivermectin and Hydroxychloroquine in patients with severe COVID-19. A randomized controlled trial. medRxiv, 2021.2002.2018.21252037. doi:10.1101/2021.02.18.21252037

Hashim, H. A., et al. (2020). Controlled randomized clinical trial on using Ivermectin with Doxycycline for treating COVID-19 patients in Baghdad, Iraq. medRxiv, 2020.2010.2026.20219345. doi:10.1101/2020.10.26.20219345

Lee, Y. H. (2018). An overview of meta-analysis for clinicians. The Korean journal of internal medicine, 33(2), 277-283. doi:10.3904/kjim.2016.195

López-Medina, E., et al. (2021). Effect of Ivermectin on Time to Resolution of Symptoms Among Adults With Mild COVID-19: A Randomized Clinical Trial. JAMA, 325(14), 1426-1435. doi:10.1001/jama.2021.3071

Mahmud, R., et al. (2021). Ivermectin in combination with doxycycline for treating COVID-19 symptoms: a randomized trial. Journal of International Medical Research, 49(5), 03000605211013550. doi:10.1177/03000605211013550

Mohan, A., et al. (2021). Ivermectin in mild and moderate COVID-19 (RIVET-COV): a randomized, placebo-controlled trial. Res Square. doi: 10.21203/rs.3.rs-191648/v1.Preprint.

Murad, M. H., et al. (2015). The Process of a Systematic Review and Meta-analysis. In G. Guyatt, D. Rennie, M. O. Meade, & D. J. Cook (Eds.), Users' Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice, 3rd ed. New York, NY: McGraw-Hill Education.

Okumus, N., et al. (2021). Evaluation of the effectiveness and safety of adding ivermectin to treatment in severe COVID-19 patients. BMC infectious diseases, 21(1), 411-411. doi:10.1186/s12879-021-06104-9

Petkov S. (2021). Multicenter, Randomized, Double-Blind, Placebo-Controlled Study Investigating Efficacy, Safety and Tolerability of Ivermectin HUVE-19 in Patients with Proven SARS-CoV-2 Infection (Covid-19) and Manifested Clinical Symptoms. Available at: https://www.clinicaltrialsregister.eu/ctr-search/trial/2020-002091-12/BG.

Popp, M., et al. (2021). Ivermectin for preventing and treating COVID-19. The Cochrane database of systematic reviews, 7(7), CD015017-CD015017. doi:10.1002/14651858.CD015017.pub2

Ravikirti, et al. (2021). Ivermectin as a potential treatment for mild to moderate COVID-19 – A double blind randomized placebo-controlled trial. medRxiv, 2021.2001.2005.21249310. doi:10.1101/2021.01.05.21249310

Shahbaznejad, L., et al. (2021). Effects of Ivermectin in Patients With COVID-19: A Multicenter, Double-blind, Randomized, Controlled Clinical Trial. Clinical Therapeutics, 43(6), 1007-1019. doi:10.1016/j.clinthera.2021.04.007

Shakhsi Niaee. M., et al. (2021). Ivermectin as an adjunct treatment for hospitalized adult COVID-19 patients: A randomized multi-center clinical trial. Asian Pac J Trop Med;14:266-73

Walker, E., et al. (2008). Meta-analysis: Its strengths and limitations. Cleve Clin J Med, 75(6), 431-439. doi:10.3949/ccjm.75.6.431


Please Login or Join to leave comments.