- - While vaccine efficacy studies conducted under optimal and controlled research conditions are necessary for vaccine licensing, vaccine effectiveness studies carried out in the real world settings are key to determining whether a vaccine program actually helps people.
- - Vaccine effectiveness provides critical information that vaccine efficacy cannot provide to inform rational policy decisions about new vaccines.
- - Observational study designs (e.g., cohort studies, case-control studies) are usually applied to evaluate vaccine effectiveness.
- - Current peer-reviewed and non-peer reviewed vaccine effectiveness studies unanimously found the Pfizer/BioNTech mRNA vaccine was effective against COVID-19 in the real world settings, and likely to be effective against the SARS-CoV-2 B.1.1.7 variant of concern.
- - The Oxford/AstraZeneca chimpanzee adenovirus-vectored vaccine was also shown in one preprint to be effective in reducing COVID-19-related hospitalizations.
“The United States reached the staggering milestone on Monday [February 22, 2021] ... The nation’s total virus toll is higher than in any other country in the world, and it means that more Americans have died from Covid-19 than they did on the battlefields of World War I, World War II and the Vietnam War combined.”
By February 22, 2021, the global death toll related to COVID-19 had exceeded 2.46 million, with the United States (US) leading all countries around the globe with over 500,000 deaths. This number is more than twice the number of the next closest country -- Brazil, which has over 240,000 deaths (Johns Hopkins Coronavirus Resource Center).
To end such horrifying tragedy, achieving herd immunity through effective vaccination (See past OE Original: The Sobering Story of SARS-CoV-2 Seroprevalence: What You May Have Heard About Herd Immunity) is considered the safest way, as opposed to through other paths such as natural infection (Fontanet et al., 2020).
Several vaccines have been licensed to fight against COVID-19, including BNT162b2 (developed by Pfizer/BioNTech), mRNA-1273 (developed by Moderna), AZD1222 (developed by Oxford/AstraZeneca) (See past OE Original: Vaccines for COVID-19: Pfizer, Moderna, and AstraZeneca Race to Finish Line). In order to be licensed, results on vaccine efficacy along with safety outcomes as measured in randomized controlled trials (RCTs) are essential.
Despite the extreme importance, results from vaccine efficacy RCTs were often limited, as Clemens et al. (1996) argued, due to the factors such as “constrained study populations, idealized conditions, narrow biological outcomes, and restrictive analytic strategies, [which] may yield findings that fail to capture the real world of public health practice.” For instance, a meta-analysis showed that the efficacy of rotavirus vaccine, which was obtained from trials conducted in children in high-income countries, was higher than the vaccine effectiveness later observed among children from low- and middle-income real world settings (Velázquez et al., 2017).
As a result, assessing vaccine performance via post-licensure evaluations of vaccine effectiveness under “real world” conditions is necessary and critical. In this OE Original, we introduce the concepts of vaccine efficacy versus vaccine effectiveness, look at the reasons why vaccine effectiveness studies are necessary, demonstrate how vaccine effectiveness is usually calculated, and examine current available evidence on the effectiveness of current authorized vaccines against COVID-19.
What is vaccine effectiveness?
Verani et al. (2017) provided detailed definitions for vaccine efficacy and vaccine effectiveness. Vaccine efficacy refers to “the percentage by which the rate of the target disease among those who are vaccinated according to the recommended schedule is reduced compared to the rate in similar unvaccinated persons. This is generally measured in the context of a placebo-controlled randomized trial as the “per protocol” efficacy (i.e., excluding persons who did not receive the recommended schedule), because the intention is to establish the biologic performance capacity of the product under optimal conditions” (Verani et al., 2017).
Vaccine effectiveness is defined as “the same percent reduction in the rate of disease as efficacy, but in the context of routine, real-world use of the vaccine. Vaccine effectiveness may be similar to the efficacy as measured in clinical trials. However, it often differs in magnitude because in routine use the population vaccinated includes some who may have a less robust immune response, and program implementation (e.g., cold-chain maintenance, dosing schedules) is more variable than in clinical trial settings” (Verani et al., 2017).
By comparing the above definitions, we can simply understand vaccine efficacy as the degree to which a vaccine prevents a disease obtained from optimal and controlled research conditions, whereas vaccine effectiveness looks at how well a vaccine performs in real world settings. Just as Fedson (1998) stated, vaccine efficacy answers the important question about whether a vaccine works, while vaccine effectiveness lets us determine whether a vaccine actually helps people in the real world.
Why do we need to know vaccine effectiveness?
Overall, vaccine effectiveness can provide critical information that vaccine efficacy cannot; helping to inform rational policy decisions about new vaccines.
Vaccine efficacy is usually obtained from phase III RCTs, also known as vaccine efficacy trials (See past OE Original: COVID-19 Vaccines: The Current Candidates in Phase III Trials). In a vaccine efficacy trial, every condition is supposed to be ideal in order to measure the intrinsic vaccine protection which is “the direct reduction of host susceptibility to infection conferred by a full course of vaccination” with adequate statistical power (Clemens et al., 1996). To determine whether a vaccine shows statistically significant protection, trialists often select those participants who are known to be at risk of the infection and who are also known to be responsive to the vaccine. They also carefully choose the vaccine dose, schedule vaccinations for maximum immunogenicity, and adopt narrowly defined efficacy outcomes (e.g., objective outcomes) (Clemens et al., 1996; Fedson, 1998; McNeil, 2020). However, there are many unknowns when using a vaccine in the real world. One example from the current COVID-19 vaccination effort is that we may have to change the schedule of vaccination due to the constraint on vaccine supply (See past OE Original: COVID-19 Vaccines: What Does the Evidence Say about Delaying the Second Dose?). Previous efficacy trials provide little evidence on this issue, therefore vaccine effectiveness studies are necessary.
Lack of generalizability is another major concern for vaccine efficacy trials, which are usually conducted in a highly selected patient population. Results from a homogeneous population may not be generalized to wider heterogeneous patient populations. Lack of generalizability is a common issue for RCTs. A systematic review showed that the generalizability of evidence was likely to be limited in RCTs published in leading medical journals, including the British Medical Journal (BMJ), JAMA, Lancet, and New England Journal of Medicine (NEJM) (Malmivaara, 2019). Observing vaccine effectiveness in real life conditions allows us to determine whether the vaccine is effective in heterogeneous populations.
Vaccine effectiveness can also provide us with important information on whether a vaccine program is cost-effective. For a vaccine program to be promoted and sustainable, we need to identify the balance point between the burden of disease and cost of vaccination. The greater the disease burden, and the greater the proportion of disease burden can be prevented by a vaccine, the more cost-effective the vaccine program is (McNeil, 2020).
How do we usually measure vaccine effectiveness?
Although RCTs could be used to measure vaccine effectiveness, we usually apply observational studies (e.g., cohort studies, case-control studies) (Clemens et al., 1996; Torvaldsen et al., 2002; US Centers for Disease Control and Prevention, 2016). One reason for this is that it is no longer ethical to perform placebo-controlled RCTs once a vaccine is approved and authorized by the licensing body (US Centers for Disease Control and Prevention, 2016). Assigning subjects to placebo while knowing a vaccine has been proven effective in RCTs violates the no harm principle. Additionally, it is often not feasible to carry out an RCT if the outcome of interest is less common (US Centers for Disease Control and Prevention, 2016).
The standard formula to calculate vaccine effectiveness (VE%) is shown below (Torvaldsen et al., 2002).
VE% = (ARU - ARV) / ARU X 100 = [1 - (ARV/ARU)] x 100
ARU: Attack rate in the unvaccinated group
ARV: Attack rate in the vaccinated group
Attack rate, usually used for acute outbreak of disease, is calculated by dividing the number of new cases of disease in a population at risk by the number of people in the population at risk.
ARV/ARU is equivalent to relative risk (RR), which can be obtained from cohort studies. If case-control studies are used, RR can be estimated by odds ratio (OR).
What do we know about the effectiveness of COVID-19 vaccine?
Evidence regarding the effectiveness of COVID-19 vaccine is emerging. Israel has vaccinated close to 50% of its population (COVID in Israel: How Many Have Already Been Vaccinated), which allows for the extracting of real world signals.
On February 24th, 2021, a major study conducted by Dagan et al. (2021), which investigated the effectiveness of the Pfizer/BioNTech mRNA COVID-19 vaccine (BNT162b2) in the Israeli nationwide mass vaccination setting, was published in the NEJM. The design of the vaccine effectiveness study was a matched case-control study. About 600,000 vaccine recipients were matched with unvaccinated individuals in a 1:1 ratio on a number of demographic and clinical factors, such as age, sex, geostatistical areas, history of influenza vaccination, pregnancy, etc.
Dagan et al. (2021) assessed vaccine effectiveness at day 14 through day 20 after the first dose, at day 21 to day 27 after the first dose, and at least seven days after the second dose of the Pfizer/BioNTech vaccine for five outcomes of interest, including i) SARS-CoV-2 infection confirmed by positive PCR test; ii) Symptomatic COVID-19; iii) Hospitalization due to COVID-19; iv) Severe COVID-19; and v) Death due to COVID-19. The vaccine effectiveness was presented as VE% described above.
The vaccine effectiveness study done by Dagan et al. (2021) suggested that the Pfizer/BioNTech mRNA vaccine was effective in the real world conditions, which corroborated the findings from the previous vaccine efficacy study (Polack et al., 2020). Detailed results are shown in Table 1. Additionally, in terms of the SARS-CoV-2 B.1.1.7 variant or concern (see past OE Original: COVID-19 Variants of Concern: Will There Be A Third Wave?), Dagan et al. (2021) were not able to draw definitive conclusions for the effectiveness of the Pfizer/BioNTech vaccine against the B.1.1.7 variant because they only estimated the average vaccine effectiveness over multiple strains. However, Dagan et al. (2021) inferred that the Pfizer/BioNTech mRNA vaccine was likely to be effective against B.1.1.7 because of the plateaus observed in the curves of cumulative incidence of bad outcomes due to multiple SARS-CoV-2 strains, among which B.1.1.7 has become an increasingly dominant SARS-CoV-2 strain in Israel.
Table 1: Vaccine effectiveness results from Dagan et al. (2021)
VE%; 95% CI at 7 or more days after 2nd dose
VE%; 95% CI at days 14 to 20 after 1st dose
VE%; 95% CI at days 21 to 27 after 1st dose
VE%; 95% CI long-term follow-up (e.g., 6 months)
Hospitalization due to COVID-19
Death due to COVID-19
VE%: Vaccine Effectiveness%; 95% CI: 95% Confidence Interval
Another observational study, published on February 22nd, 2021 in The Lancet, focused on the effectiveness of Pfizer/BioNTech COVID-19 vaccine among a retrospective cohort of over 9,000 health care workers in Israel (Amit et al., 2021). The data suggested substantial reductions in the incidences of SARS-CoV-2 infection and symptomatic COVID-19 after the first dose of Pfizer/BioNTech mRNA vaccine in a health care setting (Amit et al., 2021). Specifically, comparing vaccine recipients vs. those unvaccinated, the adjusted AE% in SARS-CoV-2 infection were 30% [95% confidence interval (CI): 2% to 50%] and 75% (95% CI: 72% to 84%) for days 1 to 14 and days 15 to 28 post the first dose of vaccine, respectively (Amit et al., 2021). For symptomatic COVID-19, the adjusted AE% were 47% (95% CI: 2% to 50%) and 75% (95% CI: 72% to 84%) for days 1 to 14 and days 15 to 28 post the first dose of vaccine, respectively (Amit et al., 2021).
We also identified three more non-peer reviewed vaccine effectiveness studies on MedRxiv, all of which were conducted in Israel (Chodick et al., 2021; Levine-Tiefenbrun et al., 2021; Petter et al., 2021). Chodick et al. (2021) assessed the short-term (i.e., days 13 to 24 after first dose) effectiveness of the first dose of the Pfizer/BioNTech COVID-19 vaccine in a cohort (with over 500,000 individuals) from another health care organization in Israel using the retrospective cohort study design. Chodick et al. (2021) found a 51.4% (95% CI: 7.2% to 78%) reduction in the incidence of SARS-CoV-2 infection confirmed by PCR testing at days 13 to 24 after the first dose of Pfizer/BioNTech vaccine, compared to that during the first 12 days. This finding was similar to Dagan’s et al. (2021) result -- a 46% reduction in the incidence of SARS-CoV-2 infection at days 14 to 20 after the 1st dose of Pfizer/BioNTech vaccine.
Petter et al. (2021) and Levine-Tiefenbrun et al. (2021) investigated whether the vaccination of Pfizer/BioNTech mRNA COVID-19 vaccine could reduce the viral load in the real world conditions. Analyzing the Ct value distribution of over 16,000 positive quantitative PCR tests, the model established by Petter et al. (2021) showed that Pfizer/BioNTech vaccine reduced the viral load by 1.6- to 20-fold in individuals who are positive for SARS-CoV-2. Consistent with results from Petter et al. (2021), Levine-Tiefenbrun et al. (2021) found that the viral load was reduced about four-fold among patients whose infections happened 12 to 28 days after the first dose of Pfizer/BioNTech vaccine.
In addition to the abovementioned Israeli vaccine effectiveness studies, two non-peer reviewed studies from the United Kingdom (UK) also released early data to inform the effectiveness of COVID-19 vaccination (Hall et al., 2021; Vasileiou et al., 2021).
The SIREN study (Sarscov2 Immunity and REinfection EvaluatioN, Hall et al., 2021) is a prospective cohort study investigating the effectiveness of the Pfizer/BioNTech mRNA vaccine in health care workers in England during a period when the dominant variant in circulation was B.1.1.7. Results from the SIREN study suggested that the Pfizer/BioNTech COVID-19 vaccine effectively prevents SARS-CoV-2 infection among health care workers. VE% against SARS-CoV-2 infection in the overall study population (N = 23,324) were 70% (95% CI: 53 to 87%) and 85% (95% CI: 74 to 96%) at 21 days after the first dose of the Pfizer/BioNTech vaccine and seven days after the second dose, respectively (Hall et al., 2021).
Vasileiou et al. (2021) also conducted a prospective cohort study to investigate the effectiveness of Pfizer/BioNTech mRNA vaccine and Oxford/AstraZeneca chimpanzee adenovirus-vectored vaccine using the Early Pandemic Evaluation and Enhanced Surveillance of COVID-19 database consisting of records of about 5.4 million people in Scotland. The hospitalization due to COVID-19 was recorded. Data indicated that a single dose of either the Pfizer/BioNTech mRNA or Oxford/AstraZeneca chimpanzee adenovirus-vectored vaccine led to substantial reductions in the incidence of hospitalization due to COVID-19 (Vasileiou et al., 2021). For instance, the effectiveness of Pfizer/BioNTech vaccine in reducing COVID-19-related hospitalization were 60% (95% CI: 50% to 68%) and 72% (95% CI: 62% to 79%) for days 14 to 20 and days 21 to 27 post the first dose of vaccine, respectively (Vasileiou et al., 2021). These data were consistent with the data from Dagan’s et al. (2021) Israeli study. Additionally, the effectiveness of Oxford/AstraZeneca vaccine in reducing hospitalization were 74% (95% CI: 66% to 81%) and 84% (95% CI: 72% to 90%) for days 14 to 20 and days 21 to 27 post the first dose of vaccine, respectively (Vasileiou et al., 2021). Vasileiou et al., (2021) also reported the VE% at more than 42 days after the first dose of the Pfizer/BioNTech vaccine (64%; 49% to 75%).
Acquiring the efficacy data of a vaccine from an optimal and controlled research condition is absolutely necessary for the licensing processes by health authorities, but not adequate to inform the health policy decisions about whether the vaccine actually helps people in the real world settings and whether the benefits of the vaccines outweigh the burden of the disease. As a result, monitoring vaccine effectiveness under real world conditions is critical.
Evidence with regard to the effectiveness of COVID-19 vaccines, especially the effectiveness of the Pfizer/BioNTech mRNA COVID-19 vaccine, have started to emerge. Current peer-reviewed and non-peer reviewed vaccine effectiveness studies unanimously found the Pfizer/BioNTech vaccine was effective against COVID-19 in the general population or among health care workers, and likely to be effective against the SARS-CoV-2 B.1.1.7 variant of concern. Although being investigated in only one preprint, Oxford/AstraZeneca chimpanzee adenovirus-vectored vaccine was also shown to be effective in reducing COVID-19-related hospitalizations. These positive findings strengthen our expectation that we will have the COVID-19 pandemic under control by reaching herd immunity via effective vaccination.
Amit, S., et al. (2021). Early rate reductions of SARS-CoV-2 infection and COVID-19 in BNT162b2 vaccine recipients. The Lancet. doi:10.1016/S0140-6736(21)00448-7
Chodick, G., et al. (2021). The effectiveness of the first dose of BNT162b2 vaccine in reducing SARS-CoV-2 infection 13-24 days after immunization: real-world evidence. medRxiv, 2021.2001.2027.21250612. doi:10.1101/2021.01.27.21250612
Clemens, J., et al. (1996). Evaluating New Vaccines for Developing Countries: Efficacy or Effectiveness? JAMA, 275(5), 390-397. doi:10.1001/jama.1996.03530290060038
Crowcroft, N. S., et al. (2018). A framework for research on vaccine effectiveness. Vaccine, 36(48), 7286-7293. doi:10.1016/j.vaccine.2018.04.016
Dagan, N., et al. (2021). BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting. New England Journal of Medicine. doi:10.1056/NEJMoa2101765
Fedson, D. (1998). Measuring protection: efficacy versus effectiveness. Dev Biol Stand, 95, 195-201.
Fontanet, A., et al. (2020). COVID-19 herd immunity: where are we? Nature Reviews Immunology, 20(10), 583-584. doi:10.1038/s41577-020-00451-5
Hall, VJ., et al. (2021). Effectiveness of BNT162b2 mRNA Vaccine Against Infection and COVID-19 Vaccine Coverage in Healthcare Workers in England, Multicentre Prospective Cohort Study (the SIREN Study). Available at SSRN: https://ssrn.com/abstract=3790399 or http://dx.doi.org/10.2139/ssrn.3790399
Levine-Tiefenbrun, M., et al. (2021). Decreased SARS-CoV-2 viral load following vaccination. medRxiv, 2021.2002.2006.21251283. doi:10.1101/2021.02.06.21251283
Malmivaara, A. (2019). Generalizability of findings from randomized controlled trials is limited in the leading general medical journals. Journal of Clinical Epidemiology, 107, 36-41. doi:10.1016/j.jclinepi.2018.11.014
McNeil, S. (2020). Overview of Vaccine Effi cacy and Vaccine Effectiveness. Retrieved from https://www.who.int/influenza_vaccines_plan/resources/Session4_VEfficacy_VEffectiveness.PDF
Petter, E., et al. (2021). Initial real world evidence for lower viral load of individuals who have been vaccinated by BNT162b2. medRxiv, 2021.2002.2008.21251329. doi:10.1101/2021.02.08.21251329
Polack, F. P., et al. (2020). Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine. New England Journal of Medicine, 383(27), 2603-2615. doi:10.1056/NEJMoa2034577
Torvaldsen, S., et al. (2002). Observational methods in epidemiologic assessment of vaccine effectiveness. Commun Dis Intell Q Rep, 26(3), 451-457.
US Centers for Disease Control and Prevention. (2016). How Flu Vaccine Effectiveness and Efficacy are Measured. Retrieved from https://www.cdc.gov/flu/vaccines-work/effectivenessqa.htm#
Vasileiou E., et al. (2021). Effectiveness of first dose of COVID-19 vaccines against hospital admissions in Scotland: national prospective cohort study of 5.4 million people. Retrieved from: https://www.ed.ac.uk/files/atoms/files/scotland_firstvaccinedata_preprint.pdf
Velázquez, R. F., et al. (2017). Efficacy, safety and effectiveness of licensed rotavirus vaccines: a systematic review and meta-analysis for Latin America and the Caribbean. BMC Pediatr, 17(1), 14. doi:10.1186/s12887-016-0771-y
Verani, J. R., et al. (2017). Case-control vaccine effectiveness studies: Preparation, design, and enrollment of cases and controls. Vaccine, 35(25), 3295-3302. doi:https://doi.org/10.1016/j.vaccine.2017.04.03