Masks and COVID as a Case Study on the Evolution of Science in a Crisis
Meta-analyses were used extensively and repeatedly to assess the vast numbers of papers generated in response to the COVID pandemic. We will use meta-analyses conducted to study masking as the basis for evaluating three things: 1) variability in methodology employed in these meta-analyses, 2) the level of bias in the design and conduct of meta-analyses, 3) the evolution of our state of understanding of the efficacy of masking over time.
What is the context of this research?
The 14,000 papers published every day have overwhelmed our ability to distill the scientific literature. Meta-analysis has come to play an increasingly important role in the interpretation of science. This was particularly true during the COVID pandemic, which generated 384,000 papers and an urgent need assess them. As a result, it has created a unique opportunity for us to examine the process of meta-analysis.
Pub Med lists more than 3,558 meta-analyses on COVID, including 53 related to masks. This number of papers gives us a unique opportunity to examine how differences in methodology influence the conclusions of meta-analyses and how our understanding of the scientific evidence evolved over the course of the pandemic.
What is the significance of this project?
Interpretation of a 2023 Cochrane study as saying masks don’t work prompted Cochrane’s Editor-in-Chief to take the extraordinary step of releasing a statement rejecting this as an “inaccurate and misleading interpretation”. Most experts on airborne transmission agree with her, but their voices need to be heard. This project will use the existing meta-analyses on this topic to reconstruct the evolution of our understanding of mask efficacy over the course of the pandemic. What can we know, what did we know, and when could we have known it?
The history of the pandemic is being written now and will define our response to the next pandemic. This approach can be used to systematically evaluate that history on any issue and inform the management of future crises.
What are the goals of the project?
The goal for the project is to understand how the evidence on the effectiveness of masks and related public health interventions to control the spread of COVID evolved over the course of the pandemic and the degree to which concurrent meta-analyses accurately reflected that evidence. To do this we will create a database of relevant studies along with the characteristics of those studies and relevant statistics drawn from them.
Factors considered will include the type and fit of masks, the setting, the extent of use, and disease prevalence. The protocol for meta-analysis will be released for ongoing feedback from the expert community. We will examine how each new study shifted our understanding and compare our findings to those of meta-analyses released at those times.
Implementation Plan
All meta-analyses related to the efficacy of masking, particularly as it relates to COVID, will be identified from the PubMed database and the preprint servers. After assembling all relevant meta-analyses, we will:
- Determine the differences in the protocol and methods used in each meta-analysis to:
- Identify relevant studies
- Select studies to analyze,
- Account for potential bias and confounding,
- Select key data points,
- Combine data,
- Account for potential bias and confounding.
- Build a database of key statistics from the original studies along with relevant covariates including
- Effect size estimates with confidence intervals or other indicators of standard error.,
- Covariates considered and relevant effect size estimate,
- Indicators of potential bias,
- Publication date.
- Define key questions and pool data using standard meta-analytic methods. To do this, we will assemble a panel with expertise in:
- Meta-analysis,
- Design and implementation of intervention studies,
- Infectious disease transmission
- Airborne particle transport.
- Evaluate the individual meta-analyses by:
- Repeating each meta-analysis using all studies available up to and including the cutoff date for that study based on best available methods,
- Identifying any differences between the meta-analysis and the repeat meta-analysis,
- Determining the reasons for any discrepancies.
Budget
The bulk of the budget will be used to pay a project manager, a research fellow, and a principal investigator with expertise in meta-analysis.
Technical support includes setting up and maintaining necessary websites and data analysis tools. Our work will be amplified by consulting with experts in epidemiology, meta-analysis, and science communications.
Overhead (at 5%) funds the basic business processes required to operate a nonprofit organization.
