Meta-analysis has a long, well-established history in clinical research and its use has been standardized by the Cochrane Collaborative. Randomized Clinical Trials are particularly well suited to meta-analysis because of their consistency in design and the limited and well-established sources of bias. Observational research involves a broad range of study designs with vastly more sources of bias and the potential for confounding that is theoretically eliminated from a well-designed RCT. This poses a daunting challenge. Addressing the complex questions that require observational research demands that we use all relevant research, but the simple act of combining that research may provoke controversy.
We are creating a novel forum for scientists to engage in open collaborative meta-analysis about science relevant to important public policy decisions. The key question with any new research discussed in our forum will be: How does it shift our understanding given what we already know?
Our Process
This process is being explicitly designed to answer that question in a quantitative way for each selected topic, as described below and summarized in the diagram above. Our process provides for open, ongoing discussion of specific methodological issues that arise in individual studies and in the course of combining research and is organized according to the following steps:
- Define a key question
Policy questions will be broken into a set of discrete, scientific questions that have been or can be addressed by the research community. - Build an initial research library
Assemble all relevant research, organized in accordance with these questions. - Create an initial research community to engage in the collaborative meta-analysis
Identify and engage with an initial group of researchers with relevant expertise. Other researchers can engage as well but must provide basic data on their training and experience. - Evaluate studies
Studies are evaluated with respect to their specific strengths and weaknesses with an eye toward the impact of these attributes on the estimates of effect sizes relevant to the specific research question. - Extract relevant statistics
Identify and extract key statistics from the papers that pertain to the specific research question. - Pool results
Combine these statistics with standard meta-analytic tools, which explicitly consider the limitations and qualifications of each individual study. - Iterate
As new research emerges, the body of research will be reassessed using a Bayesian framework, which uses the existing research to establish a Bayesian prior probability.
What makes this process unique is the fact that it is open and iterative. All the steps are explicitly stated with tools in place for ongoing community feedback. The meta-analysis is never considered final in that it is always open to revision based either on community feedback or new research. Even the authorship is open in that all members of the community who make substantive critiques or suggestions will be acknowledged.