Protocols for Observational Studies: Methods and Open Problems

TL;DR

This paper emphasizes the importance of protocols in observational studies for causal inference, illustrating their value through applications and discussing methodological considerations and open problems.

Contribution

It introduces the concept of protocols for observational studies, compares them with RCT protocols, and discusses methodological considerations and open challenges.

Findings

Protocols improve transparency and rigor in observational studies.

Applications demonstrate the practical value of well-designed protocols.

Discussion highlights key methodological differences from RCTs and open research problems.

Abstract

For learning about the causal effect of a treatment, a randomized controlled trial (RCT) is considered the gold standard. However, randomizing treatment is sometimes unethical or infeasible, and instead an observational study may be conducted. While some aspects of a well designed RCT cannot be replicated in an observational study, one aspect that can is to have a protocol with prespecified hypotheses about prespecified outcomes and a prespecified analysis. We illustrate the value of protocols for observational studies in three applications -- the effect of playing high school football on later life mental functioning, the effect of police seizing a gun when arresting a domestic violence suspect on future domestic violence and the effect of mountaintop mining on health. We then discuss methodologies for observational study protocols. We discuss considerations for protocols that are similar between observational studies and RCTs, and considerations that are different. The considerations that are different include (i) whether the protocol should be specified before treatment assignment is known or after; (ii) how multiple outcomes should be incorporated into the planned analysis and (iii) how subgroups should be incorporated into the planned analysis. We conclude with discussion of a few open problems in the methodology of observational study protocols.