PLanet: Formalizing and Analyzing Assignment Procedures in the Design of Experiments

TL;DR

PLanet introduces a formal language and static analysis tools for designing and analyzing experimental assignment procedures, making assumptions explicit and enabling exploration of causal query testability.

Contribution

It presents a new DSL grounded in matrix algebra for constructing experimental designs, improving expressivity and clarity over existing tools.

Findings

PLanet is the most expressive among existing DSLs.

PLanet's static analysis determines testable causal queries under various assumptions.

Design exploration is facilitated and assumptions are made explicit with PLanet.

Abstract

Experimental designs reflect assumptions about variable relationships that determine what causal queries researchers can answer through the experiment. Accounting for and communicating these assumptions is essential for drawing valid, generalizable conclusions from scientific experiments. Unfortunately, existing experimental design tools elide these details, expecting researchers to reason about design decisions and assumptions on their own. To surface assumptions and enable design exploration, we introduce a grammar of composable operators for constructing experimental assignment procedures grounded in matrix algebra. The PLanet DSL implements this grammar and compiles PLanet programs into constraint satisfaction problems over matrices. Together, PLanet's composable grammar and matrix representation enable a static analysis to determine which causal queries are testable under different assumptions. In an expressivity evaluation, PLanet was the most expressive of existing DSLs. Critical reflections with the authors of these DSLs revealed that PLanet makes design choices explicit without requiring procedural specification. Think-aloud studies showed that PLanet facilitated design exploration and surfaced assumptions researchers may otherwise overlook.