STAR: Statistical Tests with Auditable Results

TL;DR

STAR is a system that uses cryptographic techniques to provide auditable, mathematically guaranteed certificates for statistical tests, aiming to prevent p-hacking and false discoveries in scientific research.

Contribution

It introduces a cryptographically secure, decentralized system for certifying the validity of statistical tests with provable guarantees, enhancing trust in scientific findings.

Findings

Practical implementation using Microsoft SEAL and SPDZ protocols

Effective in real-world scenarios for certifying scientific discoveries

Demonstrates tamper-proof, auditable validation of hypothesis tests

Abstract

We present STAR: a novel system aimed at solving the complex issue of "p-hacking" and false discoveries in scientific studies. STAR provides a concrete way for ensuring the application of false discovery control procedures in hypothesis testing, using mathematically provable guarantees, with the goal of reducing the risk of data dredging. STAR generates an efficiently auditable certificate which attests to the validity of each statistical test performed on a dataset. STAR achieves this by using several cryptographic techniques which are combined specifically for this purpose. Under-the-hood, STAR uses a decentralized set of authorities (e.g., research institutions), secure computation techniques, and an append-only ledger which together enable auditing of scientific claims by 3rd parties and matches real world trust assumptions. We implement and evaluate a construction of STAR using the Microsoft SEAL encryption library and SPDZ multi-party computation protocol. Our experimental evaluation demonstrates the practicality of STAR in multiple real world scenarios as a system for certifying scientific discoveries in a tamper-proof way.