Large-Scale Validation of Hypothesis Generation Systems via Candidate Ranking

TL;DR

This paper introduces a scalable numerical evaluation framework for hypothesis generation systems, validated through real-world experiments that led to a novel scientific discovery about HIV-related neurodegeneration.

Contribution

It presents a new validation method for hypothesis generation systems using thousands of hypotheses and novel metrics, aligning automated validation with research goals.

Findings

Validation method correlates with real-world research outcomes

Discovered a new link between HAND and DDX3 through automated hypothesis ranking

Framework enables scalable, objective evaluation of hypothesis generation systems

Abstract

The first step of many research projects is to define and rank a short list of candidates for study. In the modern rapidity of scientific progress, some turn to automated hypothesis generation (HG) systems to aid this process. These systems can identify implicit or overlooked connections within a large scientific corpus, and while their importance grows alongside the pace of science, they lack thorough validation. Without any standard numerical evaluation method, many validate general-purpose HG systems by rediscovering a handful of historical findings, and some wishing to be more thorough may run laboratory experiments based on automatic suggestions. These methods are expensive, time consuming, and cannot scale. Thus, we present a numerical evaluation framework for the purpose of validating HG systems that leverages thousands of validation hypotheses. This method evaluates a HG system by its ability to rank hypotheses by plausibility; a process reminiscent of human candidate selection. Because HG systems do not produce a ranking criteria, specifically those that produce topic models, we additionally present novel metrics to quantify the plausibility of hypotheses given topic model system output. Finally, we demonstrate that our proposed validation method aligns with real-world research goals by deploying our method within Moliere, our recent topic-driven HG system, in order to automatically generate a set of candidate genes related to HIV-associated neurodegenerative disease (HAND). By performing laboratory experiments based on this candidate set, we discover a new connection between HAND and Dead Box RNA Helicase 3 (DDX3). Reproducibility: code, validation data, and results can be found at sybrandt.com/2018/validation.