Epistemic Uncertainty for Test-Time Discovery

TL;DR

This paper introduces UG-TTT, a method that measures epistemic uncertainty via ensemble disagreement to improve exploration in scientific discovery tasks using language models.

Contribution

It proposes a novel ensemble-based uncertainty measure incorporated into policy training to enhance discovery and diversity in scientific problem-solving.

Findings

UG-TTT increases maximum reward in three out of four benchmarks.

It maintains higher solution diversity compared to baseline methods.

A regularizer is crucial for sustaining exploration behavior.

Abstract

Automated scientific discovery using large language models relies on identifying genuinely novel solutions. Standard reinforcement learning penalizes high-variance mutations, which leads the policy to prioritize familiar patterns. As a result, the maximum reward plateaus even as the average reward increases. Overcoming this limitation requires a signal that distinguishes unexplored regions from intrinsically difficult problems. This necessitates measuring disagreement across independently adapted weight hypotheses rather than relying on a single network's confidence. UG-TTT addresses this challenge by maintaining a small ensemble of low-rank adapters over a frozen base model. The per-token disagreement, quantified as the mutual information between ensemble predictions and weight hypotheses, isolates epistemic uncertainty and identifies positions where insufficient coverage leads to adapter divergence rather than intrinsic problem difficulty. This measure is incorporated as an exploration bonus into the policy gradient, directing the policy toward positions where persistent adapter disagreement signals low training coverage, the same frontier where genuine discovery is possible. A nuclear norm regularizer ensures the adapters remain distinct from one another, thereby preserving the exploration signal throughout training. Across four scientific discovery benchmarks, UG-TTT increases the maximum reward on three tasks, maintains substantially higher solution diversity, and an ablation study confirms that the regularizer is essential for sustaining this behavior.