TaoBench: Do Automated Theorem Prover LLMs Generalize Beyond MathLib?

TL;DR

This paper introduces TaoBench, a new benchmark based on Terence Tao's Analysis I, to evaluate the robustness of automated theorem proving systems across different definitional frameworks, revealing significant generalization gaps.

Contribution

The work presents TaoBench, a novel benchmark that tests ATP models on problems constructed from scratch, highlighting their limited ability to generalize beyond standard Mathlib definitions.

Findings

State-of-the-art ATP models perform 26% worse on TaoBench compared to Mathlib.

Performance drops are mainly due to definitional framework differences, not task difficulty.

TaoBench provides a foundation for developing provers aligned with research mathematics.

Abstract

Automated theorem proving (ATP) benchmarks largely consist of problems formalized in MathLib, so current ATP training and evaluation are heavily biased toward MathLib's definitional framework. However, frontier mathematics is often exploratory and prototype-heavy, relying on bespoke constructions that deviate from standard libraries. In this work, we evaluate the robustness of current ATP systems when applied to a novel definitional framework, specifically examining the performance gap between standard library problems and bespoke mathematical constructions. We introduce TaoBench, an undergraduate-level benchmark derived from Terence Tao's Analysis I, which formalizes analysis by constructing core mathematical concepts from scratch, without relying on standard Mathlib definitions, as well as by mixing from-scratch and MathLib constructions. For fair evaluation, we build an agentic pipeline that automatically extracts a compilable, self-contained local environment for each problem. To isolate the effect of definitional frameworks, we additionally translate every problem into a mathematically equivalent Mathlib formulation, yielding paired TaoBench-Mathlib statements for direct comparison. While state-of-the-art ATP models perform capably within the MathLib framework, performance drops by an average of roughly 26% on the definitionally equivalent Tao formulation. This indicates that the main bottleneck is limited generalization across definitional frameworks rather than task difficulty. TaoBench thus highlights a gap between benchmark performance and applicability, and provides a concrete foundation for developing and testing provers better aligned with research mathematics.