Autoformalization in the Wild: Assessing LLMs on Real-World Mathematical Definitions

TL;DR

This paper evaluates how well large language models can autoformalize real-world mathematical definitions from sources like Wikipedia and arXiv, highlighting challenges and proposing strategies to improve their accuracy and reliability.

Contribution

It introduces new datasets for autoformalization, systematically evaluates LLMs on real-world definitions, and explores refinement and grounding strategies to enhance formalization performance.

Findings

Definitions are more challenging than benchmarks like miniF2F.

LLMs struggle with self-correction and library alignment.

Refinement and grounding improve formalization accuracy significantly.

Abstract

Thanks to their linguistic capabilities, LLMs offer an opportunity to bridge the gap between informal mathematics and formal languages through autoformalization. However, it is still unclear how well LLMs generalize to sophisticated and naturally occurring mathematical statements. To address this gap, we investigate the task of autoformalizing real-world mathematical definitions: a critical component of mathematical discourse. Specifically, we introduce two novel resources for autoformalization, collecting definitions from Wikipedia (Def_Wiki) and arXiv papers (Def_ArXiv). We then systematically evaluate a range of LLMs, analyzing their ability to formalize definitions into Isabelle/HOL. Furthermore, we investigate strategies to enhance LLMs' performance including refinement through external feedback from Proof Assistants, and formal definition grounding, where we augment LLMs' formalizations through relevant contextual elements from formal mathematical libraries. Our findings reveal that definitions present a greater challenge compared to existing benchmarks, such as miniF2F. In particular, we found that LLMs still struggle with self-correction, and aligning with relevant mathematical libraries. At the same time, structured refinement methods and definition grounding strategies yield notable improvements of up to 16% on self-correction capabilities and 43% on the reduction of undefined errors, highlighting promising directions for enhancing LLM-based autoformalization in real-world scenarios.