Using Large Language Models to Study Mathematical Practice

TL;DR

This paper uses large language models to analyze a large corpus of mathematics papers, aiming to understand mathematical explanations and practices, and evaluates the potential of AI tools in philosophy of mathematics research.

Contribution

It introduces a novel corpus analysis approach using LLMs to study mathematical explanations, providing new insights and evaluating AI's role in philosophy of mathematical practice.

Findings

Mathematicians frequently make claims about explanation.

Explanatory practices vary across mathematical subjects.

AI tools can effectively analyze large datasets of mathematical texts.

Abstract

The philosophy of mathematical practice (PMP) looks to evidence from working mathematics to help settle philosophical questions. One prominent program under the PMP banner is the study of explanation in mathematics, which aims to understand what sorts of proofs mathematicians consider explanatory and what role the pursuit of explanation plays in mathematical practice. In an effort to address worries about cherry-picked examples and file-drawer problems in PMP, a handful of authors have recently turned to corpus analysis methods as a promising alternative to small-scale case studies. This paper reports the results from such a corpus study facilitated by Google's Gemini 2.5 Pro, a model whose reasoning capabilities, advances in hallucination control and large context window allow for the accurate analysis of hundreds of pages of text per query. Based on a sample of 5000 mathematics papers from arXiv.org, the experiments yielded a dataset of hundreds of useful annotated examples. Its aim was to gain insight on questions like the following: How often do mathematicians make claims about explanation in the relevant sense? Do mathematicians' explanatory practices vary in any noticeable way by subject matter? Which philosophical theories of explanation are most consistent with a large body of non-cherry-picked examples? How might philosophers make further use of AI tools to gain insights from large datasets of this kind? As the first PMP study making extensive use of LLM methods, it also seeks to begin a conversation about these methods as research tools in practice-oriented philosophy and to evaluate the strengths and weaknesses of current models for such work.