Pipeline for Verifying LLM-Generated Mathematical Solutions

TL;DR

This paper presents a verification pipeline for mathematical solutions generated by large reasoning models, combining automatic and interactive methods to improve accuracy over traditional answer-only checks.

Contribution

It introduces a novel verification pipeline utilizing prompts and proof assistants, enabling more reliable validation of LLM-generated mathematical solutions.

Findings

Low probability of false positives in experiments

Supports formal and informal language solutions

Open-source implementation available

Abstract

With the growing popularity of Large Reasoning Models and their results in solving mathematical problems, it becomes crucial to measure their capabilities. We introduce a pipeline for both automatic and interactive verification as a more accurate alternative to only checking the answer which is currently the most popular approach for benchmarks. The pipeline can also be used as a generator of correct solutions both in formal and informal languages. 3 AI agents, which can be chosen for the benchmark accordingly, are included in the structure. The key idea is the use of prompts to obtain the solution in the specific form which allows for easier verification using proof assistants and possible use of small models ($\le 8B$). Experiments on several datasets suggest low probability of False Positives. The open-source implementation with instructions on setting up a server is available at https://github.com/LogicEnj/lean4_verification_pipeline.