Next-Token Prediction Task Assumes Optimal Data Ordering for LLM Training in Proof Generation

TL;DR

This paper shows that training large language models for proof generation is significantly improved when proofs are ordered sequentially, with an 11% success rate increase, highlighting the importance of data ordering in learning efficiency.

Contribution

It introduces the concept of intuitively sequential data order for proof training and demonstrates its effectiveness through experiments on logic and math proofs.

Findings

Training with sequential proof order improves success rates by up to 11%.

Suboptimal proof ordering is common in existing datasets, affecting model performance.

A significant portion of advanced math proofs suffer from order issues, impacting learning.

Abstract

In the field of large language model (LLM)-based proof generation, despite extensive training on large datasets such as ArXiv, LLMs still exhibit only modest performance on proving tasks of moderate difficulty. We believe that this is partly due to the widespread presence of suboptimal ordering within the data for each proof used in training. For example, published proofs often follow a purely logical order, where each step logically proceeds from the previous steps based on the deductive rules. This order is designed to facilitate the verification of the proof's soundness, rather than to help people and models learn the discovery process of the proof. In proof generation, we argue that the optimal order for one training data sample occurs when the relevant intermediate supervision for a particular proof step in the proof is always positioned to the left of that proof step. We call such order the intuitively sequential order. We validate our claims using two tasks: intuitionistic propositional logic theorem-proving and digit multiplication. Our experiments verify the order effect and provide support for our explanations. We demonstrate that training is most effective when the proof is in the intuitively sequential order. Moreover, the order effect and the performance gap between models trained on different data orders can be substantial -- with an 11 percent improvement in proof success rate observed in the propositional logic theorem-proving task, between models trained on the optimal order compared to the worst order. Lastly, we define a common type of order issue in advanced math proofs and find that 17.3 percent of theorems with nontrivial proofs in the first two chapters of a widely used graduate-level mathematics textbook suffer from this issue. A detailed list of those proofs is provided in the appendix.