Enhancing Mathematical Reasoning in LLMs with Background Operators

TL;DR

This paper introduces background operators and a Prolog-based approach to improve mathematical reasoning in large language models, achieving high accuracy and expanding solution coverage through self-training and data augmentation.

Contribution

It presents a novel method using background mathematical predicates and Prolog solutions, combined with self-training, to enhance reasoning capabilities in LLMs.

Findings

Achieved 84.6% accuracy with self-training on the cross-validated set.

Improved solution coverage by incorporating background predicates into prompts.

Successfully generated new, fully computable solutions for unseen problems.

Abstract

We propose utilizing background operators for mathematical reasoning in large language models (LLMs). To achieve this, we define a set of fundamental mathematical predicates as the basic building blocks. For each mathematical problem, we develop a Prolog solution that includes problem-specific predicates and intermediate predicates derived from these background operators, ensuring that each solution adheres to the defined operator set. We introduce the MATH-Prolog corpus, which is derived from the counting and probability categories of the MATH corpus. For efficient data augmentation, we apply K-fold cross-validated self-training. This method incrementally generates new Prolog solutions for each fold, incorporating those verified as correct into the training set throughout the model training process. Our experimental results demonstrate that 5-fold crossvalidated self-training effectively identifies new, accurate Prolog solutions, achieving an accuracy of 84.6% on the cross-validated set, and 84.8% on the test set during fine-tuning the Meta-Llama-3.1-8B-Instruct model. This approach successfully uncovers new solutions with fully computable inference steps for previously unseen problems. Additionally, incorporating the background mathematical predicates into the prompt enhances solution coverage.