Exploring an LM to generate Prolog Predicates from Mathematics Questions

TL;DR

This paper explores fine-tuning a language model to generate Prolog code from math questions, aiming to improve reasoning accuracy by integrating logic programming with chain-of-thought prompting.

Contribution

It introduces a novel approach of fine-tuning LLaMA7B for Prolog code generation from math questions and evaluates its effectiveness compared to baseline models.

Findings

Prolog code generation outperforms the baseline model.

Combining chain-of-thought with Prolog generation does not significantly improve results.

The Prolog corpus and fine-tuned model are publicly released.

Abstract

Recently, there has been a surge in interest in NLP driven by ChatGPT. ChatGPT, a transformer-based generative language model of substantial scale, exhibits versatility in performing various tasks based on natural language. Nevertheless, large language models often exhibit poor performance in solving mathematics questions that require reasoning. Prior research has demonstrated the effectiveness of chain-of-thought prompting in enhancing reasoning capabilities. Now, we aim to investigate whether fine-tuning a model for the generation of Prolog codes, a logic language, and subsequently passing these codes to a compiler can further improve accuracy. Consequently, we employ chain-of-thought to fine-tune LLaMA7B as a baseline model and develop other fine-tuned LLaMA7B models for the generation of Prolog code, Prolog code + chain-of-thought, and chain-of-thought + Prolog code, respectively. The results reveal that the Prolog generation model surpasses the baseline in performance, while the combination generation models do not yield significant improvements. The Prolog corpus based on GSM8K and the correspondingly finetuned Prolog generation model based on LLaMA7B are released to the research community.