Code Soliloquies for Accurate Calculations in Large Language Models

TL;DR

This paper introduces a novel stateful prompt design for GPT-4 that improves the accuracy of synthetic physics conversations by integrating internal code soliloquies and Python scripting, enhancing dataset quality for calculation-heavy subjects.

Contribution

The paper presents a new prompt architecture that enables GPT-4 to perform internal mathematical reasoning and code execution, improving synthetic dialogue quality for complex subjects.

Findings

Enhanced dataset quality for calculation-intensive subjects

GPT-4 effectively scripts and uses Python for accurate computations

Preliminary evaluations show improved response accuracy

Abstract

High-quality conversational datasets are crucial for the successful development of Intelligent Tutoring Systems (ITS) that utilize a Large Language Model (LLM) backend. Synthetic student-teacher dialogues, generated using advanced GPT-4 models, are a common strategy for creating these datasets. However, subjects like physics that entail complex calculations pose a challenge. While GPT-4 presents impressive language processing capabilities, its limitations in fundamental mathematical reasoning curtail its efficacy for such subjects. To tackle this limitation, we introduce in this paper an innovative stateful prompt design. Our design orchestrates a mock conversation where both student and tutorbot roles are simulated by GPT-4. Each student response triggers an internal monologue, or `code soliloquy' in the GPT-tutorbot, which assesses whether its subsequent response would necessitate calculations. If a calculation is deemed necessary, it scripts the relevant Python code and uses the Python output to construct a response to the student. Our approach notably enhances the quality of synthetic conversation datasets, especially for subjects that are calculation-intensive. Our preliminary Subject Matter Expert evaluations reveal that our Higgs model, a fine-tuned LLaMA model, effectively uses Python for computations, which significantly enhances the accuracy and computational reliability of Higgs' responses. Code, models, and datasets is available at https://github.com/luffycodes/Tutorbot-Spock-Phys.