Enhancing LLMs for Physics Problem-Solving using Reinforcement Learning with Human-AI Feedback

TL;DR

This paper introduces a reinforcement learning approach with human and AI feedback to significantly improve large language models' reasoning and accuracy on physics problems, especially in complex arithmetic and conceptual understanding.

Contribution

It presents a novel RL-based training method, RLHAIF, that enhances LLMs' physics reasoning capabilities beyond existing prompt engineering techniques.

Findings

RLHAIF improves LLM performance on physics questions.

MISTRAL-PPO achieves high METEOR and Reasoning scores.

Reinforcement learning with human-AI feedback outperforms baseline models.

Abstract

Large Language Models (LLMs) have demonstrated strong capabilities in text-based tasks but struggle with the complex reasoning required for physics problems, particularly in advanced arithmetic and conceptual understanding. While some research has explored ways to enhance LLMs in physics education using techniques such as prompt engineering and Retrieval Augmentation Generation (RAG), not enough effort has been made in addressing their limitations in physics reasoning. This paper presents a novel approach to improving LLM performance on physics questions using Reinforcement Learning with Human and Artificial Intelligence Feedback (RLHAIF). We evaluate several reinforcement learning methods, including Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), and Remax optimization. These methods are chosen to investigate RL policy performance with different settings on the PhyQA dataset, which includes challenging physics problems from high school textbooks. Our RLHAIF model, tested on leading LLMs like LLaMA2 and Mistral, achieved superior results, notably with the MISTRAL-PPO model, demonstrating marked improvements in reasoning and accuracy. It achieved high scores, with a 58.67 METEOR score and a 0.74 Reasoning score, making it a strong example for future physics reasoning research in this area.