Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models

TL;DR

Physics Reasoner enhances large language models' ability to solve physics problems by integrating explicit physics knowledge and guided reasoning, significantly improving accuracy on benchmark datasets.

Contribution

It introduces a knowledge-augmented framework with formula sets and checklists to improve physics problem-solving with LLMs, addressing knowledge gaps and application errors.

Findings

Achieves state-of-the-art accuracy on SciBench with 5.8% improvement.

Effectively mitigates knowledge insufficiency and incorrect application issues.

Employs a three-stage process: analysis, formula retrieval, and guided reasoning.

Abstract

Physics problems constitute a significant aspect of reasoning, necessitating complicated reasoning ability and abundant physics knowledge. However, existing large language models (LLMs) frequently fail due to a lack of knowledge or incorrect knowledge application. To mitigate these issues, we propose Physics Reasoner, a knowledge-augmented framework to solve physics problems with LLMs. Specifically, the proposed framework constructs a comprehensive formula set to provide explicit physics knowledge and utilizes checklists containing detailed instructions to guide effective knowledge application. Namely, given a physics problem, Physics Reasoner solves it through three stages: problem analysis, formula retrieval, and guided reasoning. During the process, checklists are employed to enhance LLMs' self-improvement in the analysis and reasoning stages. Empirically, Physics Reasoner mitigates the issues of insufficient knowledge and incorrect application, achieving state-of-the-art performance on SciBench with an average accuracy improvement of 5.8%.