Scientific Logicality Enriched Methodology for LLM Reasoning: A Practice in Physics

TL;DR

This paper introduces a methodology to enhance LLM reasoning in scientific tasks by focusing on logicality, demonstrated through physics problems, leading to improved reasoning validity and problem-solving performance.

Contribution

It systematically investigates LLM logicality in scientific reasoning and develops a logicality-enriched training approach using physics as a case study.

Findings

Training data improves logicality in LLM reasoning.

Enhanced logicality significantly boosts scientific problem-solving.

The methodology is validated across three different backbone LLMs.

Abstract

With the continuous advancement of reasoning abilities in Large Language Models (LLMs), their application to scientific reasoning tasks has gained significant research attention. Current research primarily emphasizes boosting LLMs' performance on scientific QA benchmarks by training on larger, more comprehensive datasets with extended reasoning chains. However, these approaches neglect the essence of the scientific reasoning process -- logicality, which is the rational foundation to ensure the validity of reasoning steps leading to reliable conclusions. In this work, we make the first systematic investigation into the internal logicality underlying LLM scientific reasoning, and develop a scientific logicality-enriched methodology, including a set of assessment criteria and data sampling methods for logicality-guided training, to improve the logical faithfulness as well as task performance. Further, we take physics, characterized by its diverse logical structures and formalisms, as an exemplar discipline to practise the above methodology. For data construction, we extract scientific problems from academic literature and sample a high-quality dataset exhibiting strong logicality. Experiments based on three different backbone LLMs reveal that: 1) the training data we constructed can effectively improve the scientific logicality in LLM reasoning; and 2) the enriched scientific logicality plays a critical role in solving scientific problems. Code is available at \href{https://github.com/ScienceOne-AI/PhysLogic}{https://github.com/ScienceOne-AI/PhysLogic}.