NEWTON: Are Large Language Models Capable of Physical Reasoning?

TL;DR

This paper introduces NEWTON, a comprehensive benchmark and repository to evaluate large language models' physical reasoning abilities, highlighting their strengths and limitations in understanding everyday objects and attributes.

Contribution

The paper presents a new benchmark and dataset for assessing physical reasoning in LLMs, along with a pipeline for domain-specific customization, filling a gap in existing evaluation methods.

Findings

GPT-4 shows strong scenario-based reasoning capabilities.

LLMs are less consistent than humans in object-attribute reasoning.

The benchmark enables targeted evaluation and improvement of physical reasoning in LLMs.

Abstract

Large Language Models (LLMs), through their contextualized representations, have been empirically proven to encapsulate syntactic, semantic, word sense, and common-sense knowledge. However, there has been limited exploration of their physical reasoning abilities, specifically concerning the crucial attributes for comprehending everyday objects. To address this gap, we introduce NEWTON, a repository and benchmark for evaluating the physics reasoning skills of LLMs. Further, to enable domain-specific adaptation of this benchmark, we present a pipeline to enable researchers to generate a variant of this benchmark that has been customized to the objects and attributes relevant for their application. The NEWTON repository comprises a collection of 2800 object-attribute pairs, providing the foundation for generating infinite-scale assessment templates. The NEWTON benchmark consists of 160K QA questions, curated using the NEWTON repository to investigate the physical reasoning capabilities of several mainstream language models across foundational, explicit, and implicit reasoning tasks. Through extensive empirical analysis, our results highlight the capabilities of LLMs for physical reasoning. We find that LLMs like GPT-4 demonstrate strong reasoning capabilities in scenario-based tasks but exhibit less consistency in object-attribute reasoning compared to humans (50% vs. 84%). Furthermore, the NEWTON platform demonstrates its potential for evaluating and enhancing language models, paving the way for their integration into physically grounded settings, such as robotic manipulation. Project site: https://newtonreasoning.github.io