Navigate Complex Physical Worlds via Geometrically Constrained LLM

TL;DR

This paper explores how Large Language Models can be used to understand, reconstruct, and manipulate complex physical environments through geometric conventions, multi-layer graph workflows, and genetic algorithms, advancing spatial reasoning capabilities.

Contribution

It introduces a novel workflow combining geometric conventions, multi-layer graphs, and multi-agent systems to enhance LLMs' spatial reasoning and geometric inference abilities.

Findings

LLMs can perform multi-step geometric inference in complex environments.

A workflow using multi-layer graphs improves spatial understanding.

Genetic algorithms help solve geometric constraint problems.

Abstract

This study investigates the potential of Large Language Models (LLMs) for reconstructing and constructing the physical world solely based on textual knowledge. It explores the impact of model performance on spatial understanding abilities. To enhance the comprehension of geometric and spatial relationships in the complex physical world, the study introduces a set of geometric conventions and develops a workflow based on multi-layer graphs and multi-agent system frameworks. It examines how LLMs achieve multi-step and multi-objective geometric inference in a spatial environment using multi-layer graphs under unified geometric conventions. Additionally, the study employs a genetic algorithm, inspired by large-scale model knowledge, to solve geometric constraint problems. In summary, this work innovatively explores the feasibility of using text-based LLMs as physical world builders and designs a workflow to enhance their capabilities.