Discovering High Level Patterns from Simulation Traces

TL;DR

This paper introduces an unsupervised program synthesis approach to translate simulation traces into high-level, sparse pattern representations, enhancing interpretability and reasoning about physical systems using LLMs.

Contribution

It proposes a novel unsupervised learning scheme for translating simulation data into interpretable pattern detectors via program synthesis, improving explainability in physical reasoning.

Findings

Annotated pattern representations improve natural language reasoning about physical systems.

Synthesized programs act as transparent functions mapping system states to sparse annotations.

Application to a physics benchmark demonstrates enhanced interpretability and reasoning capabilities.

Abstract

Large Language Models (LLMs) are unable to reliably reason about specific physical systems. Attempts to imbue LLMs with knowledge of the necessary physics concepts have shown great promise, but explainability and validation remain open challenges. An emerging alternative is tooling, where LLMs can query physical simulators and use the resulting simulation traces as context for validation. This approach suffers from poor scalability since simulation traces contain large volumes of fine-grained numerical and semantic data. We show that translating simulation traces to a sparse representation of "high-level" structural patterns leads to more effective interpretation by LLMs. We propose an unsupervised learning scheme to perform this translation, or annotation, via program synthesis. Our learning results in a library of programs that act as pattern detectors which can translate simulation traces to sparse, annotated pattern sequences. The detected patterns may optionally be guided by human experts via string labels (rigid collision, stretching spring, etc.). We show, using a recent physics benchmark, that such annotated representations are more amenable to natural language reasoning about specific physical systems. The synthesized programs serve as transparent, explainable functions that map system states to a sparse and efficient annotation space. As an example application, we show how goals within physical systems that are specified in natural language may be converted to reward programs which are maximized to find solutions.