Neuro-Symbolic Control with Large Language Models for Language-Guided Spatial Tasks

TL;DR

This paper introduces a neuro-symbolic control framework combining large language models and neural controllers to improve language-guided spatial tasks in embodied systems, enhancing stability and efficiency.

Contribution

It presents a modular neuro-symbolic approach that separates semantic reasoning from motion control, outperforming LLM-only methods in success rate and speed.

Findings

Achieves over 70% reduction in steps compared to LLM-only control.

Speeds up control by up to 8.83 times.

Improves robustness and interpretability of language-guided control.

Abstract

Although large language models (LLMs) have recently become effective tools for language-conditioned control in embodied systems, instability, slow convergence, and hallucinated actions continue to limit their direct application to continuous control. A modular neuro-symbolic control framework that clearly distinguishes between low-level motion execution and high-level semantic reasoning is proposed in this work. While a lightweight neural delta controller performs bounded, incremental actions in continuous space, a locally deployed LLM interprets symbolic tasks. We assess the suggested method in a planar manipulation setting with spatial relations between objects specified by language. Numerous tasks and local language models, such as Mistral, Phi, and LLaMA-3.2, are used in extensive experiments to compare LLM-only control, neural-only control, and the suggested LLM+DL framework. In comparison to LLM-only baselines, the results show that the neuro-symbolic integration consistently increases both success rate and efficiency, achieving average step reductions exceeding 70% and speedups of up to 8.83x while remaining robust to language model quality. The suggested framework enhances interpretability, stability, and generalization without any need of reinforcement learning or costly rollouts by controlling the LLM to symbolic outputs and allocating uninterpreted execution to a neural controller trained on artificial geometric data. These outputs show empirically that neuro-symbolic decomposition offers a scalable and principled way to integrate language understanding with ongoing control, this approach promotes the creation of dependable and effective language-guided embodied systems.