GhostShell: Streaming LLM Function Calls for Concurrent Embodied Programming

TL;DR

GhostShell introduces a streaming LLM-based system for real-time, concurrent embodied programming, enabling robots to act on-the-fly with improved speed and robustness across complex tasks.

Contribution

This work presents GhostShell, a novel streaming and concurrent function call framework for LLM-guided embodied systems, enhancing real-time robotic control and interaction.

Findings

Achieves a Behavioral Correctness Metric of 0.85 with Claude-4-Sonnet.

Up to 66X faster response times compared to native LLM APIs.

Effective in long-horizon multimodal tasks with strong robustness.

Abstract

We present GhostShell, a novel approach that leverages Large Language Models (LLMs) to enable streaming and concurrent behavioral programming for embodied systems. In contrast to conventional methods that rely on pre-scheduled action sequences or behavior trees, GhostShell drives embodied systems to act on-the-fly by issuing function calls incrementally as tokens are streamed from the LLM. GhostShell features a streaming XML function token parser, a dynamic function interface mapper, and a multi-channel scheduler that orchestrates intra-channel synchronous and inter-channel asynchronous function calls, thereby coordinating serial-parallel embodied actions across multiple robotic components under LLM guidance. We evaluate GhostShell on our robotic prototype COCO through comprehensive grounded experiments across 34 real-world interaction tasks and multiple LLM backends. The results demonstrate that our approach achieves a state-of-the-art Behavioral Correctness Metric of 0.85 with Claude-4-Sonnet, and up to 66X faster response times compared to native LLM function calling APIs. GhostShell also proves effective in long-horizon multimodal tasks, exhibiting strong robustness and generalization capabilities.