Distilling On-device Language Models for Robot Planning with Minimal Human Intervention

TL;DR

This paper introduces PRISM, a framework that distills large language models into small, on-device models for robot planning, enabling high performance with minimal human supervision and synthetic data, suitable for diverse environments.

Contribution

PRISM automatically synthesizes training data from existing LLMs to distill compact models for robot planning, improving on-device performance and generalization across platforms and environments.

Findings

Distilled models achieve over 93% of GPT-4o's performance.

PRISM enables on-device robot planning with minimal human intervention.

Models generalize across different robotic platforms and environments.

Abstract

Large language models (LLMs) provide robots with powerful contextual reasoning abilities and a natural human interface. Yet, current LLM-enabled robots typically depend on cloud-hosted models, limiting their usability in environments with unreliable communication infrastructure, such as outdoor or industrial settings. We present PRISM, a framework for distilling small language model (SLM)-enabled robot planners that run on-device with minimal human supervision. Starting from an existing LLM-enabled planner, PRISM automatically synthesizes diverse tasks and environments, elicits plans from the LLM, and uses this synthetic dataset to distill a compact SLM as a drop-in replacement of the source model. We apply PRISM to three LLM-enabled planners for mapping and exploration, manipulation, and household assistance, and we demonstrate that PRISM improves the performance of Llama-3.2-3B from 10-20% of GPT-4o's performance to over 93% - using only synthetic data. We further demonstrate that the distilled planners generalize across heterogeneous robotic platforms (ground and aerial) and diverse environments (indoor and outdoor). We release all software, trained models, and datasets at https://zacravichandran.github.io/PRISM.