Ro-SLM: Onboard Small Language Models for Robot Task Planning and Operation Code Generation

TL;DR

Ro-SLM is a framework that distills large language models into small, onboard-capable models for robot task planning and code generation, enabling reliable operation without internet dependence.

Contribution

The paper introduces Ro-SLM, a novel method for training small language models for robots by distilling knowledge from large models using dataset synthesis and reward-guided fine-tuning.

Findings

Ro-SLM enables small models to support robotic task planning and code generation.

Performance of SLM approaches that of large language models in UAV tasks.

Extensive experiments validate the effectiveness of Ro-SLM in real-world scenarios.

Abstract

Recent advances in large language models (LLMs) provide robots with contextual reasoning abilities to comprehend human instructions. Yet, current LLM-enabled robots typically depend on cloud-based models or high-performance computing infrastructure, which limit their deployment on robots under unreliable internet environments or with constrained computational resources, such as UAVs and small ground vehicles. Thus, deploying fine-tuned small language models (SLMs) that support onboard deployment offers a promising alternative. This paper introduces Ro-SLM, a framework that enables reliable SLM-driven robot operation by distilling LLMs' knowledge and reasoning. Ro-SLM starts from dataset synthesis by leveraging LLMs to generate diverse task instructions, produce corresponding ground truth code with minimal human assistance, and augment instructions into real-world application scenarios. Ro-SLM is then fine-tuned with the dataset, in which LLM serves as a reward function to guide the training. Extensive experiments on UAV operation tasks demonstrate that Ro-SLM improves the performance of SLM from being incapable of supporting robotic task planning and code generation to achieving performance that approaches LLM.