LAMS: LLM-Driven Automatic Mode Switching for Assistive Teleoperation

TL;DR

LAMS uses large language models to automatically switch control modes in robotic teleoperation, reducing manual effort and improving performance without prior task demonstrations.

Contribution

This paper introduces LAMS, a novel LLM-based approach for automatic mode switching in teleoperation that is generalizable and improves with user interaction.

Findings

LAMS reduces manual mode switches in complex tasks.

Participants preferred LAMS over other methods.

LAMS improves performance over time with user input.

Abstract

Teleoperating high degrees-of-freedom (DoF) robotic manipulators via low-DoF controllers like joysticks often requires frequent switching between control modes, where each mode maps controller movements to specific robot actions. Manually performing this frequent switching can make teleoperation cumbersome and inefficient. On the other hand, existing automatic mode-switching solutions, such as heuristic-based or learning-based methods, are often task-specific and lack generalizability. In this paper, we introduce LLM-Driven Automatic Mode Switching (LAMS), a novel approach that leverages Large Language Models (LLMs) to automatically switch control modes based on task context. Unlike existing methods, LAMS requires no prior task demonstrations and incrementally improves by integrating user-generated mode-switching examples. We validate LAMS through an ablation study and a user study with 10 participants on complex, long-horizon tasks, demonstrating that LAMS effectively reduces manual mode switches, is preferred over alternative methods, and improves performance over time. The project website with supplementary materials is at https://lams-assistance.github.io/.