Towards Natural Language-Driven Assembly Using Foundation Models

TL;DR

This paper introduces a novel approach using Large Language Models to develop a global control policy for robots, enabling high-precision assembly tasks through dynamic skill switching and enhanced language understanding.

Contribution

It presents a framework that leverages LLMs for controlling robots with high precision by dynamically switching between specialized skills, addressing limitations of generalist policies in industrial tasks.

Findings

Effective transfer of control policies to high-precision skills

Enhanced language interpretation for robotic control

Successful dynamic context switching in assembly tasks

Abstract

Large Language Models (LLMs) and strong vision models have enabled rapid research and development in the field of Vision-Language-Action models that enable robotic control. The main objective of these methods is to develop a generalist policy that can control robots with various embodiments. However, in industrial robotic applications such as automated assembly and disassembly, some tasks, such as insertion, demand greater accuracy and involve intricate factors like contact engagement, friction handling, and refined motor skills. Implementing these skills using a generalist policy is challenging because these policies might integrate further sensory data, including force or torque measurements, for enhanced precision. In our method, we present a global control policy based on LLMs that can transfer the control policy to a finite set of skills that are specifically trained to perform high-precision tasks through dynamic context switching. The integration of LLMs into this framework underscores their significance in not only interpreting and processing language inputs but also in enriching the control mechanisms for diverse and intricate robotic operations.