On the capabilities of LLMs for classifying and segmenting time series of fruit picking motions into primitive actions

TL;DR

This paper explores the use of Large Language Models to classify and segment fruit-picking motions into primitive actions, aiming to improve Learning by Demonstration methods for robotic tasks.

Contribution

It investigates the capabilities of LLMs for motion segmentation in fruit picking, comparing three fine-tuning approaches on kinesthetic datasets, enhancing applicability in real-world scenarios.

Findings

LLMs can effectively classify primitive actions in fruit-picking motions.

Fine-tuning approaches significantly impact segmentation accuracy.

The method shows promise for deployment in robotic learning tasks.

Abstract

Despite their recent introduction to human society, Large Language Models (LLMs) have significantly affected the way we tackle mental challenges in our everyday lives. From optimizing our linguistic communication to assisting us in making important decisions, LLMs, such as ChatGPT, are notably reducing our cognitive load by gradually taking on an increasing share of our mental activities. In the context of Learning by Demonstration (LbD), classifying and segmenting complex motions into primitive actions, such as pushing, pulling, twisting etc, is considered to be a key-step towards encoding a task. In this work, we investigate the capabilities of LLMs to undertake this task, considering a finite set of predefined primitive actions found in fruit picking operations. By utilizing LLMs instead of simple supervised learning or analytic methods, we aim at making the method easily applicable and deployable in a real-life scenario. Three different fine-tuning approaches are investigated, compared on datasets captured kinesthetically, using a UR10e robot, during a fruit-picking scenario.