GestLLM: Advanced Hand Gesture Interpretation via Large Language Models for Human-Robot Interaction

TL;DR

GestLLM leverages large language models and advanced feature extraction to interpret a wide range of hand gestures for more natural and inclusive human-robot interaction, surpassing limitations of traditional gesture recognition systems.

Contribution

Introduces GestLLM, a novel system combining large language models with MediaPipe for flexible, complex gesture interpretation without additional training or prompt engineering.

Findings

Achieves performance comparable to leading vision-language models.

Supports recognition of culturally and contextually diverse gestures.

Enhances naturalness and inclusivity in robot control.

Abstract

This paper introduces GestLLM, an advanced system for human-robot interaction that enables intuitive robot control through hand gestures. Unlike conventional systems, which rely on a limited set of predefined gestures, GestLLM leverages large language models and feature extraction via MediaPipe to interpret a diverse range of gestures. This integration addresses key limitations in existing systems, such as restricted gesture flexibility and the inability to recognize complex or unconventional gestures commonly used in human communication. By combining state-of-the-art feature extraction and language model capabilities, GestLLM achieves performance comparable to leading vision-language models while supporting gestures underrepresented in traditional datasets. For example, this includes gestures from popular culture, such as the ``Vulcan salute" from Star Trek, without any additional pretraining, prompt engineering, etc. This flexibility enhances the naturalness and inclusivity of robot control, making interactions more intuitive and user-friendly. GestLLM provides a significant step forward in gesture-based interaction, enabling robots to understand and respond to a wide variety of hand gestures effectively. This paper outlines its design, implementation, and evaluation, demonstrating its potential applications in advanced human-robot collaboration, assistive robotics, and interactive entertainment.