A Gesture-Based Visual Learning Model for Acoustophoretic Interactions using a Swarm of AcoustoBots

TL;DR

This paper introduces a gesture-based visual learning system for AcoustoBots, enabling intuitive human-swarm interaction with high accuracy and low latency, leveraging vision-language models and multimodal control.

Contribution

It presents a novel gesture recognition framework using visual learning models for real-time control of acoustophoretic robot swarms, improving accuracy and responsiveness.

Findings

Validation accuracy increased from 67% to 98% with more data.

Achieved 87.8% gesture-to-modality switching accuracy in experiments.

Average end-to-end latency was 3.95 seconds.

Abstract

AcoustoBots are mobile acoustophoretic robots capable of delivering mid-air haptics, directional audio, and acoustic levitation, but existing implementations rely on scripted commands and lack an intuitive interface for real-time human control. This work presents a gesture-based visual learning framework for contactless human-swarm interaction with a multimodal AcoustoBot platform. The system combines ESP32-CAM gesture capture, PhaseSpace motion tracking, centralized processing, and an OpenCLIP-based visual learning model (VLM) with linear probing to classify three hand gestures and map them to haptics, audio, and levitation modalities. Validation accuracy improved from about 67% with a small dataset to nearly 98% with the largest dataset. In integrated experiments with two AcoustoBots, the system achieved an overall gesture-to-modality switching accuracy of 87.8% across 90 trials, with an average end-to-end latency of 3.95 seconds. These results demonstrate the feasibility of using a vision-language-model-based gesture interface for multimodal human-swarm interaction. While the current system is limited by centralized processing, a static gesture set, and controlled-environment evaluation, it establishes a foundation for more expressive, scalable, and accessible swarm robotic interfaces.