# Visio-Verbal Teleimpedance Interface: Enabling Semi-Autonomous Control of Physical Interaction via Eye Tracking and Speech

## TL;DR

This paper introduces a novel teleimpedance interface that combines eye tracking and speech recognition to enable intuitive semi-autonomous control of remote robots' physical interactions, validated through experiments with a robotic arm.

## Contribution

It presents a new visio-verbal interface integrating gaze and speech inputs with a Visual Language Model for remote robot control, enhancing interaction intuitiveness.

## Key findings

- Effective control of remote robot stiffness via gaze and speech
- Optimal prompt configuration improves interface performance
- Successful demonstration on a slide-in-the-groove task

## Abstract

The paper presents a visio-verbal teleimpedance interface for commanding 3D stiffness ellipsoids to the remote robot with a combination of the operator's gaze and verbal interaction. The gaze is detected by an eye-tracker, allowing the system to understand the context in terms of what the operator is currently looking at in the scene. Along with verbal interaction, a Visual Language Model (VLM) processes this information, enabling the operator to communicate their intended action or provide corrections. Based on these inputs, the interface can then generate appropriate stiffness matrices for different physical interaction actions. To validate the proposed visio-verbal teleimpedance interface, we conducted a series of experiments on a setup including a Force Dimension Sigma.7 haptic device to control the motion of the remote Kuka LBR iiwa robotic arm. The human operator's gaze is tracked by Tobii Pro Glasses 2, while human verbal commands are processed by a VLM using GPT-4o. The first experiment explored the optimal prompt configuration for the interface. The second and third experiments demonstrated different functionalities of the interface on a slide-in-the-groove task.