Intent at a Glance: Gaze-Guided Robotic Manipulation via Foundation Models

TL;DR

This paper introduces GAMMA, a gaze-guided robotic manipulation system that uses foundation models to interpret eye gaze and scene context, enabling intuitive and autonomous robot control in manipulation tasks.

Contribution

The work presents a novel system that combines ego-centric gaze tracking with vision-language models to infer user intent and control robots without task-specific training.

Findings

GAMMA achieves robust and intuitive control in tabletop manipulation tasks.

The system generalizes well across different tasks without retraining.

GAMMA outperforms baseline gaze control methods in user studies.

Abstract

Designing intuitive interfaces for robotic control remains a central challenge in enabling effective human-robot interaction, particularly in assistive care settings. Eye gaze offers a fast, non-intrusive, and intent-rich input modality, making it an attractive channel for conveying user goals. In this work, we present GAMMA (Gaze Assisted Manipulation for Modular Autonomy), a system that leverages ego-centric gaze tracking and a vision-language model to infer user intent and autonomously execute robotic manipulation tasks. By contextualizing gaze fixations within the scene, the system maps visual attention to high-level semantic understanding, enabling skill selection and parameterization without task-specific training. We evaluate GAMMA on a range of table-top manipulation tasks and compare it against baseline gaze-based control without reasoning. Results demonstrate that GAMMA provides robust, intuitive, and generalizable control, highlighting the potential of combining foundation models and gaze for natural and scalable robot autonomy. Project website: https://gamma0.vercel.app/