Beyond Static Instruction: A Multi-agent AI Framework for Adaptive Augmented Reality Robot Training

TL;DR

This paper introduces a multi-agent AI framework for augmented reality robot training that dynamically adapts to learners' needs, enhancing the educational experience through real-time reasoning and multimodal data processing.

Contribution

It proposes a novel multi-agent AI system utilizing LLMs to enable adaptive AR-based robot training tailored to individual learner profiles.

Findings

Baseline AR interface had high usability but variable task durations.

Disparities in learner characteristics indicate need for adaptive systems.

Framework integrates multimodal input preprocessing and real-time adaptation.

Abstract

Augmented Reality (AR) offers powerful visualization capabilities for industrial robot training, yet current interfaces remain predominantly static, failing to account for learners' diverse cognitive profiles. In this paper, we present an AR application for robot training and propose a multi-agent AI framework for future integration that bridges the gap between static visualization and pedagogical intelligence. We report on the evaluation of the baseline AR interface with 36 participants performing a robotic pick-and-place task. While overall usability was high, notable disparities in task duration and learner characteristics highlighted the necessity for dynamic adaptation. To address this, we propose a multi-agent framework that orchestrates multiple components to perform complex preprocessing of multimodal inputs (e.g., voice, physiology, robot data) and adapt the AR application to the learner's needs. By utilizing autonomous Large Language Model (LLM) agents, the proposed system would dynamically adapt the learning environment based on advanced LLM reasoning in real-time.