ViSTAR: Virtual Skill Training with Augmented Reality with 3D Avatars and LLM coaching agent

TL;DR

ViSTAR is an AR-based virtual skill training system for basketball that uses 3D avatars and an LLM coaching agent to provide real-time, personalized feedback on skills, improving self-guided practice effectiveness.

Contribution

The paper introduces a novel AR training system that integrates 3D motion analysis with LLM-generated verbal feedback for sports skill development.

Findings

Participants preferred AI feedback over coach feedback.

ViSTAR helped users identify posture and balance issues.

The system enhanced movement refinement beyond self-observation.

Abstract

We present ViSTAR, a Virtual Skill Training system in AR that supports self-guided basketball skill practice, with feedback on balance, posture, and timing. From a formative study with basketball players and coaches, the system addresses three challenges: understanding skills, identifying errors, and correcting mistakes. ViSTAR follows the Behavioral Skills Training (BST) framework-instruction, modeling, rehearsal, and feedback. It provides feedback through visual overlays, rhythm and timing cues, and an AI-powered coaching agent using 3D motion reconstruction. We generate verbal feedback by analyzing spatio-temporal joint data and mapping features to natural-language coaching cues via a Large Language Model (LLM). A key novelty is this feedback generation: motion features become concise coaching insights. In two studies (N=16), participants generally preferred our AI-generated feedback to coach feedback and reported that ViSTAR helped them notice posture and balance issues and refine movements beyond self-observation.