Accessible Gesture-Driven Augmented Reality Interaction System

TL;DR

This paper introduces a gesture-based AR interaction system using deep learning and federated learning to improve accessibility for users with motor impairments, demonstrating significant efficiency and satisfaction gains.

Contribution

It presents a novel gesture recognition system combining ViTs, TCNs, and GATs with federated learning for privacy, enhancing AR accessibility for motor-impaired users.

Findings

20% improvement in task completion efficiency

25% increase in user satisfaction

Effective privacy-preserving gesture recognition

Abstract

Augmented reality (AR) offers immersive interaction but remains inaccessible for users with motor impairments or limited dexterity due to reliance on precise input methods. This study proposes a gesture-based interaction system for AR environments, leveraging deep learning to recognize hand and body gestures from wearable sensors and cameras, adapting interfaces to user capabilities. The system employs vision transformers (ViTs), temporal convolutional networks (TCNs), and graph attention networks (GATs) for gesture processing, with federated learning ensuring privacy-preserving model training across diverse users. Reinforcement learning optimizes interface elements like menu layouts and interaction modes. Experiments demonstrate a 20% improvement in task completion efficiency and a 25% increase in user satisfaction for motor-impaired users compared to baseline AR systems. This approach enhances AR accessibility and scalability. Keywords: Deep learning, Federated learning, Gesture recognition, Augmented reality, Accessibility, Human-computer interaction