Accessible Gesture-Driven Augmented Reality Interaction System
Yikan Wang

TL;DR
This paper introduces a gesture-based AR interaction system using deep learning and federated learning to improve accessibility for users with motor impairments, demonstrating significant efficiency and satisfaction gains.
Contribution
It presents a novel gesture recognition system combining ViTs, TCNs, and GATs with federated learning for privacy, enhancing AR accessibility for motor-impaired users.
Findings
20% improvement in task completion efficiency
25% increase in user satisfaction
Effective privacy-preserving gesture recognition
Abstract
Augmented reality (AR) offers immersive interaction but remains inaccessible for users with motor impairments or limited dexterity due to reliance on precise input methods. This study proposes a gesture-based interaction system for AR environments, leveraging deep learning to recognize hand and body gestures from wearable sensors and cameras, adapting interfaces to user capabilities. The system employs vision transformers (ViTs), temporal convolutional networks (TCNs), and graph attention networks (GATs) for gesture processing, with federated learning ensuring privacy-preserving model training across diverse users. Reinforcement learning optimizes interface elements like menu layouts and interaction modes. Experiments demonstrate a 20% improvement in task completion efficiency and a 25% increase in user satisfaction for motor-impaired users compared to baseline AR systems. This approach…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAugmented Reality Applications
