Deep Learning-based Lightweight RGB Object Tracking for Augmented Reality Devices

TL;DR

This paper introduces a lightweight deep learning-based RGB object tracker optimized for AR devices, achieving real-time performance with high accuracy through model compression techniques.

Contribution

It presents a compact Siamese neural network with pruning, quantization, and distillation, enabling real-time object tracking on resource-limited AR hardware.

Findings

Achieves comparable accuracy to state-of-the-art trackers

Runs at 30 FPS on mobile AR headsets

Over an order of magnitude faster than prior methods

Abstract

Augmented Reality (AR) applications often require robust real-time tracking of objects in the user's environment to correctly overlay virtual content. Recent advances in computer vision have produced highly accurate deep learning-based object trackers, but these models are typically too heavy in computation and memory for wearable AR devices. In this paper, we present a lightweight RGB object tracking algorithm designed specifically for resource-constrained AR platforms. The proposed tracker employs a compact Siamese neural network architecture and incorporates optimization techniques such as model pruning, quantization, and knowledge distillation to drastically reduce model size and inference cost while maintaining high tracking accuracy. We train the tracker offline on large video datasets using deep convolutional neural networks and then deploy it on-device for real-time tracking. Experimental results on standard tracking benchmarks show that our approach achieves comparable accuracy to state-of-the-art trackers, yet runs in real-time on a mobile AR headset at around 30 FPS -- more than an order of magnitude faster than prior high-performance trackers on the same hardware. This work enables practical, robust object tracking for AR use-cases, opening the door to more interactive and dynamic AR experiences on lightweight devices.