Vision-Aided Beam Tracking: Explore the Proper Use of Camera Images with Deep Learning

TL;DR

This paper explores how camera images combined with deep learning can improve wireless beam tracking in mmWave bands, especially in challenging NLOS conditions, by reformulating datasets and analyzing different training scenarios.

Contribution

It introduces a reformulated dataset to avoid image repetition issues and investigates the effectiveness of camera images in deep learning models for beam prediction under various NLOS conditions.

Findings

Using images improves beam tracking in serious NLOS scenarios.

Including NLOS-like data in training benefits beam prediction in NLOS conditions.

Careful dataset design is crucial for effective deep learning-based beam tracking.

Abstract

We investigate the problem of wireless beam tracking on mmWave bands with the assistance of camera images. In particular, based on the user's beam indices used and camera images taken in the trajectory, we predict the optimal beam indices in the next few time spots. To resolve this problem, we first reformulate the "ViWi" dataset in [1] to get rid of the image repetition problem. Then we develop a deep learning approach and investigate various model components to achieve the best performance. Finally, we explore whether, when, and how to use the image for better beam prediction. To answer this question, we split the dataset into three clusters -- (LOS, light NLOS, serious NLOS)-like -- based on the standard deviation of the beam sequence. With experiments we demonstrate that using the image indeed helps beam tracking especially when the user is in serious NLOS, and the solution relies on carefully-designed dataset for training a model. Generally speaking, including NLOS-like data for training a model does not benefit beam tracking of the user in LOS, but including light NLOS-like data for training a model benefits beam tracking of the user in serious NLOS.