Real-Time Object Tracking with On-Device Deep Learning for Adaptive Beamforming in Dynamic Acoustic Environments

TL;DR

This paper presents an embedded system that combines deep learning-based object tracking with adaptive beamforming to localize and capture sound sources accurately in dynamic environments, enhancing applications like teleconferencing and smart devices.

Contribution

It introduces a novel integrated system that uses vision and deep learning for real-time 3D object tracking to dynamically steer acoustic beamforming arrays.

Findings

Significant improvement in signal-to-interference ratio.

Robust performance with multiple and moving sound sources.

Effective real-time adaptation of beam steering based on visual tracking.

Abstract

Advances in object tracking and acoustic beamforming are driving new capabilities in surveillance, human-computer interaction, and robotics. This work presents an embedded system that integrates deep learning-based tracking with beamforming to achieve precise sound source localization and directional audio capture in dynamic environments. The approach combines single-camera depth estimation and stereo vision to enable accurate 3D localization of moving objects. A planar concentric circular microphone array constructed with MEMS microphones provides a compact, energy-efficient platform supporting 2D beam steering across azimuth and elevation. Real-time tracking outputs continuously adapt the array's focus, synchronizing the acoustic response with the target's position. By uniting learned spatial awareness with dynamic steering, the system maintains robust performance in the presence of multiple or moving sources. Experimental evaluation demonstrates significant gains in signal-to-interference ratio, making the design well-suited for teleconferencing, smart home devices, and assistive technologies.