Mask-based Neural Beamforming for Moving Speakers with Self-Attention-based Tracking

TL;DR

This paper introduces a neural network with self-attention layers to compute optimal attention weights for mask-based beamforming, significantly improving performance for moving speakers while maintaining effectiveness for stationary sources.

Contribution

It proposes a learning-based framework using self-attention to adaptively compute attention weights for beamforming with moving sources, surpassing classical heuristic methods.

Findings

Enhanced beamforming performance with moving sources.

Maintained high performance for stationary sources.

Demonstrated effectiveness of self-attention in source tracking.

Abstract

Beamforming is a powerful tool designed to enhance speech signals from the direction of a target source. Computing the beamforming filter requires estimating spatial covariance matrices (SCMs) of the source and noise signals. Time-frequency masks are often used to compute these SCMs. Most studies of mask-based beamforming have assumed that the sources do not move. However, sources often move in practice, which causes performance degradation. In this paper, we address the problem of mask-based beamforming for moving sources. We first review classical approaches to tracking a moving source, which perform online or blockwise computation of the SCMs. We show that these approaches can be interpreted as computing a sum of instantaneous SCMs weighted by attention weights. These weights indicate which time frames of the signal to consider in the SCM computation. Online or blockwise computation assumes a heuristic and deterministic way of computing these attention weights that, although simple, may not result in optimal performance. We thus introduce a learning-based framework that computes optimal attention weights for beamforming. We achieve this using a neural network implemented with self-attention layers. We show experimentally that our proposed framework can greatly improve beamforming performance in moving source situations while maintaining high performance in non-moving situations, thus enabling the development of mask-based beamformers robust to source movements.