Weighted delay-and-sum beamforming guided by visual tracking for human-robot interaction

TL;DR

This paper presents a method combining visual tracking and weighted delay-and-sum beamforming for improved human-robot interaction, demonstrating significant reduction in word error rate in real robotic experiments.

Contribution

It introduces a novel integration of visual servoing with beamforming using weighted delay-and-sum, enhancing source localization accuracy in robotic environments.

Findings

Up to 34% reduction in word error rate.

Effective source tracking with static and dynamic robot heads.

Robust performance with multiple noise sources.

Abstract

This paper describes the integration of weighted delay-and-sum beamforming with speech source localization using image processing and robot head visual servoing for source tracking. We take into consideration the fact that the directivity gain provided by the beamforming depends on the angular distance between its main lobe and the main response axis of the microphone array. A visual servoing scheme is used to reduce the angular distance between the center of the video frame of a robot camera and a target object. Additionally, the beamforming strategy presented combines two information sources: the direction of the target object obtained with image processing and the audio signals provided by a microphone array. These sources of information were integrated by making use of a weighted delay-and-sum beamforming method. Experiments were carried out with a real mobile robotic testbed built with a PR2 robot. Static and dynamic robot head as well as the use of one and two external noise sources were considered. The results presented here show that the appropriate integration of visual source tracking with visual servoing and a beamforming method can lead to a reduction in WER as high as 34% compared to beamforming alone.