Sensor Transformation Attention Networks

TL;DR

This paper introduces Sensor Transformation Attention Networks that dynamically switch among multiple sensors using attention mechanisms, improving robustness and performance in noisy environments for audio and visual tasks.

Contribution

The work presents a novel attentional switching method for sensor inputs, enhancing robustness, interpretability, and transferability across datasets in noisy conditions.

Findings

Improved performance on TIDIGITS, WSJ, and GRID datasets.

Dynamic sensor switching responds to environmental noise levels.

Transfer of sensor attention mechanism reduces errors in larger models.

Abstract

Recent work on encoder-decoder models for sequence-to-sequence mapping has shown that integrating both temporal and spatial attention mechanisms into neural networks increases the performance of the system substantially. In this work, we report on the application of an attentional signal not on temporal and spatial regions of the input, but instead as a method of switching among inputs themselves. We evaluate the particular role of attentional switching in the presence of dynamic noise in the sensors, and demonstrate how the attentional signal responds dynamically to changing noise levels in the environment to achieve increased performance on both audio and visual tasks in three commonly-used datasets: TIDIGITS, Wall Street Journal, and GRID. Moreover, the proposed sensor transformation network architecture naturally introduces a number of advantages that merit exploration, including ease of adding new sensors to existing architectures, attentional interpretability, and increased robustness in a variety of noisy environments not seen during training. Finally, we demonstrate that the sensor selection attention mechanism of a model trained only on the small TIDIGITS dataset can be transferred directly to a pre-existing larger network trained on the Wall Street Journal dataset, maintaining functionality of switching between sensors to yield a dramatic reduction of error in the presence of noise.