Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation

TL;DR

This paper introduces Sensor Dropout and auxiliary loss techniques to enhance the robustness and smoothness of end-to-end multisensory policies for autonomous navigation, demonstrated through simulation in a racing game.

Contribution

It proposes Sensor Dropout for sensor failure robustness and an auxiliary loss to reduce jerks during sensor switching in end-to-end sensorimotor policies.

Findings

Sensor Dropout improves policy robustness to sensor failure.

Auxiliary loss reduces jerks during sensor switching.

Policies learn a shared latent state representation despite diverse observations.

Abstract

Multisensory polices are known to enhance both state estimation and target tracking. However, in the space of end-to-end sensorimotor control, this multi-sensor outlook has received limited attention. Moreover, systematic ways to make policies robust to partial sensor failure are not well explored. In this work, we propose a specific customization of Dropout, called \textit{Sensor Dropout}, to improve multisensory policy robustness and handle partial failure in the sensor-set. We also introduce an additional auxiliary loss on the policy network in order to reduce variance in the band of potential multi- and uni-sensory policies to reduce jerks during policy switching triggered by an abrupt sensor failure or deactivation/activation. Finally, through the visualization of gradients, we show that the learned policies are conditioned on the same latent states representation despite having diverse observations spaces - a hallmark of true sensor-fusion. Simulation results of the multisensory policy, as visualized in TORCS racing game, can be seen here: https://youtu.be/QAK2lcXjNZc.