Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors

TL;DR

This paper introduces CCM, a self-supervised model enabling robots to detect and compensate for corrupted sensor data by leveraging multiple modalities, improving robustness in manipulation tasks.

Contribution

The paper presents CCM, a novel self-supervised model that detects corrupted sensors and compensates using remaining modalities, enhancing robotic perception.

Findings

CCM effectively detects corrupted sensor modalities.

CCM enables robust manipulation despite unseen sensor corruptions.

Rich state representations improve manipulation performance.

Abstract

Using sensor data from multiple modalities presents an opportunity to encode redundant and complementary features that can be useful when one modality is corrupted or noisy. Humans do this everyday, relying on touch and proprioceptive feedback in visually-challenging environments. However, robots might not always know when their sensors are corrupted, as even broken sensors can return valid values. In this work, we introduce the Crossmodal Compensation Model (CCM), which can detect corrupted sensor modalities and compensate for them. CMM is a representation model learned with self-supervision that leverages unimodal reconstruction loss for corruption detection. CCM then discards the corrupted modality and compensates for it with information from the remaining sensors. We show that CCM learns rich state representations that can be used for contact-rich manipulation policies, even when input modalities are corrupted in ways not seen during training time.