SEMI: Self-supervised Exploration via Multisensory Incongruity

TL;DR

SEMI introduces a self-supervised exploration method in reinforcement learning that uses multisensory incongruity as an intrinsic reward, improving exploration efficiency without external rewards.

Contribution

The paper proposes a novel intrinsic reward based on multisensory incongruity, combining perception and action incongruity for enhanced exploration in RL.

Findings

SEMI improves sample efficiency in various benchmarks.

It effectively combines perception and action incongruity signals.

The method enhances exploration in environments with sparse rewards.

Abstract

Efficient exploration is a long-standing problem in reinforcement learning since extrinsic rewards are usually sparse or missing. A popular solution to this issue is to feed an agent with novelty signals as intrinsic rewards. In this work, we introduce SEMI, a self-supervised exploration policy by incentivizing the agent to maximize a new novelty signal: multisensory incongruity, which can be measured in two aspects, perception incongruity and action incongruity. The former represents the misalignment of the multisensory inputs, while the latter represents the variance of an agent's policies under different sensory inputs. Specifically, an alignment predictor is learned to detect whether multiple sensory inputs are aligned, the error of which is used to measure perception incongruity. A policy model takes different combinations of the multisensory observations as input and outputs actions for exploration. The variance of actions is further used to measure action incongruity. Using both incongruities as intrinsic rewards, SEMI allows an agent to learn skills by exploring in a self-supervised manner without any external rewards. We further show that SEMI is compatible with extrinsic rewards and it improves sample efficiency of policy learning. The effectiveness of SEMI is demonstrated across a variety of benchmark environments including object manipulation and audio-visual games.