Effective Multimodal Reinforcement Learning with Modality Alignment and Importance Enhancement

TL;DR

This paper introduces a novel multimodal reinforcement learning method that aligns modalities and emphasizes their importance, leading to improved state representation and better policy learning in complex environments.

Contribution

It proposes a new approach for multimodal RL that addresses heterogeneity and importance variability through modality alignment and importance enhancement.

Findings

Outperforms state-of-the-art methods in learning speed

Achieves higher policy quality in multimodal tasks

Enhances state representation learning in multimodal environments

Abstract

Many real-world applications require an agent to make robust and deliberate decisions with multimodal information (e.g., robots with multi-sensory inputs). However, it is very challenging to train the agent via reinforcement learning (RL) due to the heterogeneity and dynamic importance of different modalities. Specifically, we observe that these issues make conventional RL methods difficult to learn a useful state representation in the end-to-end training with multimodal information. To address this, we propose a novel multimodal RL approach that can do multimodal alignment and importance enhancement according to their similarity and importance in terms of RL tasks respectively. By doing so, we are able to learn an effective state representation and consequentially improve the RL training process. We test our approach on several multimodal RL domains, showing that it outperforms state-of-the-art methods in terms of learning speed and policy quality.