# Embodied Multimodal Multitask Learning

## TL;DR

This paper introduces a multitask learning model with a novel Dual-Attention unit that improves visual navigation and question answering by disentangling and aligning textual and visual representations, enabling better transfer and interpretability.

## Contribution

The paper presents a new multitask model with a Dual-Attention unit that disentangles and aligns textual and visual knowledge, enhancing transferability and interpretability across multimodal tasks.

## Key findings

- Outperforms baselines on semantic goal navigation and embodied question answering.
- Enables transfer to new words using object detectors.
- Produces modular and interpretable representations.

## Abstract

Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for different multimodal tasks, such as semantic goal navigation and embodied question answering. In this paper, we propose a multitask model capable of jointly learning these multimodal tasks, and transferring knowledge of words and their grounding in visual objects across the tasks. The proposed model uses a novel Dual-Attention unit to disentangle the knowledge of words in the textual representations and visual concepts in the visual representations, and align them with each other. This disentangled task-invariant alignment of representations facilitates grounding and knowledge transfer across both tasks. We show that the proposed model outperforms a range of baselines on both tasks in simulated 3D environments. We also show that this disentanglement of representations makes our model modular, interpretable, and allows for transfer to instructions containing new words by leveraging object detectors.