Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction

TL;DR

This paper introduces a goal-aware representation learning approach with adaptive horizon prediction to improve goal-conditioned multi-task policies in Minecraft, achieving significant performance gains and zero-shot generalization in complex open-ended environments.

Contribution

The paper proposes a novel Goal-Sensitive Backbone and adaptive horizon prediction module to address task indistinguishability and non-stationary dynamics in open-world multi-task learning.

Findings

Outperforms baseline methods on 20 Minecraft tasks

Doubles performance in many tasks

Achieves zero-shot generalization to new scenes

Abstract

We study the problem of learning goal-conditioned policies in Minecraft, a popular, widely accessible yet challenging open-ended environment for developing human-level multi-task agents. We first identify two main challenges of learning such policies: 1) the indistinguishability of tasks from the state distribution, due to the vast scene diversity, and 2) the non-stationary nature of environment dynamics caused by partial observability. To tackle the first challenge, we propose Goal-Sensitive Backbone (GSB) for the policy to encourage the emergence of goal-relevant visual state representations. To tackle the second challenge, the policy is further fueled by an adaptive horizon prediction module that helps alleviate the learning uncertainty brought by the non-stationary dynamics. Experiments on 20 Minecraft tasks show that our method significantly outperforms the best baseline so far; in many of them, we double the performance. Our ablation and exploratory studies then explain how our approach beat the counterparts and also unveil the surprising bonus of zero-shot generalization to new scenes (biomes). We hope our agent could help shed some light on learning goal-conditioned, multi-task agents in challenging, open-ended environments like Minecraft.