CQM: Curriculum Reinforcement Learning with a Quantized World Model

TL;DR

This paper introduces a novel curriculum reinforcement learning method that automatically defines a semantic goal space using vector quantized autoencoders and graph structures, improving scalability and performance in complex tasks.

Contribution

It proposes an automatic goal space construction method with VQ-VAE and graph modeling, enabling scalable and efficient curriculum RL without manual goal specification.

Findings

Outperforms state-of-the-art curriculum RL methods in data efficiency.

Enables effective exploration with raw goal examples.

Achieves superior performance in goal-reaching tasks with visual inputs.

Abstract

Recent curriculum Reinforcement Learning (RL) has shown notable progress in solving complex tasks by proposing sequences of surrogate tasks. However, the previous approaches often face challenges when they generate curriculum goals in a high-dimensional space. Thus, they usually rely on manually specified goal spaces. To alleviate this limitation and improve the scalability of the curriculum, we propose a novel curriculum method that automatically defines the semantic goal space which contains vital information for the curriculum process, and suggests curriculum goals over it. To define the semantic goal space, our method discretizes continuous observations via vector quantized-variational autoencoders (VQ-VAE) and restores the temporal relations between the discretized observations by a graph. Concurrently, ours suggests uncertainty and temporal distance-aware curriculum goals that converges to the final goals over the automatically composed goal space. We demonstrate that the proposed method allows efficient explorations in an uninformed environment with raw goal examples only. Also, ours outperforms the state-of-the-art curriculum RL methods on data efficiency and performance, in various goal-reaching tasks even with ego-centric visual inputs.