Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning

TL;DR

Skipper is a reinforcement learning framework inspired by human consciousness that uses spatio-temporal abstractions to improve generalization by decomposing tasks into manageable subtasks and focusing computation on relevant environment parts.

Contribution

It introduces a novel model-based RL approach that automatically learns task decompositions and abstracted proxy problems with theoretical guarantees and improved zero-shot generalization.

Findings

Skipper outperforms existing hierarchical planning methods in zero-shot generalization.

The approach provides theoretical performance guarantees under certain assumptions.

It effectively decomposes tasks into subtasks using learned spatio-temporal abstractions.

Abstract

Inspired by human conscious planning, we propose Skipper, a model-based reinforcement learning framework utilizing spatio-temporal abstractions to generalize better in novel situations. It automatically decomposes the given task into smaller, more manageable subtasks, and thus enables sparse decision-making and focused computation on the relevant parts of the environment. The decomposition relies on the extraction of an abstracted proxy problem represented as a directed graph, in which vertices and edges are learned end-to-end from hindsight. Our theoretical analyses provide performance guarantees under appropriate assumptions and establish where our approach is expected to be helpful. Generalization-focused experiments validate Skipper's significant advantage in zero-shot generalization, compared to some existing state-of-the-art hierarchical planning methods.