Modularization of End-to-End Learning: Case Study in Arcade Games

TL;DR

This paper demonstrates that decomposing arcade game environments into interacting objects and using specialized modules can significantly improve learning efficiency and generalization in end-to-end learning systems.

Contribution

It introduces a modular framework for end-to-end learning in arcade games, emphasizing environment decomposition and specialized modules for different interactions.

Findings

Achieves human-level performance within 10-15 minutes of gameplay.

Modular approach improves learning speed and generalization.

Environment decomposition is crucial for effective learning.

Abstract

Complex environments and tasks pose a difficult problem for holistic end-to-end learning approaches. Decomposition of an environment into interacting controllable and non-controllable objects allows supervised learning for non-controllable objects and universal value function approximator learning for controllable objects. Such decomposition should lead to a shorter learning time and better generalisation capability. Here, we consider arcade-game environments as sets of interacting objects (controllable, non-controllable) and propose a set of functional modules that are specialized on mastering different types of interactions in a broad range of environments. The modules utilize regression, supervised learning, and reinforcement learning algorithms. Results of this case study in different Atari games suggest that human-level performance can be achieved by a learning agent within a human amount of game experience (10-15 minutes game time) when a proper decomposition of an environment or a task is provided. However, automatization of such decomposition remains a challenging problem. This case study shows how a model of a causal structure underlying an environment or a task can benefit learning time and generalization capability of the agent, and argues in favor of exploiting modular structure in contrast to using pure end-to-end learning approaches.