Collect & Infer -- a fresh look at data-efficient Reinforcement Learning

TL;DR

This paper introduces the 'Collect and Infer' paradigm for data-efficient Reinforcement Learning, emphasizing the importance of separately modeling data collection and knowledge inference to improve learning efficiency.

Contribution

It proposes a novel paradigm that explicitly separates data collection from inference in RL, guiding future research in data-efficient algorithms.

Findings

Highlights the importance of data collection strategies in RL.

Connects existing methods under the new 'Collect and Infer' framework.

Suggests directions for future research in data-efficient RL.

Abstract

This position paper proposes a fresh look at Reinforcement Learning (RL) from the perspective of data-efficiency. Data-efficient RL has gone through three major stages: pure on-line RL where every data-point is considered only once, RL with a replay buffer where additional learning is done on a portion of the experience, and finally transition memory based RL, where, conceptually, all transitions are stored and re-used in every update step. While inferring knowledge from all explicitly stored experience has lead to a tremendous gain in data-efficiency, the question of how this data is collected has been vastly understudied. We argue that data-efficiency can only be achieved through careful consideration of both aspects. We propose to make this insight explicit via a paradigm that we call 'Collect and Infer', which explicitly models RL as two separate but interconnected processes, concerned with data collection and knowledge inference respectively. We discuss implications of the paradigm, how its ideas are reflected in the literature, and how it can guide future research into data efficient RL.