T-Learning

TL;DR

T-Learning introduces a novel reinforcement learning approach that efficiently handles problems with few states but many actions by focusing on state transitions rather than state-action pairs.

Contribution

The paper proposes T-Learning, a new RL method that scales better with many actions by evaluating state transitions independently of policies.

Findings

T-Learning outperforms Q-learning in scenarios with many actions.

Performance improvements of T-Learning can be arbitrarily large.

T-Learning effectively handles real-world problems with few relevant states and many actions.

Abstract

Traditional Reinforcement Learning (RL) has focused on problems involving many states and few actions, such as simple grid worlds. Most real world problems, however, are of the opposite type, Involving Few relevant states and many actions. For example, to return home from a conference, humans identify only few subgoal states such as lobby, taxi, airport etc. Each valid behavior connecting two such states can be viewed as an action, and there are trillions of them. Assuming the subgoal identification problem is already solved, the quality of any RL method---in real-world settings---depends less on how well it scales with the number of states than on how well it scales with the number of actions. This is where our new method T-Learning excels, by evaluating the relatively few possible transits from one state to another in a policy-independent way, rather than a huge number of state-action pairs, or states in traditional policy-dependent ways. Illustrative experiments demonstrate that performance improvements of T-Learning over Q-learning can be arbitrarily large.