Scope Loss for Imbalanced Classification and RL Exploration

TL;DR

This paper introduces Scope Loss, a novel loss function that addresses exploration-exploitation and dataset imbalance issues in reinforcement learning and classification, outperforming state-of-the-art methods without tuning.

Contribution

The paper establishes an equivalence between RL and classification problems and proposes Scope Loss, a new loss function that improves performance by balancing gradients without tuning.

Findings

Scope Loss outperforms SOTA loss functions on benchmark RL tasks.

Scope Loss effectively handles dataset imbalance in classification.

No tuning required for Scope Loss to improve performance.

Abstract

We demonstrate equivalence between the reinforcement learning problem and the supervised classification problem. We consequently equate the exploration exploitation trade-off in reinforcement learning to the dataset imbalance problem in supervised classification, and find similarities in how they are addressed. From our analysis of the aforementioned problems we derive a novel loss function for reinforcement learning and supervised classification. Scope Loss, our new loss function, adjusts gradients to prevent performance losses from over-exploitation and dataset imbalances, without the need for any tuning. We test Scope Loss against SOTA loss functions over a basket of benchmark reinforcement learning tasks and a skewed classification dataset, and show that Scope Loss outperforms other loss functions.