Hyperbolically-Discounted Reinforcement Learning on Reward-Punishment Framework

TL;DR

This paper introduces a novel reinforcement learning approach using hyperbolic discounting combined with reward-punishment frameworks, leading to improved policy learning that reflects more realistic discounting behaviors observed in animals.

Contribution

It presents a new hyperbolic discounting scheme integrated with a recursive temporal difference error for reinforcement learning, which outperforms standard methods.

Findings

Outperforms standard reinforcement learning in simulations

Discount factors for reward and punishment differ, resembling animal behavior

Performance depends on reward and punishment design

Abstract

This paper proposes a new reinforcement learning with hyperbolic discounting. Combining a new temporal difference error with the hyperbolic discounting in recursive manner and reward-punishment framework, a new scheme to learn the optimal policy is derived. In simulations, it is found that the proposal outperforms the standard reinforcement learning, although the performance depends on the design of reward and punishment. In addition, the averages of discount factors w.r.t. reward and punishment are different from each other, like a sign effect in animal behaviors.