# Reinforcement with Fading Memories

## TL;DR

This paper analyzes how imperfect memory affects decision-making in stochastic sequential tasks, deriving formulas for steady-state choices and highlighting the importance of the update-to-decay rate ratio.

## Contribution

It provides closed-form solutions for the steady-state choice distribution under large memory span and elucidates the critical role of update and decay rates in decision accuracy.

## Key findings

- When update rate exceeds decay rate, the agent nearly always chooses the optimal action.
- If decay rate exceeds update rate, choices are proportional to reward rates.
- The model offers insights into decision-making with fading memories in stochastic environments.

## Abstract

We study the effect of imperfect memory on decision making in the context of a stochastic sequential action-reward problem. An agent chooses a sequence of actions which generate discrete rewards at different rates. She is allowed to make new choices at rate $\beta$, while past rewards disappear from her memory at rate $\mu$. We focus on a family of decision rules where the agent makes a new choice by randomly selecting an action with a probability approximately proportional to the amount of past rewards associated with each action in her memory. We provide closed-form formulae for the agent's steady-state choice distribution in the regime where the memory span is large ($\mu \to 0$), and show that the agent's success critically depends on how quickly she updates her choices relative to the speed of memory decay. If $\beta \gg \mu$, the agent almost always chooses the best action, i.e., the one with the highest reward rate. Conversely, if $\beta \ll \mu$, the agent chooses an action with a probability roughly proportional to its reward rate.