A State Representation for Diminishing Rewards
Ted Moskovitz, Samo Hromadka, Ahmed Touati, Diana Borsa, Maneesh, Sahani

TL;DR
This paper introduces the $ ext{ extlambda}$ representation ($ ext{ extlambda}$R), a new state representation for reinforcement learning that generalizes successor representations to better handle diminishing rewards and shifting priorities.
Contribution
The paper proposes the $ ext{ extlambda}$R, a novel state representation that extends successor representations to account for diminishing marginal utility in sequential tasks.
Findings
$ ext{ extlambda}$R generalizes SR and other state representations.
$ ext{ extlambda}$R has normative advantages in machine learning.
$ ext{ extlambda}$R is useful for modeling natural foraging behaviors.
Abstract
A common setting in multitask reinforcement learning (RL) demands that an agent rapidly adapt to various stationary reward functions randomly sampled from a fixed distribution. In such situations, the successor representation (SR) is a popular framework which supports rapid policy evaluation by decoupling a policy's expected discounted, cumulative state occupancies from a specific reward function. However, in the natural world, sequential tasks are rarely independent, and instead reflect shifting priorities based on the availability and subjective perception of rewarding stimuli. Reflecting this disjunction, in this paper we study the phenomenon of diminishing marginal utility and introduce a novel state representation, the representation (R) which, surprisingly, is required for policy evaluation in this setting and which generalizes the SR as well as several other…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsGame Theory and Applications · Reinforcement Learning in Robotics · Auction Theory and Applications
