Estimating scale-invariant future in continuous time

TL;DR

This paper introduces a scale-invariant timeline mechanism for estimating future outcomes in continuous time, inspired by psychology and neuroscience, which could improve reinforcement learning by providing efficient, flexible predictions of future rewards.

Contribution

It proposes a novel, scale-invariant computational mechanism for representing future time on a logarithmic scale, addressing limitations of existing RL algorithms.

Findings

Efficiently computes future inputs on a logarithmic scale.

Generates scale-invariant power-law discounting of future rewards.

Produces neural and behavioral predictions for future time representation.

Abstract

Natural learners must compute an estimate of future outcomes that follow from a stimulus in continuous time. Widely used reinforcement learning algorithms discretize continuous time and estimate either transition functions from one step to the next (model-based algorithms) or a scalar value of exponentially-discounted future reward using the Bellman equation (model-free algorithms). An important drawback of model-based algorithms is that computational cost grows linearly with the amount of time to be simulated. On the other hand, an important drawback of model-free algorithms is the need to select a time-scale required for exponential discounting. We present a computational mechanism, developed based on work in psychology and neuroscience, for computing a scale-invariant timeline of future outcomes. This mechanism efficiently computes an estimate of inputs as a function of future time on a logarithmically-compressed scale, and can be used to generate a scale-invariant power-law-discounted estimate of expected future reward. The representation of future time retains information about what will happen when. The entire timeline can be constructed in a single parallel operation which generates concrete behavioral and neural predictions. This computational mechanism could be incorporated into future reinforcement learning algorithms.