# From Task Distributions to Expected Paths Lengths Distributions: Value Function Initialization in Sparse Reward Environments for Lifelong Reinforcement Learning

## TL;DR

This paper introduces a new method for initializing value functions in reinforcement learning by analyzing the distribution of path lengths to goals in sparse reward environments.

## Contribution

The paper proposes LogQInit, a novel value function transfer method based on the log-normal distribution of expected optimal path lengths.

## Key findings

- The distribution of expected optimal path lengths is approximately normal.
- The value function distribution follows a log-normal pattern under this assumption.
- LogQInit outperforms existing methods in value function initialization and policy transfer.

## Abstract

This paper studies value function transfer within reinforcement learning frameworks, focusing on tasks continuously assigned to an agent through a probabilistic distribution. Specifically, we focus on environments characterized by sparse rewards with a terminal goal. Initially, we propose and theoretically demonstrate that the distribution of the computed value function from such environments, whether in cases where the goals or the dynamics are changing across tasks, can be reformulated as the distribution of the number of steps to the goal generated by their optimal policies, which we name the expected optimal path length. To test our propositions, we hypothesize that the distribution of the expected optimal path lengths resulting from the task distribution is normal. This claim leads us to propose that if the distribution is normal, then the distribution of the value function follows a log-normal pattern. Leveraging this insight, we introduce “LogQInit” as a novel value function transfer method, based on the properties of log-normality. Finally, we run experiments on a scenario of goals and dynamics distributions, validate our proposition by providing an a dequate analysis of the results, and demonstrate that LogQInit outperforms existing methods of value function initialization, policy transfer, and reward shaping.