Approximating Fixpoints of Approximated Functions

TL;DR

This paper introduces a modified Mann iteration scheme to approximate the least fixpoint of monotone, non-expansive functions, with applications in reinforcement learning and probabilistic systems, ensuring convergence despite uncertainties.

Contribution

It proposes a dampening-based iteration method that guarantees convergence to the least fixpoint for approximated functions, extending applicability to probabilistic and reinforcement learning contexts.

Findings

The iteration scheme converges to the least fixpoint under certain conditions.

Applicable to systems with probabilistic approximation errors.

Demonstrated relevance in model-based reinforcement learning and stochastic systems.

Abstract

Fixpoints are ubiquitous in computer science and when dealing with quantitative semantics and verification one often considers least fixpoints of (higher-dimensional) functions over the non-negative reals. We show how to approximate the least fixpoint of such functions, focusing on the case in which they are not known precisely, but represented by a sequence of approximating functions that converge to them. We concentrate on monotone and non-expansive functions, for which uniqueness of fixpoints is not guaranteed and standard fixpoint iteration schemes might get stuck at a fixpoint that is not the least. Our main contribution is the identification of an iteration scheme, a variation of Mann iteration with a dampening factor, which, under suitable conditions, is shown to guarantee convergence to the least fixpoint of the function of interest. We then argue that these results are relevant in the context of model-based reinforcement learning for Markov decision processes, showing how the proposed iteration scheme instantiates and allows us to derive convergence to the optimal expected return. More generally, we show that our results can be used to iterate to the least fixpoint almost surely for systems where the function of interest can be approximated with given probabilistic error bounds, as it happens for probabilistic systems, such as simple stochastic games, which can be explored via sampling.