Proximal Bellman mappings for reinforcement learning and their application to robust adaptive filtering

TL;DR

This paper introduces proximal Bellman mappings in RKHSs for reinforcement learning, enabling robust adaptive filtering by selecting optimal p-norm exponents online, with demonstrated superior performance over existing methods.

Contribution

It proposes a novel class of proximal Bellman mappings in RKHSs, facilitating new RL designs and robust adaptive filtering without prior data or outlier statistics.

Findings

Numerical tests show superior performance over existing schemes.

Mappings are nonexpansive and versatile for RL design.

Effective outlier handling in adaptive filtering.

Abstract

This paper aims at the algorithmic/theoretical core of reinforcement learning (RL) by introducing the novel class of proximal Bellman mappings. These mappings are defined in reproducing kernel Hilbert spaces (RKHSs), to benefit from the rich approximation properties and inner product of RKHSs, they are shown to belong to the powerful Hilbertian family of (firmly) nonexpansive mappings, regardless of the values of their discount factors, and possess ample degrees of design freedom to even reproduce attributes of the classical Bellman mappings and to pave the way for novel RL designs. An approximate policy-iteration scheme is built on the proposed class of mappings to solve the problem of selecting online, at every time instance, the "optimal" exponent $p$ in a $p$-norm loss to combat outliers in linear adaptive filtering, without training data and any knowledge on the statistical properties of the outliers. Numerical tests on synthetic data showcase the superior performance of the proposed framework over several non-RL and kernel-based RL schemes.