# Alice's Adventures in the Markovian World

## TL;DR

This paper introduces Alice, an online decision-making algorithm for linear stochastic systems that learns without prior knowledge or training, extending Follow-the-Leader with regularization and constraints, and provides theoretical guarantees.

## Contribution

It generalizes Follow-the-Leader to linear Gauss-Markov processes with a novel regularization and online constraint, offering a no-regret proof without estimating system parameters.

## Key findings

- The algorithm achieves no-regret in linear stochastic environments.
- Simulations demonstrate Alice's flexibility and superior performance.
- Theoretical analysis provides conditions for no-regret guarantees.

## Abstract

This paper proposes an algorithm Alice having no access to the physics law of the environment, which is actually linear with stochastic noise, and learns to make decisions directly online without a training phase or a stable policy as initial input. Neither estimating the system parameters nor the value functions online, the proposed algorithm generalizes one of the most fundamental online learning algorithms Follow-the-Leader into a linear Gauss-Markov process setting, with a regularization term similar to the momentum method in the gradient descent algorithm, and a feasible online constraint inspired by Lyapunov's Second Theorem. The proposed algorithm is considered as a mirror optimization to the model predictive control. Only knowing the state-action alignment relationship, with the ability to observe every state exactly, a no-regret proof of the algorithm without state noise is given. The analysis of the general linear system with stochastic noise is shown with a sufficient condition for the no-regret proof. The simulations compare the performance of Alice with another recent work and verify the great flexibility of Alice.