Optimal adaptive control with separable drift uncertainty

TL;DR

This paper develops a framework for optimal adaptive control under separable drift uncertainty, employing stochastic Perron methods to characterize solutions and demonstrate the benefits of adaptive strategies over certainty equivalence.

Contribution

It introduces a novel approach to handle separable drift uncertainty in stochastic control by embedding the problem into a Markovian framework and applying viscosity solution techniques.

Findings

The value function is uniquely characterized as a viscosity solution to the HJB equation.

Explicit construction of ε-optimal controls is provided.

Adaptive control significantly outperforms certainty equivalence control in numerical tests.

Abstract

We consider a problem of stochastic optimal control with separable drift uncertainty in strong formulation on a finite horizon. The drift coefficient of the state $Y^{u}$ is multiplicatively influenced by an unknown random variable $\lambda$, while admissible controls $u$ are required to be adapted to the observation filtration. Choosing a control actively influences the state and information acquisition simultaneously and comes with a learning effect. The problem, initially non-Markovian, is embedded into a higher-dimensional Markovian, full information control problem with control-dependent filtration and noise. To that problem, we apply the stochastic Perron method to characterize the value function as the unique viscosity solution to the HJB equation, explicitly construct $\varepsilon$-optimal controls and show that the values of strong and weak formulations agree. Numerical illustrations show a significant difference between the adaptive control and the certainty equivalence control.