An entropy penalized approach for stochastic control problems. Complete version

TL;DR

This paper introduces an entropy penalized approach to stochastic control problems, transforming them into more tractable subproblems and providing an iterative solution method with proven convergence, demonstrated on high-dimensional electrical system control.

Contribution

It presents a novel entropy penalization technique for stochastic control, enabling efficient alternating optimization and convergence analysis.

Findings

The penalized problem approximates the original well with high entropy weight.

The proposed method converges at a rate of O(1/k).

Effective in high-dimensional electrical system control scenarios.

Abstract

In this paper, we propose an original approach to stochastic control problems. We consider a weak formulation that is written as an optimization (minimization) problem on the space of probability measures. We then introduce a penalized version of this problem obtained by splitting the minimization variables and penalizing the discrepancy between the two variables via an entropy term. We show that the penalized problem provides a good approximation of the original problem when the weight of the entropy penalization term is large enough. Moreover, the penalized problem has the advantage of giving rise to two optimization subproblems that are easy to solve in each of the two optimization variables when the other is fixed. We take advantage of this property to propose an alternating optimization procedure that converges to the infimum of the penalized problem with a rate $O(1/k)$, where $k$ is the number of iterations. The relevance of this approach is illustrated by solving a high-dimensional stochastic control problem aimed at controlling consumption in electrical systems.