Data-driven rules for multidimensional reflection problems

TL;DR

This paper advances the understanding of multivariate singular control problems for reversible diffusions by explicitly characterizing long-term costs, proposing a shape optimization approach, and developing data-driven algorithms for unknown dynamics.

Contribution

It extends stochastic control analysis to multivariate cases, introduces a shape optimization framework, and develops data-driven algorithms for unknown diffusion dynamics.

Findings

Explicit long-run average cost functional derived

Gradient descent algorithm for domain optimization proposed

Data-driven domain estimation with minimax optimal rate achieved

Abstract

Over the recent past data-driven algorithms for solving stochastic optimal control problems in face of model uncertainty have become an increasingly active area of research. However, for singular controls and underlying diffusion dynamics the analysis has so far been restricted to the scalar case. In this paper we fill this gap by studying a multivariate singular control problem for reversible diffusions with controls of reflection type. Our contributions are threefold. We first explicitly determine the long-run average costs as a domain-dependent functional, showing that the control problem can be equivalently characterized as a shape optimization problem. For given diffusion dynamics, assuming the optimal domain to be strongly star-shaped, we then propose a gradient descent algorithm based on polytope approximations to numerically determine a cost-minimizing domain. Finally, we investigate data-driven solutions when the diffusion dynamics are unknown to the controller. Using techniques from nonparametric statistics for stochastic processes, we construct an optimal domain estimator, whose static regret is bounded by the minimax optimal estimation rate of the unreflected process' invariant density. In the most challenging situation, when the dynamics must be learned simultaneously to controlling the process, we develop an episodic learning algorithm to overcome the emerging exploration-exploitation dilemma and show that given the static regret as a baseline, the loss in its sublinear regret per time unit is of natural order compared to the one-dimensional case.