Reinforcement-learning-based Algorithms for Optimization Problems and Applications to Inverse Problems

TL;DR

This paper introduces REINFORCE-OPT, a reinforcement learning-based iterative algorithm for optimization and inverse problems, demonstrating superior performance and robustness over traditional methods, with applications to nonlinear inverse problems and uncertainty quantification.

Contribution

The paper presents a novel RL-based optimization algorithm, REINFORCE-OPT, and establishes its theoretical convergence, practical advantages, and applicability to inverse problems with uncertainty quantification.

Findings

REINFORCE-OPT outperforms gradient descent, genetic algorithms, and particle swarm optimization.

The method effectively escapes local optima and is robust to initial conditions.

It can quantify uncertainty and identify multiple solutions in ill-posed inverse problems.

Abstract

We design a new iterative algorithm, called REINFORCE-OPT, for solving a general type of optimization problems. This algorithm parameterizes the solution search rule and iteratively updates the parameter using a reinforcement learning (RL) algorithm resembling REINFORCE. To gain a deeper understanding of the RL-based methods, we show that REINFORCE-OPT essentially solves a stochastic version of the given optimization problem, and that under standard assumptions, the searching rule parameter almost surely converges to a locally optimal value. Experiments show that REINFORCE-OPT outperforms other optimization methods such as gradient descent, the genetic algorithm, and particle swarm optimization, via its ability to escape from locally optimal solutions and its robustness to the choice of initial values. With rigorous derivations, we formally introduce the use of reinforcement learning to deal with inverse problems. By choosing specific probability models for the action-selection rule, we can also connect our approach to the conventional methods of Tikhonov regularization and iterative regularization. We take non-linear integral equations and parameter-identification problems in partial differential equations as examples to show how reinforcement learning can be applied in solving non-linear inverse problems. The numerical experiments highlight the strong performance of REINFORCE-OPT, as well as its ability to quantify uncertainty in error estimates and identify multiple solutions for ill-posed inverse problems that lack solution stability and uniqueness.