Novel Optimization Techniques for Parameter Estimation

TL;DR

This paper presents CRNAS, a second-order optimization algorithm tailored for parameter estimation in non-convex biological models, leveraging Hessian information and affine scaling to improve convergence to second-order optimal points.

Contribution

The paper introduces CRNAS, a novel cubic regularized Newton method with affine scaling for constrained parameter estimation, capable of efficiently finding second-order optimal points.

Findings

CRNAS converges within $O(\epsilon^{-3/2})$ iterations to approximate second-order optimality.

CRNAS performs comparably or better than MATLAB's fmincon in accuracy and computational cost.

CRNAS is particularly effective for problems with heterogeneous populations in biological systems.

Abstract

In this paper, we introduce a new optimization algorithm that is well suited for solving parameter estimation problems. We call our new method cubic regularized Newton with affine scaling (CRNAS). In contrast to so-called first-order methods which rely solely on the gradient of the objective function, our method utilizes the Hessian of the objective. As a result it is able to focus on points satisfying the second-order optimality conditions, as opposed to first-order methods that simply converge to critical points. This is an important feature in parameter estimation problems where the objective function is often non-convex and as a result there can be many critical points making it is near impossible to identify the global minimum. An important feature of parameter estimation in mathematical models of biological systems is that the parameters are constrained by either physical constraints or prior knowledge. We use an affine scaling approach to handle a wide class of constraints. We establish that CRNAS identifies a point satisfying $\epsilon$-approximate second-order optimality conditions within $O(\epsilon^{-3/2})$ iterations. Finally, we compare CRNAS with MATLAB's optimization solver fmincon on three different test problems. These test problems all feature mixtures of heterogeneous populations, a problem setting that CRNAS is particularly well-suited for. Our numerical simulations show CRNAS has favorable performance, performing comparable if not better than fmincon in accuracy and computational cost for most of our examples.