From the simplex to the sphere: Faster constrained optimization using the Hadamard parametrization

TL;DR

This paper introduces a novel approach to constrained optimization by transforming the probability simplex into a sphere, enabling the use of efficient, projection-free algorithms on a smooth manifold, with demonstrated advantages in numerical experiments.

Contribution

The paper presents a Hadamard parametrization that converts the simplex constraint into a sphere, facilitating faster, projection-free optimization algorithms with theoretical equivalence to traditional methods.

Findings

The transformation preserves KKT and strict-saddle points.

The new algorithms outperform existing methods in experiments.

Extension to various simplex and `1-norm sphere constraints is possible.

Abstract

The standard simplex in R^n, also known as the probability simplex, is the set of nonnegative vectors whose entries sum up to 1. They frequently appear as constraints in optimization problems that arise in machine learning, statistics, data science, operations research, and beyond. We convert the standard simplex to the unit sphere and thus transform the corresponding constrained optimization problem into an optimization problem on a simple, smooth manifold. We show that KKT points and strict-saddle points of the minimization problem on the standard simplex all correspond to those of the transformed problem, and vice versa. So, solving one problem is equivalent to solving the other problem. Then, we propose several simple, efficient, and projection-free algorithms using the manifold structure. The equivalence and the proposed algorithm can be extended to optimization problems with unit simplex, weighted probability simplex, or `1-norm sphere constraints. Numerical experiments between the new algorithms and existing ones show the advantages of the new approach