High-dimensional Optimization with Low Rank Tensor Sampling and Local Search

TL;DR

This paper introduces TESALOCS, a hybrid high-dimensional optimization method combining low-rank tensor sampling with local search, outperforming traditional gradient-based and discrete methods on challenging functions.

Contribution

The paper proposes TESALOCS, a novel hybrid approach that integrates low-rank tensor sampling with local search to efficiently optimize high-dimensional functions.

Findings

Outperforms gradient-based methods by orders of magnitude on 20 challenging problems

Efficiently handles 100-dimensional optimization tasks

Demonstrates the effectiveness of low-rank tensor techniques in high-dimensional optimization

Abstract

We present a novel method called TESALOCS (TEnsor SAmpling and LOCal Search) for multidimensional optimization, combining the strengths of gradient-free discrete methods and gradient-based approaches. The discrete optimization in our method is based on low-rank tensor techniques, which, thanks to their low-parameter representation, enable efficient optimization of high-dimensional problems. For the second part, i.e., local search, any effective gradient-based method can be used, whether existing (such as quasi-Newton methods) or any other developed in the future. Our approach addresses the limitations of gradient-based methods, such as getting stuck in local optima; the limitations of discrete methods, which cannot be directly applied to continuous functions; and limitations of gradient-free methods that require large computational budgets. Note that we are not limited to a single type of low-rank tensor decomposition for discrete optimization, but for illustrative purposes, we consider a specific efficient low-rank tensor train decomposition. For 20 challenging 100-dimensional functions, we demonstrate that our method can significantly outperform results obtained with gradient-based methods like Conjugate Gradient, BFGS, SLSQP, and other methods, improving them by orders of magnitude with the same computing budget.