Importance sampling strategy for non-convex randomized block-coordinate descent

TL;DR

This paper introduces an importance sampling strategy for randomized block-coordinate descent algorithms, improving convergence focus in large-scale non-convex optimization problems common in statistics and machine learning.

Contribution

It proposes a novel importance sampling framework tailored for non-convex, non-smooth optimization problems, enhancing the efficiency of block coordinate descent methods.

Findings

Importance sampling improves convergence speed.

Experimental results favor the proposed method over uniform sampling.

The approach is effective for large-scale non-convex problems.

Abstract

As the number of samples and dimensionality of optimization problems related to statistics an machine learning explode, block coordinate descent algorithms have gained popularity since they reduce the original problem to several smaller ones. Coordinates to be optimized are usually selected randomly according to a given probability distribution. We introduce an importance sampling strategy that helps randomized coordinate descent algorithms to focus on blocks that are still far from convergence. The framework applies to problems composed of the sum of two possibly non-convex terms, one being separable and non-smooth. We have compared our algorithm to a full gradient proximal approach as well as to a randomized block coordinate algorithm that considers uniform sampling and cyclic block coordinate descent. Experimental evidences show the clear benefit of using an importance sampling strategy.