On solving convex min-min problems with smoothness and strong convexity in one variable group and small dimension of the other

TL;DR

This paper introduces efficient algorithms for convex min-min problems with one strongly convex variable group, achieving linear convergence, and demonstrates their effectiveness in machine learning tasks like logistic regression.

Contribution

It proposes novel approaches combining Vaidya's cutting plane method and Nesterov's accelerated gradient for solving specific convex min-min problems with proven linear convergence.

Findings

Algorithms outperform traditional methods in convergence speed.

Numerical experiments show advantages in logistic regression.

Variance reduction improves performance with large data sets.

Abstract

This paper is devoted to some approaches for convex min-min problems with smoothness and strong convexity in only one of the two variable groups. It is shown that the proposed approaches, based on Vaidya's cutting plane method and Nesterov's fast gradient method, achieve the linear convergence. The outer minimization problem is solved using Vaidya's cutting plane method, and the inner problem (smooth and strongly convex) is solved using the fast gradient method. Due to the importance of machine learning applications, we also consider the case when the objective function is a sum of a large number of functions. In this case, the variance-reduced accelerated gradient algorithm is used instead of Nesterov's fast gradient method. The numerical experiments' results illustrate the advantages of the proposed procedures for logistic regression with the prior on one of the parameter groups.