Sign-Constrained Regularized Loss Minimization

TL;DR

This paper introduces sign constraints into regularized loss minimization, developing two algorithms with theoretical guarantees, and demonstrates improved generalization in applications like correlation exploitation and SVM-Pairwise.

Contribution

It proposes sign-constrained optimization algorithms (SC-Pega and SC-SDCA) with theoretical convergence guarantees and shows their effectiveness in practical applications.

Findings

Sign constraints improve generalization performance.

The algorithms maintain convergence rates despite sign correction steps.

Applications demonstrate significant performance improvements.

Abstract

In practical analysis, domain knowledge about analysis target has often been accumulated, although, typically, such knowledge has been discarded in the statistical analysis stage, and the statistical tool has been applied as a black box. In this paper, we introduce sign constraints that are a handy and simple representation for non-experts in generic learning problems. We have developed two new optimization algorithms for the sign-constrained regularized loss minimization, called the sign-constrained Pegasos (SC-Pega) and the sign-constrained SDCA (SC-SDCA), by simply inserting the sign correction step into the original Pegasos and SDCA, respectively. We present theoretical analyses that guarantee that insertion of the sign correction step does not degrade the convergence rate for both algorithms. Two applications, where the sign-constrained learning is effective, are presented. The one is exploitation of prior information about correlation between explanatory variables and a target variable. The other is introduction of the sign-constrained to SVM-Pairwise method. Experimental results demonstrate significant improvement of generalization performance by introducing sign constraints in both applications.