GAGA: Deciphering Age-path of Generalized Self-paced Regularizer

TL;DR

GAGA is a novel algorithm that efficiently computes the entire solution path for self-paced learning with various regularizers, addressing limitations of previous methods and enabling better understanding and optimization of the age parameter.

Contribution

GAGA introduces the first exact path-following algorithm for the age-path in self-paced learning with general regularizers, using ODEs and set control for improved efficiency.

Findings

Significant speedup over existing algorithms

Effective learning of the entire solution spectrum

Demonstrated on real-world datasets

Abstract

Nowadays self-paced learning (SPL) is an important machine learning paradigm that mimics the cognitive process of humans and animals. The SPL regime involves a self-paced regularizer and a gradually increasing age parameter, which plays a key role in SPL but where to optimally terminate this process is still non-trivial to determine. A natural idea is to compute the solution path w.r.t. age parameter (i.e., age-path). However, current age-path algorithms are either limited to the simplest regularizer, or lack solid theoretical understanding as well as computational efficiency. To address this challenge, we propose a novel \underline{G}eneralized \underline{Ag}e-path \underline{A}lgorithm (GAGA) for SPL with various self-paced regularizers based on ordinary differential equations (ODEs) and sets control, which can learn the entire solution spectrum w.r.t. a range of age parameters. To the best of our knowledge, GAGA is the first exact path-following algorithm tackling the age-path for general self-paced regularizer. Finally the algorithmic steps of classic SVM and Lasso are described in detail. We demonstrate the performance of GAGA on real-world datasets, and find considerable speedup between our algorithm and competing baselines.