Understanding Self-Paced Learning under Concave Conjugacy Theory

TL;DR

This paper introduces a concave conjugacy theory to analyze and design self-paced learning regimes, providing a unified framework that explains their effectiveness and guides the development of new models.

Contribution

It proposes a novel theoretical framework linking SPL to latent concave objectives, enabling better understanding and design of curriculum learning methods.

Findings

Proves SPL regime equivalence to a latent concave objective

Facilitates design of new SPL models for specific tasks

Provides theoretical insights into partial order and group curriculums

Abstract

By simulating the easy-to-hard learning manners of humans/animals, the learning regimes called curriculum learning~(CL) and self-paced learning~(SPL) have been recently investigated and invoked broad interests. However, the intrinsic mechanism for analyzing why such learning regimes can work has not been comprehensively investigated. To this issue, this paper proposes a concave conjugacy theory for looking into the insight of CL/SPL. Specifically, by using this theory, we prove the equivalence of the SPL regime and a latent concave objective, which is closely related to the known non-convex regularized penalty widely used in statistics and machine learning. Beyond the previous theory for explaining CL/SPL insights, this new theoretical framework on one hand facilitates two direct approaches for designing new SPL models for certain tasks, and on the other hand can help conduct the latent objective of self-paced curriculum learning, which is the advanced version of both CL/SPL and possess advantages of both learning regimes to a certain extent. This further facilitates a theoretical understanding for SPCL, instead of only CL/SPL as conventional. Under this theory, we attempt to attain intrinsic latent objectives of two curriculum forms, the partial order and group curriculums, which easily follow the theoretical understanding of the corresponding SPCL regimes.