Passive Learning with Target Risk

TL;DR

This paper introduces a passive learning approach that leverages prior knowledge of target risk, leading to exponentially reduced sample complexity for strongly convex and smooth loss functions, with a practical stochastic optimization algorithm.

Contribution

It explicitly incorporates target risk into the learning process, achieving exponential sample complexity reduction and providing a practical, efficient algorithm.

Findings

Sample complexity reduces to O(log(1/ε)) for certain loss functions.

The proposed algorithm is computationally efficient and practically useful.

Theoretical analysis confirms exponential improvement over traditional methods.

Abstract

In this paper we consider learning in passive setting but with a slight modification. We assume that the target expected loss, also referred to as target risk, is provided in advance for learner as prior knowledge. Unlike most studies in the learning theory that only incorporate the prior knowledge into the generalization bounds, we are able to explicitly utilize the target risk in the learning process. Our analysis reveals a surprising result on the sample complexity of learning: by exploiting the target risk in the learning algorithm, we show that when the loss function is both strongly convex and smooth, the sample complexity reduces to $\O(\log (\frac{1}{\epsilon}))$, an exponential improvement compared to the sample complexity $\O(\frac{1}{\epsilon})$ for learning with strongly convex loss functions. Furthermore, our proof is constructive and is based on a computationally efficient stochastic optimization algorithm for such settings which demonstrate that the proposed algorithm is practically useful.