Cost-Sensitive Self-Training for Optimizing Non-Decomposable Metrics

TL;DR

This paper introduces a cost-sensitive self-training framework that effectively optimizes complex, non-decomposable metrics in semi-supervised deep learning, outperforming existing methods across various tasks.

Contribution

The paper presents a novel CSST framework that generalizes self-training to optimize non-decomposable metrics, with theoretical guarantees and practical algorithms for vision and NLP.

Findings

CSST improves non-decomposable metric optimization in semi-supervised learning.

Experimental results show CSST outperforms state-of-the-art methods.

Framework applicable to both vision and NLP tasks.

Abstract

Self-training based semi-supervised learning algorithms have enabled the learning of highly accurate deep neural networks, using only a fraction of labeled data. However, the majority of work on self-training has focused on the objective of improving accuracy, whereas practical machine learning systems can have complex goals (e.g. maximizing the minimum of recall across classes, etc.) that are non-decomposable in nature. In this work, we introduce the Cost-Sensitive Self-Training (CSST) framework which generalizes the self-training-based methods for optimizing non-decomposable metrics. We prove that our framework can better optimize the desired non-decomposable metric utilizing unlabeled data, under similar data distribution assumptions made for the analysis of self-training. Using the proposed CSST framework, we obtain practical self-training methods (for both vision and NLP tasks) for optimizing different non-decomposable metrics using deep neural networks. Our results demonstrate that CSST achieves an improvement over the state-of-the-art in majority of the cases across datasets and objectives.