Binary Classification from Positive Data with Skewed Confidence

TL;DR

This paper addresses the challenge of skewed confidence in positive-confidence classification by introducing a parameterized model and hyperparameter selection method, improving performance in both synthetic and real-world scenarios.

Contribution

It proposes a novel model for skewed confidence and a hyperparameter selection technique that mitigates bias effects in positive-confidence classification.

Findings

Effective in synthetic experiments with linear models

Improves performance on benchmark neural network tasks

Successfully applied to drivers' drowsiness prediction

Abstract

Positive-confidence (Pconf) classification [Ishida et al., 2018] is a promising weakly-supervised learning method which trains a binary classifier only from positive data equipped with confidence. However, in practice, the confidence may be skewed by bias arising in an annotation process. The Pconf classifier cannot be properly learned with skewed confidence, and consequently, the classification performance might be deteriorated. In this paper, we introduce the parameterized model of the skewed confidence, and propose the method for selecting the hyperparameter which cancels out the negative impact of skewed confidence under the assumption that we have the misclassification rate of positive samples as a prior knowledge. We demonstrate the effectiveness of the proposed method through a synthetic experiment with simple linear models and benchmark problems with neural network models. We also apply our method to drivers' drowsiness prediction to show that it works well with a real-world problem where confidence is obtained based on manual annotation.