A Correction Method of a Binary Classifier Applied to Multi-label Pairwise Models

TL;DR

This paper introduces a novel correction method for binary classifiers in multi-label pairwise models, improving error estimation and handling imbalanced labels, with significant gains in zero-one loss across benchmark datasets.

Contribution

The paper proposes a new correction approach that estimates classifier competence and cross-competence, enhancing multi-label classification accuracy, especially for imbalanced datasets.

Findings

Significantly outperforms baseline in zero-one loss

Effective in handling imbalanced labels

Improves error estimation in multi-label ensembles

Abstract

In this work, we addressed the issue of applying a stochastic classifier and a local, fuzzy confusion matrix under the framework of multi-label classification. We proposed a novel solution to the problem of correcting label pairwise ensembles. The main step of the correction procedure is to compute classifier- specific competence and cross-competence measures, which estimates error pattern of the underlying classifier. We considered two improvements of the method of obtaining confusion matrices. The first one is aimed to deal with imbalanced labels. The other utilizes double labelled instances which are usually removed during the pairwise transformation. The proposed methods were evaluated using 29 benchmark datasets. In order to assess the efficiency of the introduced models, they were compared against 1 state-of-the-art approach and the correction scheme based on the original method of confusion matrix estimation. The comparison was performed using four different multi-label evaluation measures: macro and micro-averaged F1 loss, zero-one loss and Hamming loss. Additionally, we investigated relations between classification quality, which is expressed in terms of different quality criteria, and characteristics of multi-label datasets such as average imbalance ratio or label density. The experimental study reveals that the correction approaches significantly outperforms the reference method only in terms of zero-one loss.