Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations

TL;DR

This paper introduces Batch Nuclear-norm Maximization (BNM), a novel method to enhance discriminability and diversity in deep network training under label-insufficient conditions, improving performance in semi-supervised learning, domain adaptation, and open domain recognition.

Contribution

The paper proposes BNM, a new approach that maximizes the nuclear-norm of batch output matrices to improve discriminability and diversity, with theoretical analysis and extensive experiments showing its effectiveness.

Findings

BNM outperforms existing methods in semi-supervised learning.

BNM enhances performance in domain adaptation tasks.

BNM effectively improves diversity and discriminability in various scenarios.

Abstract

The learning of the deep networks largely relies on the data with human-annotated labels. In some label insufficient situations, the performance degrades on the decision boundary with high data density. A common solution is to directly minimize the Shannon Entropy, but the side effect caused by entropy minimization, i.e., reduction of the prediction diversity, is mostly ignored. To address this issue, we reinvestigate the structure of classification output matrix of a randomly selected data batch. We find by theoretical analysis that the prediction discriminability and diversity could be separately measured by the Frobenius-norm and rank of the batch output matrix. Besides, the nuclear-norm is an upperbound of the Frobenius-norm, and a convex approximation of the matrix rank. Accordingly, to improve both discriminability and diversity, we propose Batch Nuclear-norm Maximization (BNM) on the output matrix. BNM could boost the learning under typical label insufficient learning scenarios, such as semi-supervised learning, domain adaptation and open domain recognition. On these tasks, extensive experimental results show that BNM outperforms competitors and works well with existing well-known methods. The code is available at https://github.com/cuishuhao/BNM.