TabMixer: Excavating Label Distribution Learning with Small-scale Features

TL;DR

TabMixer introduces a novel approach to label distribution learning by modeling feature uncertainty with Gaussian augmentation and a local attention mixer, improving performance on small-scale feature datasets.

Contribution

The paper proposes a new method combining Gaussian-based feature augmentation with a local attention mixer to enhance LDL performance on small feature spaces.

Findings

Outperforms existing LDL algorithms on multiple benchmarks.

Effectively models feature uncertainty to improve label distribution estimation.

Demonstrates robustness in small-scale feature scenarios.

Abstract

Label distribution learning (LDL) differs from multi-label learning which aims at representing the polysemy of instances by transforming single-label values into descriptive degrees. Unfortunately, the feature space of the label distribution dataset is affected by human factors and the inductive bias of the feature extractor causing uncertainty in the feature space. Especially, for datasets with small-scale feature spaces (the feature space dimension $\approx$ the label space), the existing LDL algorithms do not perform well. To address this issue, we seek to model the uncertainty augmentation of the feature space to alleviate the problem in LDL tasks. Specifically, we start with augmenting each feature value in the feature vector of a sample into a vector (sampling on a Gaussian distribution function). Which, the variance parameter of the Gaussian distribution function is learned by using a sub-network, and the mean parameter is filled by this feature value. Then, each feature vector is augmented to a matrix which is fed into a mixer with local attention (\textit{TabMixer}) to extract the latent feature. Finally, the latent feature is squeezed to yield an accurate label distribution via a squeezed network. Extensive experiments verify that our proposed algorithm can be competitive compared to other LDL algorithms on several benchmarks.