Orthogonal Activation with Implicit Group-Aware Bias Learning for Class Imbalance

TL;DR

This paper introduces OGAB, a novel orthogonal activation function with group-aware bias learning, designed to improve deep learning classifier performance on imbalanced datasets by enhancing feature independence and class separability.

Contribution

The work proposes a new activation function, OGAB, that incorporates orthogonality and bias learning to address class imbalance without explicit supervision, improving feature discrimination.

Findings

OGAB improves minority class recognition in imbalanced datasets.

The method outperforms traditional activation functions on real-world data.

It effectively maintains feature independence for minority classes.

Abstract

Class imbalance is a common challenge in machine learning and data mining, often leading to suboptimal performance in classifiers. While deep learning excels in feature extraction, its performance still deteriorates under imbalanced data. In this work, we propose a novel activation function, named OGAB, designed to alleviate class imbalance in deep learning classifiers. OGAB incorporates orthogonality and group-aware bias learning to enhance feature distinguishability in imbalanced scenarios without explicitly requiring label information. Our key insight is that activation functions can be used to introduce strong inductive biases that can address complex data challenges beyond traditional non-linearity. Our work demonstrates that orthogonal transformations can preserve information about minority classes by maintaining feature independence, thereby preventing the dominance of majority classes in the embedding space. Further, the proposed group-aware bias mechanism automatically identifies data clusters and adjusts embeddings to enhance class separability without the need for explicit supervision. Unlike existing approaches that address class imbalance through preprocessing data modifications or post-processing corrections, our proposed approach tackles class imbalance during the training phase at the embedding learning level, enabling direct integration with the learning process. We demonstrate the effectiveness of our solution on both real-world and synthetic imbalanced datasets, showing consistent performance improvements over both traditional and learnable activation functions.