Decoupled Entropy Minimization

TL;DR

This paper introduces AdaDEM, a novel approach that decouples entropy minimization into separate components to improve learning in noisy and dynamic environments, addressing limitations of classical EM.

Contribution

The paper proposes a decoupled formulation of entropy minimization and introduces AdaDEM, which outperforms previous methods in imperfectly supervised learning tasks.

Findings

AdaDEM outperforms DEM* in various tasks.

Decoupling improves learning stability and accuracy.

Addresses classical EM limitations in noisy environments.

Abstract

Entropy Minimization (EM) is beneficial to reducing class overlap, bridging domain gap, and restricting uncertainty for various tasks in machine learning, yet its potential is limited. To study the internal mechanism of EM, we reformulate and decouple the classical EM into two parts with opposite effects: cluster aggregation driving factor (CADF) rewards dominant classes and prompts a peaked output distribution, while gradient mitigation calibrator (GMC) penalizes high-confidence classes based on predicted probabilities. Furthermore, we reveal the limitations of classical EM caused by its coupled formulation: 1) reward collapse impedes the contribution of high-certainty samples in the learning process, and 2) easy-class bias induces misalignment between output distribution and label distribution. To address these issues, we propose Adaptive Decoupled Entropy Minimization (AdaDEM), which normalizes the reward brought from CADF and employs a marginal entropy calibrator (MEC) to replace GMC. AdaDEM outperforms DEM*, an upper-bound variant of classical EM, and achieves superior performance across various imperfectly supervised learning tasks in noisy and dynamic environments.