Beyond Softmax: Dual-Branch Sigmoid Architecture for Accurate Class Activation Maps

TL;DR

This paper introduces a dual-branch sigmoid architecture that improves class activation maps by decoupling localization from classification, enhancing explanation fidelity without sacrificing recognition accuracy.

Contribution

A simple, architecture-agnostic dual-branch sigmoid head that enhances CAM explanations by preserving feature sign and magnitude, compatible with pretrained models.

Findings

Improved explanation fidelity on fine-grained and WSOL benchmarks.

Achieved consistent Top-1 Localization gains.

No drop in classification accuracy.

Abstract

Class Activation Mapping (CAM) and its extensions have become indispensable tools for visualizing the evidence behind deep network predictions. However, by relying on a final softmax classifier, these methods suffer from two fundamental distortions: additive logit shifts that arbitrarily bias importance scores, and sign collapse that conflates excitatory and inhibitory features. We propose a simple, architecture-agnostic dual-branch sigmoid head that decouples localization from classification. Given any pretrained model, we clone its classification head into a parallel branch ending in per-class sigmoid outputs, freeze the original softmax head, and fine-tune only the sigmoid branch with class-balanced binary supervision. At inference, softmax retains recognition accuracy, while class evidence maps are generated from the sigmoid branch -- preserving both magnitude and sign of feature contributions. Our method integrates seamlessly with most CAM variants and incurs negligible overhead. Extensive evaluations on fine-grained tasks (CUB-200-2011, Stanford Cars) and WSOL benchmarks (ImageNet-1K, OpenImages30K) show improved explanation fidelity and consistent Top-1 Localization gains -- without any drop in classification accuracy. Code is available at https://github.com/finallyupper/beyond-softmax.