Task Relevance Is Not Local Replaceability: A Two-Axis View of Channel Information

TL;DR

This paper introduces a two-axis framework to distinguish between task relevance and local replaceability of channels in vision networks, revealing that these aspects are weakly aligned and impact pruning decisions.

Contribution

It proposes a novel two-axis view separating task relevance and local replaceability, supported by analysis and experiments across multiple architectures and datasets.

Findings

Local and target axes are weakly aligned and separate during training.

Peer support refines channel removability beyond input capture and task relevance.

Local-axis metrics outperform target-axis metrics in predicting channel removability.

Abstract

Channel importance in vision networks is usually summarized by a single score. That summary hides two different questions: how much a channel is related to the task, and whether its function can be supplied by same-layer peers when the channel is removed. We call the second property local replaceability. We introduce a two-axis view that separates these questions. The local axis measures input capture and peer overlap, while the target axis measures task information and target-excess information. Across ResNet-18, VGG-16, and MobileNetV2 trained on CIFAR-100, the two axes are weakly aligned, induce different channel groupings, and separate rapidly during training despite being strongly coupled at random initialization. A Gaussian linear analysis accounts for how this separation can arise through residualized gradient directions, and lesion plus peer-replacement experiments show that peer support refines removability beyond input capture and task relevance alone. Under the fixed FLOPs-matched pruning protocol, local-axis metrics are more reliable predictors of removability than target-axis metrics across the three CIFAR-100 backbones, with the same direction preserved in stress tests on CIFAR-10, Tiny-ImageNet, ImageNet-100, and a ConvNeXt-T/ImageNet-100 pilot. These findings identify an axis-level distinction rather than a universal ranking of pruning scores: local replaceability is a more reliable guide to removability than target relevance, while norm-based baselines remain competitive in architectures such as VGG-16. Relevance-based scores ask what a channel says about the task; pruning asks whether the network still needs that channel when its peers remain available.