Simplifying Multi-Task Architectures Through Task-Specific Normalization

TL;DR

This paper demonstrates that task-specific normalization layers, especially the proposed TSσBN, can effectively address multi-task learning challenges, reducing complexity while maintaining or improving performance across various benchmarks.

Contribution

The paper introduces TSσBN, a lightweight task-specific normalization method that simplifies multi-task architectures and provides interpretability of task relationships.

Findings

TSσBN matches or exceeds state-of-the-art performance on multiple benchmarks.

Task-specific normalization reduces architectural complexity and overhead.

Learned gates in TSσBN offer insights into task capacity and filter specialization.

Abstract

Multi-task learning (MTL) aims to leverage shared knowledge across tasks to improve generalization and parameter efficiency, yet balancing resources and mitigating interference remain open challenges. Architectural solutions often introduce elaborate task-specific modules or routing schemes, increasing complexity and overhead. In this work, we show that normalization layers alone are sufficient to address many of these challenges. Simply replacing shared normalization with task-specific variants already yields competitive performance, questioning the need for complex designs. Building on this insight, we propose Task-Specific Sigmoid Batch Normalization (TS$\sigma$BN), a lightweight mechanism that enables tasks to softly allocate network capacity while fully sharing feature extractors. TS$\sigma$BN improves stability across CNNs and Transformers, matching or exceeding performance on NYUv2, Cityscapes, CelebA, and PascalContext, while remaining highly parameter-efficient. Moreover, its learned gates provide a natural framework for analyzing MTL dynamics, offering interpretable insights into capacity allocation, filter specialization, and task relationships. Our findings suggest that complex MTL architectures may be unnecessary and that task-specific normalization offers a simple, interpretable, and efficient alternative.