FairMT: Fairness for Heterogeneous Multi-Task Learning

TL;DR

FairMT is a novel framework that ensures fairness across heterogeneous multi-task learning models with different task types and incomplete labels, improving fairness without sacrificing utility.

Contribution

It introduces a unified fairness-aware MTL framework with an asymmetric constraint mechanism and joint utility-fairness optimization, addressing limitations of existing methods.

Findings

Achieves substantial fairness improvements across diverse benchmarks.

Maintains superior task utility compared to existing fairness methods.

Handles heterogeneous tasks with incomplete supervision effectively.

Abstract

Fairness in machine learning has been extensively studied in single-task settings, while fair multi-task learning (MTL), especially with heterogeneous tasks (classification, detection, regression) and partially missing labels, remains largely unexplored. Existing fairness methods are predominantly classification-oriented and fail to extend to continuous outputs, making a unified fairness objective difficult to formulate. Further, existing MTL optimization is structurally misaligned with fairness: constraining only the shared representation, allowing task heads to absorb bias and leading to uncontrolled task-specific disparities. Finally, most work treats fairness as a zero-sum trade-off with utility, enforcing symmetric constraints that achieve parity by degrading well-served groups. We introduce FairMT, a unified fairness-aware MTL framework that accommodates all three task types under incomplete supervision. At its core is an Asymmetric Heterogeneous Fairness Constraint Aggregation mechanism, which consolidates task-dependent asymmetric violations into a unified fairness constraint. Utility and fairness are jointly optimized via a primal--dual formulation, while a head-aware multi-objective optimization proxy provides a tractable descent geometry that explicitly accounts for head-induced anisotropy. Across three homogeneous and heterogeneous MTL benchmarks encompassing diverse modalities and supervision regimes, FairMT consistently achieves substantial fairness gains while maintaining superior task utility. Code will be released upon paper acceptance.