StableMTL: Repurposing Latent Diffusion Models for Multi-Task Learning from Partially Annotated Synthetic Datasets

TL;DR

StableMTL leverages diffusion models for multi-task learning on partially labeled synthetic datasets, enabling zero-shot multi-task training with improved cross-task sharing and performance.

Contribution

It introduces StableMTL, a novel framework that repurposes latent diffusion models for multi-task learning from synthetic data with partial labels, using a unified loss and task-attention mechanism.

Findings

Outperforms baselines on 7 tasks across 8 benchmarks.

Enables zero-shot multi-task learning with synthetic datasets.

Uses a unified latent loss and task-attention for efficient cross-task sharing.

Abstract

Multi-task learning for dense prediction is limited by the need for extensive annotation for every task, though recent works have explored training with partial task labels. Leveraging the generalization power of diffusion models, we extend the partial learning setup to a zero-shot setting, training a multi-task model on multiple synthetic datasets, each labeled for only a subset of tasks. Our method, StableMTL, repurposes image generators for latent regression. Adapting a denoising framework with task encoding, per-task conditioning and a tailored training scheme. Instead of per-task losses requiring careful balancing, a unified latent loss is adopted, enabling seamless scaling to more tasks. To encourage inter-task synergy, we introduce a multi-stream model with a task-attention mechanism that converts N-to-N task interactions into efficient 1-to-N attention, promoting effective cross-task sharing. StableMTL outperforms baselines on 7 tasks across 8 benchmarks.