Prompt Guided Transformer for Multi-Task Dense Prediction

TL;DR

The paper introduces a lightweight Prompt Guided Transformer that uses task-specific prompts within a shared architecture to efficiently perform multi-task dense prediction, achieving state-of-the-art results with fewer parameters.

Contribution

It proposes a novel prompt-conditioned Transformer block integrated into a task-conditional model, balancing performance and parameter efficiency in multi-task dense prediction.

Findings

Achieves state-of-the-art results on PASCAL-Context and NYUD-v2 benchmarks.

Uses only 2.7% of total parameters in the lightweight decoder.

Maintains high performance with significantly fewer parameters.

Abstract

Task-conditional architecture offers advantage in parameter efficiency but falls short in performance compared to state-of-the-art multi-decoder methods. How to trade off performance and model parameters is an important and difficult problem. In this paper, we introduce a simple and lightweight task-conditional model called Prompt Guided Transformer (PGT) to optimize this challenge. Our approach designs a Prompt-conditioned Transformer block, which incorporates task-specific prompts in the self-attention mechanism to achieve global dependency modeling and parameter-efficient feature adaptation across multiple tasks. This block is integrated into both the shared encoder and decoder, enhancing the capture of intra- and inter-task features. Moreover, we design a lightweight decoder to further reduce parameter usage, which accounts for only 2.7% of the total model parameters. Extensive experiments on two multi-task dense prediction benchmarks, PASCAL-Context and NYUD-v2, demonstrate that our approach achieves state-of-the-art results among task-conditional methods while using fewer parameters, and maintains a significant balance between performance and parameter size.