Task Prototype-Based Knowledge Retrieval for Multi-Task Learning from Partially Annotated Data

TL;DR

This paper introduces a prototype-based knowledge retrieval framework for multi-task learning that improves robustness and performance when only partial task annotations are available, addressing issues of negative transfer.

Contribution

It proposes a novel prototype-based framework with an association knowledge generating loss to enhance task association and feature refinement in partially labeled multi-task learning.

Findings

Outperforms existing methods on multiple benchmarks

Effectively handles partially annotated data

Reduces negative transfer in multi-task learning

Abstract

Multi-task learning (MTL) is critical in real-world applications such as autonomous driving and robotics, enabling simultaneous handling of diverse tasks. However, obtaining fully annotated data for all tasks is impractical due to labeling costs. Existing methods for partially labeled MTL typically rely on predictions from unlabeled tasks, making it difficult to establish reliable task associations and potentially leading to negative transfer and suboptimal performance. To address these issues, we propose a prototype-based knowledge retrieval framework that achieves robust MTL instead of relying on predictions from unlabeled tasks. Our framework consists of two key components: (1) a task prototype embedding task-specific characteristics and quantifying task associations, and (2) a knowledge retrieval transformer that adaptively refines feature representations based on these associations. To achieve this, we introduce an association knowledge generating (AKG) loss to ensure the task prototype consistently captures task-specific characteristics. Extensive experiments demonstrate the effectiveness of our framework, highlighting its potential for robust multi-task learning, even when only a subset of tasks is annotated.