SGW-based Multi-Task Learning in Vision Tasks

TL;DR

This paper introduces a novel SGW-based multi-task learning framework that employs an information bottleneck and neural collapse to improve knowledge sharing and task performance in complex vision tasks.

Contribution

It proposes a new KEM module with ETF space projection to reduce inter-task interference and enhance robustness in multi-task learning.

Findings

Significant performance improvements over existing methods.

Effective reduction of inter-task interference.

Enhanced robustness through ETF space projection.

Abstract

Multi-task-learning(MTL) is a multi-target optimization task. Neural networks try to realize each target using a shared interpretative space within MTL. However, as the scale of datasets expands and the complexity of tasks increases, knowledge sharing becomes increasingly challenging. In this paper, we first re-examine previous cross-attention MTL methods from the perspective of noise. We theoretically analyze this issue and identify it as a flaw in the cross-attention mechanism. To address this issue, we propose an information bottleneck knowledge extraction module (KEM). This module aims to reduce inter-task interference by constraining the flow of information, thereby reducing computational complexity. Furthermore, we have employed neural collapse to stabilize the knowledge-selection process. That is, before input to KEM, we projected the features into ETF space. This mapping makes our method more robust. We implemented and conducted comparative experiments with this method on multiple datasets. The results demonstrate that our approach significantly outperforms existing methods in multi-task learning.