TBGC: Task-level Backbone-Oriented Gradient Clip for Multi-Task Foundation Model Learning

TL;DR

This paper introduces a task-level gradient clipping method for multi-task learning that independently clips and rescales task gradients to reduce bias, along with a multi-branch data augmentation strategy, achieving top results in CVPR2023 Foundation Model Challenge.

Contribution

The paper proposes a novel task-level gradient clipping paradigm and a multi-branch data augmentation strategy to improve multi-task foundation model training.

Findings

Relieves gradient bias in multi-task learning.

Achieves 1st place in Leaderboard A and 2nd in Leaderboard B of CVPR2023 Challenge.

Effective in handling conflicting task gradients.

Abstract

The AllInOne training paradigm squeezes a wide range of tasks into a unified model in a multi-task learning manner. However, optimization in multi-task learning is more challenge than single-task learning, as the gradient norm from different tasks may vary greatly, making the backbone overly biased towards one specific task. To address this issue, we propose the task-level backbone-oriented gradient clip paradigm, compared with the vanilla gradient clip method, it has two points of emphasis:1) gradient clip is performed independently for each task. 2) backbone gradients generated from each task are rescaled to the same norm scale. Based on the experimental results, we argue that the task-level backbone-oriented gradient clip paradigm can relieve the gradient bias problem to some extent. We also propose a novel multi-branch data augmentation strategy where conflict augmentations are placed in different branches. Our approach has been shown to be effective and finally achieve 1st place in the Leaderboard A and 2nd place in the Leaderboard B of the CVPR2023 Foundation Model Challenge. It's worth noting that instead of evaluating all three tasks(detection, segmentation and fine-grained classification) in Leaderboard A, the segmentation task is not evaluated in Leaderboard B, in which our team has a huge advantage.