Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Prompts

TL;DR

Musketeer is a unified vision-language model trained on multiple tasks using Task Explanation Prompts to reduce interference, achieving competitive or superior results compared to single-task models.

Contribution

The paper introduces Musketeer, a multi-task vision-language model that employs Task Explanation Prompts to enable joint training across heterogeneous tasks.

Findings

Achieves comparable or better results than single-task models across multiple tasks.

Uses Task Explanation Prompts to effectively reduce task interference.

Demonstrates the feasibility of fully shared multi-task training in vision-language models.

Abstract

We present a vision-language model whose parameters are jointly trained on all tasks and fully shared among multiple heterogeneous tasks which may interfere with each other, resulting in a single model which we named Musketeer. The integration of knowledge across heterogeneous tasks is enabled by a novel feature called Task Explanation Prompt (TEP). With rich and structured information such as task input/output format, TEP reduces interference among tasks, allowing the model to focus on their shared structure. With a single model, Musketeer achieves results comparable to or better than strong baselines trained on single tasks, almost uniformly across multiple tasks.