Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment

TL;DR

Task Preference Optimization (TPO) enhances multimodal large language models by learning task-specific preferences, significantly improving performance and zero-shot capabilities across visual tasks through multi-task co-training.

Contribution

The paper introduces TPO, a scalable method with learnable task tokens that improves multimodal models' performance by leveraging rich visual labels and multi-task co-training.

Findings

14.6% overall performance improvement

Robust zero-shot capabilities across tasks

Synergistic benefits from multi-task co-training

Abstract

Current multimodal large language models (MLLMs) struggle with fine-grained or precise understanding of visuals although they give comprehensive perception and reasoning in a spectrum of vision applications. Recent studies either develop tool-using or unify specific visual tasks into the autoregressive framework, often at the expense of overall multimodal performance. To address this issue and enhance MLLMs with visual tasks in a scalable fashion, we propose Task Preference Optimization (TPO), a novel method that utilizes differentiable task preferences derived from typical fine-grained visual tasks. TPO introduces learnable task tokens that establish connections between multiple task-specific heads and the MLLM. By leveraging rich visual labels during training, TPO significantly enhances the MLLM's multimodal capabilities and task-specific performance. Through multi-task co-training within TPO, we observe synergistic benefits that elevate individual task performance beyond what is achievable through single-task training methodologies. Our instantiation of this approach with VideoChat and LLaVA demonstrates an overall 14.6% improvement in multimodal performance compared to baseline models. Additionally, MLLM-TPO demonstrates robust zero-shot capabilities across various tasks, performing comparably to state-of-the-art supervised models. The code will be released at https://github.com/OpenGVLab/TPO