Unleashing the Potentials of Likelihood Composition for Multi-modal Language Models

TL;DR

This paper introduces a likelihood composition framework for fusing heterogeneous multi-modal models post-hoc, demonstrating its effectiveness in visual-question-answering tasks across multiple datasets and model architectures.

Contribution

The paper proposes a novel likelihood composition framework for model fusion, enabling off-the-shelf combination of diverse models in multi-modal tasks.

Findings

Likelihood composition improves VQA performance over simple ensemble methods.

The framework is effective across 9 VQA datasets and 10 different MLMs.

New composition methods can be easily developed within this framework.

Abstract

Model fusing has always been an important topic, especially in an era where large language models (LLM) and multi-modal language models (MLM) with different architectures, parameter sizes and training pipelines, are being created all the time. In this work, we propose a post-hoc framework, aiming at fusing heterogeneous models off-the-shell, which we call \textit{likelihood composition}, and the basic idea is to compose multiple models' likelihood distribution when doing a multi-choice visual-question-answering task. Here the core concept, \textit{likelihood}, is actually the log-probability of the candidate answer. In \textit{likelihood composition}, we introduce some basic operations: \textit{debias}, \textit{highlight}, \textit{majority-vote} and \textit{ensemble}. By combining (composing) these basic elements, we get the mixed composition methods: \textit{mix-composition}. Through conducting comprehensive experiments on 9 VQA datasets and 10 MLMs, we prove the effectiveness of \textit{mix-composition} compared with simple \textit{ensemble} or \textit{majority-vote} methods. In this framework, people can propose new basic composition methods and combine them to get the new mixed composition methods. We hope our proposed \textit{likelihood composition} can provide a new perspective of fusing heterogeneous models and inspire the exploration under this framework.