Exploring Fusion Strategies for Multimodal Vision-Language Systems

TL;DR

This paper investigates different data fusion strategies in multimodal vision-language models, analyzing the tradeoffs between accuracy and latency using various fusion points and vision backbones on a benchmark dataset.

Contribution

It introduces and compares three fusion architectures at different stages in a multimodal model, highlighting the accuracy-latency tradeoff in multimodal fusion strategies.

Findings

Late fusion achieves highest accuracy.

Early fusion provides lowest latency.

Tradeoff between accuracy and inference speed.

Abstract

Modern machine learning models often combine multiple input streams of data to more accurately capture the information that informs their decisions. In multimodal machine learning, choosing the strategy for fusing data together requires careful consideration of the application's accuracy and latency requirements, as fusing the data at earlier or later stages in the model architecture can lead to performance changes in accuracy and latency. To demonstrate this tradeoff, we investigate different fusion strategies using a hybrid BERT and vision network framework that integrates image and text data. We explore two different vision networks: MobileNetV2 and ViT. We propose three models for each vision network, which fuse data at late, intermediate, and early stages in the architecture. We evaluate the proposed models on the CMU MOSI dataset and benchmark their latency on an NVIDIA Jetson Orin AGX. Our experimental results demonstrate that while late fusion yields the highest accuracy, early fusion offers the lowest inference latency. We describe the three proposed model architectures and discuss the accuracy and latency tradeoffs, concluding that data fusion earlier in the model architecture results in faster inference times at the cost of accuracy.