Scaling Law Hypothesis for Multimodal Model

TL;DR

This paper introduces a scaling law hypothesis for multimodal models, predicting performance based on modality-specific factors, and investigates how multi-modal data can reduce model size for efficient deployment.

Contribution

It extends established scaling laws to multimodal systems and explores data-driven model size reduction for resource-efficient deployment.

Findings

Performance can be predicted by modality-specific compression and tokenization.

Multi-modal training data can reduce model size.

Potential for deploying smaller models on resource-constrained devices.

Abstract

We propose a scaling law hypothesis for multimodal models processing text, audio, images, and video within a shared token and embedding space. Our framework predicts model performance based on modality-specific compression and tokenization efficiency, extending established scaling laws from text-based decoder models to mixed-modality systems. We explore whether leveraging more training data in multiple modalities can reduce the size of the multimodal model, enabling efficient deployment on resource-constrained devices.