TCM-Serve: Modality-aware Scheduling for Multimodal Large Language Model Inference

TL;DR

This paper introduces TCM-Serve, a modality-aware scheduler for multimodal large language models that significantly reduces latency and improves responsiveness by classifying and prioritizing different request types.

Contribution

The paper presents TCM-Serve, a novel scheduling system that dynamically classifies and prioritizes multimodal requests to optimize inference performance.

Findings

Reduces time-to-first-token by 54% overall.

Achieves 78.5% reduction in latency for critical requests.

Improves responsiveness of multimodal LLMs significantly.

Abstract

Multimodal Large Language Models (MLLMs) power platforms like ChatGPT, Gemini, and Copilot, enabling richer interactions with text, images, and videos. These heterogeneous workloads introduce additional inference stages, such as vision preprocessing and encoding, that inflate latency and memory demand. Existing LLM serving systems, optimized for text-only workloads, fail under multimodality: large requests (e.g., videos) monopolize resources, causing severe head-of-line blocking and performance degradation. Our key insight is that multimodal requests differ by orders of magnitude in resource demands, which we capture through a simple abstraction: videos behave like trucks, images like cars, and text like motorcycles. We design TCM-Serve, a modality-aware scheduler that lets motorcycles flow quickly through cars and trucks, ensuring interactive responsiveness while avoiding starvation. TCM-Serve classifies requests, prioritizes them dynamically, and applies aging to avoid starvation. Evaluation across state-of-the-art MLLMs shows that TCM-Serve reduces, on average, time-to-first-token (TTFT) by 54% overall, and by 78.5% for latency-critical requests, compared to current systems. TCM-Serve delivers LLM-like responsiveness for MLLMs, with modality-aware scheduling and by making the most efficient use of the available resources.