Empower Vision Applications with LoRA LMM

TL;DR

This paper introduces VaLoRA, an end-to-end system that enhances vision applications with LoRA LMMs, achieving high accuracy and low latency through innovative adapter generation, batching, and orchestration techniques.

Contribution

The paper presents a novel system, VaLoRA, that significantly improves the efficiency and accuracy of LoRA-based vision tasks in large multimodal models.

Findings

Improves accuracy by 24-62% over original LMMs.

Reduces latency by 20-89% compared to existing LoRA serving systems.

Demonstrates effectiveness on five vision tasks across three LMMs.

Abstract

Large Multimodal Models (LMMs) have shown significant progress in various complex vision tasks with the solid linguistic and reasoning capacity inherited from large language models (LMMs). Low-rank adaptation (LoRA) offers a promising method to integrate external knowledge into LMMs, compensating for their limitations on domain-specific tasks. However, the existing LoRA model serving is excessively computationally expensive and causes extremely high latency. In this paper, we present an end-to-end solution that empowers diverse vision tasks and enriches vision applications with LoRA LMMs. Our system, VaLoRA, enables accurate and efficient vision tasks by 1) an accuracy-aware LoRA adapter generation approach that generates LoRA adapters rich in domain-specific knowledge to meet application-specific accuracy requirements, 2) an adaptive-tiling LoRA adapters batching operator that efficiently computes concurrent heterogeneous LoRA adapters, and 3) a flexible LoRA adapter orchestration mechanism that manages application requests and LoRA adapters to achieve the lowest average response latency. We prototype VaLoRA on five popular vision tasks on three LMMs. Experiment results reveal that VaLoRA improves 24-62% of the accuracy compared to the original LMMs and reduces 20-89% of the latency compared to the state-of-the-art LoRA model serving systems.