LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments

TL;DR

LiteVLM presents a low-latency vision-language inference pipeline optimized for embedded devices, combining patch and token selection with speculative decoding to enable real-time performance in resource-constrained settings.

Contribution

The paper introduces a novel, efficient VLM pipeline that significantly reduces latency on embedded platforms through combined filtering, token reduction, and accelerated decoding techniques.

Findings

Achieves 2.5x latency reduction on NVIDIA DRIVE Thor

Further improves to 3.2x speed-up with FP8 quantization

Maintains task accuracy despite computational optimizations

Abstract

This paper introduces an efficient Vision-Language Model (VLM) pipeline specifically optimized for deployment on embedded devices, such as those used in robotics and autonomous driving. The pipeline significantly reduces the computational overhead by jointly leveraging patch selection to filter irrelevant camera views, a token selection module to reduce input sequence length for the LLM, and speculative decoding to accelerate token generation. Evaluation on the NVIDIA DRIVE Thor platform for automonous driving application, our pipeline achieves $2.5\times$ end-to-end latency reduction without compromising task accuracy. The speed-up further increases to $3.2\times$ when applying FP8 post-training quantization. These results demonstrate our pipeline as a viable solution for enabling real-time VLM deployment in resource-constrained environments.