Vision-Language Models on the Edge for Real-Time Robotic Perception

TL;DR

This paper explores deploying vision-language models on edge infrastructure like 6G Open RAN and MEC to enable real-time robotic perception, balancing latency, resource constraints, and accuracy.

Contribution

It demonstrates the deployment of VLMs on edge infrastructure using a WebRTC pipeline and evaluates models like LLaMA-3.2-11B-Vision-Instruct and Qwen2-VL-2B-Instruct in real-time robotic scenarios.

Findings

Edge deployment reduces latency by 5% compared to cloud.

Compact models achieve sub-second response times.

Trade-off observed between latency reduction and accuracy.

Abstract

Vision-Language Models (VLMs) enable multimodal reasoning for robotic perception and interaction, but their deployment in real-world systems remains constrained by latency, limited onboard resources, and privacy risks of cloud offloading. Edge intelligence within 6G, particularly Open RAN and Multi-access Edge Computing (MEC), offers a pathway to address these challenges by bringing computation closer to the data source. This work investigates the deployment of VLMs on ORAN/MEC infrastructure using the Unitree G1 humanoid robot as an embodied testbed. We design a WebRTC-based pipeline that streams multimodal data to an edge node and evaluate LLaMA-3.2-11B-Vision-Instruct deployed at the edge versus in the cloud under real-time conditions. Our results show that edge deployment preserves near-cloud accuracy while reducing end-to-end latency by 5\%. We further evaluate Qwen2-VL-2B-Instruct, a compact model optimized for resource-constrained environments, which achieves sub-second responsiveness, cutting latency by more than half but at the cost of accuracy.