Nonlinear Performance Degradation of Vision-Based Teleoperation under Network Latency

TL;DR

This paper investigates how network latency causes nonlinear stability degradation in vision-based teleoperation for autonomous vehicles, using a new testbed to quantify the effects and identify critical latency thresholds.

Contribution

It introduces the LAVT framework for precise latency measurement and demonstrates the nonlinear stability collapse at specific latency thresholds through extensive experiments.

Findings

Stability sharply drops between 150 ms and 225 ms latency

Route completion rates fall from 100% to below 50% in this range

Additional control-channel delay accelerates system failure

Abstract

Teleoperation is increasingly being adopted as a critical fallback for autonomous vehicles. However, the impact of network latency on vision-based, perception-driven control remains insufficiently studied. The present work investigates the nonlinear degradation of closed-loop stability in camera-based lane keeping under varying network delays. To conduct this study, we developed the Latency-Aware Vision Teleoperation testbed (LAVT), a research-oriented ROS 2 framework that enables precise, distributed one-way latency measurement and reproducible delay injection. Using LAVT, we performed 180 closed-loop experiments in simulation across diverse road geometries. Our findings reveal a sharp collapse in stability between 150 ms and 225 ms of one-way perception latency, where route completion rates drop from 100% to below 50% as oscillatory instability and phase-lag effects emerge. We further demonstrate that additional control-channel delay compounds these effects, significantly accelerating system failure even under constant visual latency. By combining this systematic empirical characterization with the LAVT testbed, this work provides quantitative insights into perception-driven instability and establishes a reproducible baseline for future latency-compensation and predictive control strategies. Project page, supplementary video, and code are available at https://bimilab.github.io/paper-LAVT