LSRE: Latent Semantic Rule Encoding for Real-Time Semantic Risk Detection in Autonomous Driving

TL;DR

This paper introduces LSRE, a lightweight framework that encodes semantic safety rules into a latent space, enabling real-time hazard detection in autonomous driving without heavy vision-language model inference.

Contribution

The paper presents LSRE, a novel latent semantic rule encoding method that achieves real-time safety monitoring by converting sparse VLM judgments into a compact latent classifier.

Findings

Achieves 10 Hz semantic risk detection in real-time.

Maintains accuracy comparable to large VLM baselines.

Generalizes well to unseen semantic scenarios.

Abstract

Real-world autonomous driving must adhere to complex human social rules that extend beyond legally codified traffic regulations. Many of these semantic constraints, such as yielding to emergency vehicles, complying with traffic officers' gestures, or stopping for school buses, are intuitive for humans yet difficult to encode explicitly. Although large vision-language models (VLMs) can interpret such semantics, their inference cost makes them impractical for real-time deployment. This work proposes LSRE, a Latent Semantic Rule Encoding framework that converts sparsely sampled VLM judgments into decision boundaries within the latent space of a recurrent world model. By encoding language-defined safety semantics into a lightweight latent classifier, LSRE enables real-time semantic risk assessment at 10 Hz without per-frame VLM queries. Experiments on six semantic-failure scenarios in CARLA demonstrate that LSRE attains semantic risk detection accuracy comparable to a large VLM baseline, while providing substantially earlier hazard anticipation and maintaining low computational latency. LSRE further generalizes to rarely seen semantic-similar test cases, indicating that language-guided latent classification offers an effective and deployable mechanism for semantic safety monitoring in autonomous driving.