Cascading multi-agent anomaly detection in surveillance systems via vision-language models and embedding-based classification

TL;DR

This paper presents a multi-agent cascade framework that combines reconstruction, object detection, and vision-language reasoning to improve anomaly detection in surveillance, achieving faster response times and high interpretability.

Contribution

It introduces a novel multi-agent cascade architecture that unifies different paradigms for efficient, interpretable anomaly detection in visual surveillance systems.

Findings

Achieves threefold reduction in latency compared to direct vision-language inference.

Maintains high perceptual quality with PSNR of 38.3 dB and SSIM of 0.965.

Demonstrates scalable deployment and high semantic labeling accuracy.

Abstract

Intelligent anomaly detection in dynamic visual environments requires reconciling real-time performance with semantic interpretability. Conventional approaches address only fragments of this challenge. Reconstruction-based models capture low-level deviations without contextual reasoning, object detectors provide speed but limited semantics, and large vision-language systems deliver interpretability at prohibitive computational cost. This work introduces a cascading multi-agent framework that unifies these complementary paradigms into a coherent and interpretable architecture. Early modules perform reconstruction-gated filtering and object-level assessment, while higher-level reasoning agents are selectively invoked to interpret semantically ambiguous events. The system employs adaptive escalation thresholds and a publish-subscribe communication backbone, enabling asynchronous coordination and scalable deployment across heterogeneous hardware. Extensive evaluation on large-scale monitoring data demonstrates that the proposed cascade achieves a threefold reduction in latency compared to direct vision-language inference, while maintaining high perceptual fidelity (PSNR = 38.3 dB, SSIM = 0.965) and consistent semantic labeling. The framework advances beyond conventional detection pipelines by combining early-exit efficiency, adaptive multi-agent reasoning, and explainable anomaly attribution, establishing a reproducible and energy-efficient foundation for scalable intelligent visual monitoring.