Real-Time Inference for Distributed Multimodal Systems under Communication Delay Uncertainty

TL;DR

This paper introduces a neuro-inspired, delay-aware inference framework for distributed multimodal systems that adaptively manages communication delays to improve real-time performance and robustness over existing methods.

Contribution

It proposes a novel non-blocking inference paradigm using adaptive temporal windows, relaxing the reference-modality requirement and enhancing robustness against stochastic delays.

Findings

Outperforms state-of-the-art methods in audio-visual event localization.

Demonstrates improved robustness to network delay variability.

Offers finer control over accuracy and latency tradeoffs.

Abstract

Connected cyber-physical systems perform inference based on real-time inputs from multiple data streams. Uncertain communication delays across data streams challenge the temporal flow of the inference process. State-of-the-art (SotA) non-blocking inference methods rely on a reference-modality paradigm, requiring one modality input to be fully received before processing, while depending on costly offline profiling. We propose a novel, neuro-inspired non-blocking inference paradigm that primarily employs adaptive temporal windows of integration (TWIs) to dynamically adjust to stochastic delay patterns across heterogeneous streams while relaxing the reference-modality requirement. Our communication-delay-aware framework achieves robust real-time inference with finer-grained control over the accuracy-latency tradeoff. Experiments on the audio-visual event localization (AVEL) task demonstrate superior adaptability to network dynamics compared to SotA approaches.