Motion-Compensated Latent Semantic Canvases for Visual Situational Awareness on Edge

TL;DR

This paper introduces Motion-Compensated Latent Semantic Canvases (MCLSC), a method for efficient visual situational awareness on edge devices by maintaining persistent semantic data with motion compensation, reducing segmentation calls and processing time.

Contribution

The paper presents a novel approach that combines motion compensation with latent semantic canvases to significantly reduce computational load on resource-constrained devices.

Findings

Reduces segmentation calls by over 30 times

Lowers processing time by over 20 times

Maintains coherent semantic overlays

Abstract

We propose Motion-Compensated Latent Semantic Canvases (MCLSC) for visual situational awareness on resource-constrained edge devices. The core idea is to maintain persistent semantic metadata in two latent canvases - a slowly accumulating static layer and a rapidly updating dynamic layer - defined in a baseline coordinate frame stabilized from the video stream. Expensive panoptic segmentation (Mask2Former) runs asynchronously and is motion-gated: inference is triggered only when motion indicates new information, while stabilization/motion compensation preserves a consistent coordinate system for latent semantic memory. On prerecorded 480p clips, our prototype reduces segmentation calls by >30x and lowers mean end-to-end processing time by >20x compared to naive per-frame segmentation, while maintaining coherent static/dynamic semantic overlays.