Shift-Memory Network for Temporal Scene Segmentation

TL;DR

This paper introduces a Shift-Memory Network (SMN) that enhances real-time temporal scene segmentation by reusing network computations, achieving faster inference and reduced memory without sacrificing accuracy.

Contribution

The paper proposes a novel SMN architecture that reduces redundant computation in temporal segmentation, enabling real-time performance on edge devices.

Findings

SMN achieves comparable accuracy to shift-mode segmentation.

SMN significantly improves inference speed.

SMN requires less memory for deployment.

Abstract

Semantic segmentation has achieved great accuracy in understanding spatial layout. For real-time tasks based on dynamic scenes, we extend semantic segmentation in temporal domain to enhance the spatial accuracy with motion. We utilize a shift-mode network over streaming input to ensure zero-latency output. For the data overlap under shifting network, this paper identifies repeated computation in fixed periods across network layers. To avoid this redundancy, we derive a Shift-Memory Network (SMN) from encoding-decoding baseline to reuse the network values without accuracy loss. Trained in patch-mode, the SMN extracts the network parameters for SMN to perform inference promptly in compact memory. We segment dynamic scenes from 1D scanning input and 2D video. The experiments of SMN achieve equivalent accuracy as shift-mode but in faster inference speeds and much smaller memory. This will facilitate semantic segmentation in real-time application on edge devices.