Context-aware Sparse Spatiotemporal Learning for Event-based Vision

TL;DR

This paper introduces CSSL, a framework that dynamically adjusts neuron activations in event-based vision models, achieving high sparsity and competitive performance, thus enhancing efficiency for neuromorphic applications.

Contribution

CSSL employs context-aware thresholding to automatically regulate neuron activations, eliminating the need for manual sparsity tuning in event-based vision tasks.

Findings

CSSL achieves comparable or better accuracy than state-of-the-art methods.

CSSL maintains extremely high neuronal sparsity.

CSSL enhances efficiency for neuromorphic event-based vision applications.

Abstract

Event-based camera has emerged as a promising paradigm for robot perception, offering advantages with high temporal resolution, high dynamic range, and robustness to motion blur. However, existing deep learning-based event processing methods often fail to fully leverage the sparse nature of event data, complicating their integration into resource-constrained edge applications. While neuromorphic computing provides an energy-efficient alternative, spiking neural networks struggle to match of performance of state-of-the-art models in complex event-based vision tasks, like object detection and optical flow. Moreover, achieving high activation sparsity in neural networks is still difficult and often demands careful manual tuning of sparsity-inducing loss terms. Here, we propose Context-aware Sparse Spatiotemporal Learning (CSSL), a novel framework that introduces context-aware thresholding to dynamically regulate neuron activations based on the input distribution, naturally reducing activation density without explicit sparsity constraints. Applied to event-based object detection and optical flow estimation, CSSL achieves comparable or superior performance to state-of-the-art methods while maintaining extremely high neuronal sparsity. Our experimental results highlight CSSL's crucial role in enabling efficient event-based vision for neuromorphic processing.