Spiking Variational Graph Representation Inference for Video Summarization

TL;DR

This paper introduces SpiVG, a novel spiking neural network-based framework for video summarization that captures global temporal dependencies, reduces noise impact, and improves summarization accuracy across multiple datasets.

Contribution

The paper presents a new Spiking Variational Graph Network with a keyframe extractor, dynamic graph reasoning, and variational inference modules, advancing video summarization techniques.

Findings

Outperforms existing methods on SumMe, TVSum, VideoXum, and QFVS datasets.

Effectively captures global temporal dependencies and semantic coherence.

Reduces noise influence during multi-channel feature fusion.

Abstract

With the rise of short video content, efficient video summarization techniques for extracting key information have become crucial. However, existing methods struggle to capture the global temporal dependencies and maintain the semantic coherence of video content. Additionally, these methods are also influenced by noise during multi-channel feature fusion. We propose a Spiking Variational Graph (SpiVG) Network, which enhances information density and reduces computational complexity. First, we design a keyframe extractor based on Spiking Neural Networks (SNN), leveraging the event-driven computation mechanism of SNNs to learn keyframe features autonomously. To enable fine-grained and adaptable reasoning across video frames, we introduce a Dynamic Aggregation Graph Reasoner, which decouples contextual object consistency from semantic perspective coherence. We present a Variational Inference Reconstruction Module to address uncertainty and noise arising during multi-channel feature fusion. In this module, we employ Evidence Lower Bound Optimization (ELBO) to capture the latent structure of multi-channel feature distributions, using posterior distribution regularization to reduce overfitting. Experimental results show that SpiVG surpasses existing methods across multiple datasets such as SumMe, TVSum, VideoXum, and QFVS. Our codes and pre-trained models are available at https://github.com/liwrui/SpiVG.