EventRPG: Event Data Augmentation with Relevance Propagation Guidance
Mingyuan Sun, Donghao Zhang, Zongyuan Ge, Jiaxu Wang, Jia Li, Zheng, Fang, Renjing Xu
TL;DR
EventRPG introduces relevance propagation-based data augmentation for event-based SNNs, significantly improving object and action recognition accuracy by generating saliency maps for better training.
Contribution
The paper presents the first relevance propagation methods for SNNs to produce saliency maps, enabling effective data augmentation for event-based neural networks.
Findings
Achieved state-of-the-art accuracy on N-Caltech101 and CIFAR10-DVS datasets.
Improved recognition performance on event-based datasets with relevance-guided augmentation.
Demonstrated the effectiveness of saliency-based augmentation in SNNs.
Abstract
Event camera, a novel bio-inspired vision sensor, has drawn a lot of attention for its low latency, low power consumption, and high dynamic range. Currently, overfitting remains a critical problem in event-based classification tasks for Spiking Neural Network (SNN) due to its relatively weak spatial representation capability. Data augmentation is a simple but efficient method to alleviate overfitting and improve the generalization ability of neural networks, and saliency-based augmentation methods are proven to be effective in the image processing field. However, there is no approach available for extracting saliency maps from SNNs. Therefore, for the first time, we present Spiking Layer-Time-wise Relevance Propagation rule (SLTRP) and Spiking Layer-wise Relevance Propagation rule (SLRP) in order for SNN to generate stable and accurate CAMs and saliency maps. Based on this, we propose…
Peer Reviews
Decision·ICLR 2024 poster
1) The authors mention that the code will be released; to me this is important as it enables other researchers to easily build on top of this paper. 2) I particularly enjoyed the theoretical introduction into the SNNs, this makes the rest of the paper much easier to read.
1) If anything, I would like to notice here that the improvements compared to competing methods are generally small. It would be interesting to see an apples-to-apples comparison in therms of compute overhead (which is mentioned, but I do not see numberical benchmark results); or, another compelling reason to use the presented methods vs e.g. second best. 2) Also see 'questions': it would be good to show that saliency on motion-related datasets is more than just event density. This could be don
This paper proposes Spiking Layer-wise Relevance Propagation(SLRP) rule and Spiking Layer-Time-wise Relevance Propagation(SLTRP) rule, the layer-wise relevance propagation method of SNNs for the first time, which can obtain the feature contribution at each pixel. RGBDrop and RGBMix are established to achieve data augmentation based on the generated CAMs. The results of experiments prove the usefulness of SLRP and SLTRP both for accuracy and efficiency. The EventRPG shows good performance in obje
1. The description of RPGMix is quite simple and unclear. Section 4.3 fails to clearly illustrate the algorithm flow of RPGMix. Many operations in Fig 3(b) are not carefully analyzed, such as sample position in the Nonoverlapping Region, which makes it hard to understand. I think Fig3 is the main figure of this paper and Fig 3(b) accounts for the most part of Fig3, hence the authors need to spend more space to describe it. Otherwise, the readers may feel confused about RPGMix. 2. Equation 15 lac
- The derivation of the CAM and saliency map of SNNs itself is a clear contribution. The results in Table 1 prove the correctness of this - EventRPG is able to consistently improve performance across event datasets - The time cost of EventRPG is comparable to similar augmentations in conventional vision
My main concern is regarding the experiments: - The paper motivates the need for event data augmentation with the statement that "the lack of huge event-based datasets similar to Imagenet prevents us from improving the model performance on relatively small datasets using a Pretrain-Finetune paradigm". However, there is an event camera version of ImageNet available [1], and its paper shows that pre-training on N-ImageNet can greatly improve the accuracy on other datasets via transfer learning. Th
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsData Quality and Management · Scientific Computing and Data Management · Advanced Database Systems and Queries
MethodsSpiking Neural Networks