Benchmarking Conventional Vision Models on Neuromorphic Fall Detection and Action Recognition Dataset

TL;DR

This paper benchmarks fine-tuned conventional vision models on neuromorphic datasets for fall detection and action recognition, demonstrating that the MViT architecture achieves the highest accuracy and F1 score among tested models.

Contribution

It introduces a benchmarking framework for conventional vision models on neuromorphic data and identifies the MViT architecture as the most effective for this application.

Findings

DVS-MViT achieves 95.8% accuracy and F1 score.

DVS-C2D achieves 91.6% accuracy and F1 score.

DVS-CSN and DVS-X3D perform less effectively.

Abstract

Neuromorphic vision-based sensors are gaining popularity in recent years with their ability to capture Spatio-temporal events with low power sensing. These sensors record events or spikes over traditional cameras which helps in preserving the privacy of the subject being recorded. These events are captured as per-pixel brightness changes and the output data stream is encoded with time, location, and pixel intensity change information. This paper proposes and benchmarks the performance of fine-tuned conventional vision models on neuromorphic human action recognition and fall detection datasets. The Spatio-temporal event streams from the Dynamic Vision Sensing cameras are encoded into a standard sequence image frames. These video frames are used for benchmarking conventional deep learning-based architectures. In this proposed approach, we fine-tuned the state-of-the-art vision models for this Dynamic Vision Sensing (DVS) application and named these models as DVS-R2+1D, DVS-CSN, DVS-C2D, DVS-SlowFast, DVS-X3D, and DVS-MViT. Upon comparing the performance of these models, we see the current state-of-the-art MViT based architecture DVS-MViT outperforms all the other models with an accuracy of 0.958 and an F-1 score of 0.958. The second best is the DVS-C2D with an accuracy of 0.916 and an F-1 score of 0.916. Third and Fourth are DVS-R2+1D and DVS-SlowFast with an accuracy of 0.875 and 0.833 and F-1 score of 0.875 and 0.861 respectively. DVS-CSN and DVS-X3D were the least performing models with an accuracy of 0.708 and 0.625 and an F1 score of 0.722 and 0.625 respectively.