Key Frame Extraction with Attention Based Deep Neural Networks

TL;DR

This paper presents a deep learning approach using an attention-based autoencoder for automatic keyframe extraction from videos, improving summarization efficiency and accuracy.

Contribution

It introduces a novel deep autoencoder with attention for keyframe detection, combining feature extraction, clustering, and selection for better summarization.

Findings

Achieved 77% accuracy on TVSUM dataset.

Outperformed many existing keyframe extraction methods.

Demonstrated effectiveness for video summarization and retrieval.

Abstract

Automatic keyframe detection from videos is an exercise in selecting scenes that can best summarize the content for long videos. Providing a summary of the video is an important task to facilitate quick browsing and content summarization. The resulting photos are used for automated works (e.g. summarizing security footage, detecting different scenes used in music clips) in different industries. In addition, processing high-volume videos in advanced machine learning methods also creates resource costs. Keyframes obtained; It can be used as an input feature to the methods and models to be used. In this study; We propose a deep learning-based approach for keyframe detection using a deep auto-encoder model with an attention layer. The proposed method first extracts the features from the video frames using the encoder part of the autoencoder and applies segmentation using the k-means clustering algorithm to group these features and similar frames together. Then, keyframes are selected from each cluster by selecting the frames closest to the center of the clusters. The method was evaluated on the TVSUM video dataset and achieved a classification accuracy of 0.77, indicating a higher success rate than many existing methods. The proposed method offers a promising solution for key frame extraction in video analysis and can be applied to various applications such as video summarization and video retrieval.