PETA: Photo Albums Event Recognition using Transformers Attention

TL;DR

This paper introduces PETA, a transformer-based approach for recognizing events in personal photo albums, effectively handling disordered images and achieving state-of-the-art results on multiple benchmarks.

Contribution

It combines CNNs and transformers for global reasoning in photo albums, addressing limitations of previous temporal models and exploring image importance correlation.

Findings

Achieves above 90% mAP on all tested datasets.

Outperforms previous methods on three benchmarks.

Demonstrates correlation between learned attention and human-annotated importance.

Abstract

In recent years the amounts of personal photos captured increased significantly, giving rise to new challenges in multi-image understanding and high-level image understanding. Event recognition in personal photo albums presents one challenging scenario where life events are recognized from a disordered collection of images, including both relevant and irrelevant images. Event recognition in images also presents the challenge of high-level image understanding, as opposed to low-level image object classification. In absence of methods to analyze multiple inputs, previous methods adopted temporal mechanisms, including various forms of recurrent neural networks. However, their effective temporal window is local. In addition, they are not a natural choice given the disordered characteristic of photo albums. We address this gap with a tailor-made solution, combining the power of CNNs for image representation and transformers for album representation to perform global reasoning on image collection, offering a practical and efficient solution for photo albums event recognition. Our solution reaches state-of-the-art results on 3 prominent benchmarks, achieving above 90\% mAP on all datasets. We further explore the related image-importance task in event recognition, demonstrating how the learned attentions correlate with the human-annotated importance for this subjective task, thus opening the door for new applications.