Recognizing and Curating Photo Albums via Event-Specific Image Importance

TL;DR

This paper presents a hybrid deep learning system that simultaneously recognizes event types in photo albums and predicts image importance, improving organization and curation of personal photos.

Contribution

It introduces a combined approach using siamese networks, CNNs, and LSTMs for joint event recognition and image importance prediction, along with an iterative updating method.

Findings

Image importance prediction and event recognition mutually improve each other's accuracy.

The proposed system outperforms baseline methods in album organization tasks.

The dataset refined from CUFED enables effective training and evaluation.

Abstract

Automatic organization of personal photos is a problem with many real world ap- plications, and can be divided into two main tasks: recognizing the event type of the photo collection, and selecting interesting images from the collection. In this paper, we attempt to simultaneously solve both tasks: album-wise event recognition and image- wise importance prediction. We collected an album dataset with both event type labels and image importance labels, refined from an existing CUFED dataset. We propose a hybrid system consisting of three parts: A siamese network-based event-specific image importance prediction, a Convolutional Neural Network (CNN) that recognizes the event type, and a Long Short-Term Memory (LSTM)-based sequence level event recognizer. We propose an iterative updating procedure for event type and image importance score prediction. We experimentally verified that image importance score prediction and event type recognition can each help the performance of the other.