Smartphone picture organization: A hierarchical approach

TL;DR

This paper introduces a hierarchical method for automatically organizing large, unstructured smartphone photo collections into topics and categories using probabilistic models and neural networks, improving user satisfaction.

Contribution

It presents a novel hierarchical approach combining probabilistic Latent Semantic Analysis and CNNs for organizing smartphone photos into meaningful topics and categories.

Findings

Improved organization accuracy over existing methods

Successful estimation of latent photo topics

Enhanced user satisfaction with the organization

Abstract

We live in a society where the large majority of the population has a camera-equipped smartphone. In addition, hard drives and cloud storage are getting cheaper and cheaper, leading to a tremendous growth in stored personal photos. Unlike photo collections captured by a digital camera, which typically are pre-processed by the user who organizes them into event-related folders, smartphone pictures are automatically stored in the cloud. As a consequence, photo collections captured by a smartphone are highly unstructured and because smartphones are ubiquitous, they present a larger variability compared to pictures captured by a digital camera. To solve the need of organizing large smartphone photo collections automatically, we propose here a new methodology for hierarchical photo organization into topics and topic-related categories. Our approach successfully estimates latent topics in the pictures by applying probabilistic Latent Semantic Analysis, and automatically assigns a name to each topic by relying on a lexical database. Topic-related categories are then estimated by using a set of topic-specific Convolutional Neuronal Networks. To validate our approach, we ensemble and make public a large dataset of more than 8,000 smartphone pictures from 10 persons. Experimental results demonstrate better user satisfaction with respect to state of the art solutions in terms of organization.