# Preferences Prediction using a Gallery of Mobile Device based on Scene Recognition and Object Detection

## TL;DR

This paper presents a novel method for predicting user preferences on mobile devices by analyzing photos and videos using scene recognition, object detection, and facial analysis, optimized for Android platforms.

## Contribution

It introduces a new engine combining scene recognition, object detection, and facial analysis for user preference prediction on mobile devices, with an efficient processing pipeline.

## Key findings

- Effective preference prediction without significant accuracy loss.
- Efficient processing on mobile devices using offline and remote analysis.
- Open-source Android implementation available.

## Abstract

In this paper user modeling task is examined by processing a gallery of photos and videos on a mobile device. We propose novel engine for user preference prediction based on scene recognition, object detection and facial analysis. At first, all faces in a gallery are clustered and all private photos and videos with faces from large clusters are processed on the embedded system in offline mode. Other photos may be sent to the remote server to be analyzed by very deep models. The visual features of each photo are obtained from scene recognition and object detection models. These features are aggregated into a single user descriptor in the neural attention block. The proposed pipeline is implemented for the Android mobile platform. Experimental results with a subset of Photo Event Collection, Web Image Dataset for Event Recognition and Amazon Fashion datasets demonstrate the possibility to process images very efficiently without significant accuracy degradation. The source code of Android mobile application is publicly available at https://github.com/HSE-asavchenko/mobile-visual-preferences.