Collaborative Image Understanding

TL;DR

This paper introduces a multitask learning framework that leverages user interaction signals to enhance image classification accuracy, demonstrating significant improvements in e-commerce and social media datasets.

Contribution

It presents a novel approach integrating collaborative signals into image classification, which was not previously explored in this context.

Findings

Improved classification accuracy by up to 9.1%.

Effective use of collaborative signals enhances image understanding.

Framework applicable to e-commerce and social media data.

Abstract

Automatically understanding the contents of an image is a highly relevant problem in practice. In e-commerce and social media settings, for example, a common problem is to automatically categorize user-provided pictures. Nowadays, a standard approach is to fine-tune pre-trained image models with application-specific data. Besides images, organizations however often also collect collaborative signals in the context of their application, in particular how users interacted with the provided online content, e.g., in forms of viewing, rating, or tagging. Such signals are commonly used for item recommendation, typically by deriving latent user and item representations from the data. In this work, we show that such collaborative information can be leveraged to improve the classification process of new images. Specifically, we propose a multitask learning framework, where the auxiliary task is to reconstruct collaborative latent item representations. A series of experiments on datasets from e-commerce and social media demonstrates that considering collaborative signals helps to significantly improve the performance of the main task of image classification by up to 9.1%.