Do Better ImageNet Models Transfer Better... for Image Recommendation?

TL;DR

This paper investigates whether CNNs that perform well on ImageNet also excel in transfer learning for image recommendation, examining the effects of fine-tuning and different training strategies.

Contribution

It provides a comparative analysis of various pre-trained CNN models for image recommendation, highlighting that better ImageNet performance does not always translate to better transfer learning results.

Findings

Better ImageNet performance does not guarantee improved transfer learning for recommendation.

Fine-tuning can be beneficial even with small datasets, but not all methods are effective.

Model selection should consider transfer learning performance, not just ImageNet accuracy.

Abstract

Visual embeddings from Convolutional Neural Networks (CNN) trained on the ImageNet dataset for the ILSVRC challenge have shown consistently good performance for transfer learning and are widely used in several tasks, including image recommendation. However, some important questions have not yet been answered in order to use these embeddings for a larger scope of recommendation domains: a) Do CNNs that perform better in ImageNet are also better for transfer learning in content-based image recommendation?, b) Does fine-tuning help to improve performance? and c) Which is the best way to perform the fine-tuning? In this paper we compare several CNN models pre-trained with ImageNet to evaluate their transfer learning performance to an artwork image recommendation task. Our results indicate that models with better performance in the ImageNet challenge do not always imply better transfer learning for recommendation tasks (e.g. NASNet vs. ResNet). Our results also show that fine-tuning can be helpful even with a small dataset, but not every fine-tuning works. Our results can inform other researchers and practitioners on how to train their CNNs for better transfer learning towards image recommendation systems.