Hierarchical Attention Network for Visually-aware Food Recommendation

TL;DR

This paper introduces HAFR, a neural network model that improves food recommendation accuracy by integrating user history, recipe ingredients, and images, outperforming existing methods on a large dataset.

Contribution

The paper proposes a novel Hierarchical Attention Network for food recommendation that captures collaborative filtering, ingredient preferences, and visual cues, advancing personalized food recommendation systems.

Findings

HAFR outperforms Factorization Machine and Visual BPR by 12% on average.

The model effectively captures multi-faceted user preferences.

Large-scale dataset demonstrates the model's scalability and effectiveness.

Abstract

Food recommender systems play an important role in assisting users to identify the desired food to eat. Deciding what food to eat is a complex and multi-faceted process, which is influenced by many factors such as the ingredients, appearance of the recipe, the user's personal preference on food, and various contexts like what had been eaten in the past meals. In this work, we formulate the food recommendation problem as predicting user preference on recipes based on three key factors that determine a user's choice on food, namely, 1) the user's (and other users') history; 2) the ingredients of a recipe; and 3) the descriptive image of a recipe. To address this challenging problem, we develop a dedicated neural network based solution Hierarchical Attention based Food Recommendation (HAFR) which is capable of: 1) capturing the collaborative filtering effect like what similar users tend to eat; 2) inferring a user's preference at the ingredient level; and 3) learning user preference from the recipe's visual images. To evaluate our proposed method, we construct a large-scale dataset consisting of millions of ratings from AllRecipes.com. Extensive experiments show that our method outperforms several competing recommender solutions like Factorization Machine and Visual Bayesian Personalized Ranking with an average improvement of 12%, offering promising results in predicting user preference for food. Codes and dataset will be released upon acceptance.