Recipe2Vec: Multi-modal Recipe Representation Learning with Graph Neural Networks

TL;DR

Recipe2Vec introduces a multi-modal recipe embedding model using graph neural networks, integrating images, text, and relations, and demonstrates superior performance on food classification tasks with a large new recipe graph dataset.

Contribution

The paper presents Recipe2Vec, a novel GNN-based model for multi-modal recipe representation, and introduces Large-RG, the largest recipe graph dataset to date.

Findings

Outperforms state-of-the-art baselines in cuisine classification.

Effective integration of multi-modal data improves recipe embeddings.

Adversarial training enhances model stability and performance.

Abstract

Learning effective recipe representations is essential in food studies. Unlike what has been developed for image-based recipe retrieval or learning structural text embeddings, the combined effect of multi-modal information (i.e., recipe images, text, and relation data) receives less attention. In this paper, we formalize the problem of multi-modal recipe representation learning to integrate the visual, textual, and relational information into recipe embeddings. In particular, we first present Large-RG, a new recipe graph data with over half a million nodes, making it the largest recipe graph to date. We then propose Recipe2Vec, a novel graph neural network based recipe embedding model to capture multi-modal information. Additionally, we introduce an adversarial attack strategy to ensure stable learning and improve performance. Finally, we design a joint objective function of node classification and adversarial learning to optimize the model. Extensive experiments demonstrate that Recipe2Vec outperforms state-of-the-art baselines on two classic food study tasks, i.e., cuisine category classification and region prediction. Dataset and codes are available at https://github.com/meettyj/Recipe2Vec.