Learning Joint Embedding with Modality Alignments for Cross-Modal Retrieval of Recipes and Food Images

TL;DR

This paper introduces JEMA, a three-tier modality alignment method for learning joint embeddings of recipes and food images, significantly improving cross-modal retrieval accuracy on large-scale datasets.

Contribution

The paper proposes a novel three-tier modality alignment framework that enhances text and image embeddings and incorporates auxiliary cross-modal regularizations for better retrieval performance.

Findings

Outperforms state-of-the-art methods on Recipe1M dataset

Achieves higher retrieval accuracy in cross-modal tasks

Demonstrates effectiveness of multi-tier alignment in joint embedding learning

Abstract

This paper presents a three-tier modality alignment approach to learning text-image joint embedding, coined as JEMA, for cross-modal retrieval of cooking recipes and food images. The first tier improves recipe text embedding by optimizing the LSTM networks with term extraction and ranking enhanced sequence patterns, and optimizes the image embedding by combining the ResNeXt-101 image encoder with the category embedding using wideResNet-50 with word2vec. The second tier modality alignment optimizes the textual-visual joint embedding loss function using a double batch-hard triplet loss with soft-margin optimization. The third modality alignment incorporates two types of cross-modality alignments as the auxiliary loss regularizations to further reduce the alignment errors in the joint learning of the two modality-specific embedding functions. The category-based cross-modal alignment aims to align the image category with the recipe category as a loss regularization to the joint embedding. The cross-modal discriminator-based alignment aims to add the visual-textual embedding distribution alignment to further regularize the joint embedding loss. Extensive experiments with the one-million recipes benchmark dataset Recipe1M demonstrate that the proposed JEMA approach outperforms the state-of-the-art cross-modal embedding methods for both image-to-recipe and recipe-to-image retrievals.