Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval

TL;DR

This paper introduces T-Food, a novel cross-modal food retrieval framework that combines multimodal regularization with unimodal encoders, achieving significant performance improvements on large-scale datasets.

Contribution

The paper presents a new retrieval framework that exploits cross-modal interaction via regularization while maintaining efficient unimodal encoders at test time, and introduces dynamic triplet loss variants.

Findings

Outperforms existing methods on Recipe1M dataset

Achieves 8.1% and 10.9% absolute improvements in R@1 on 1k and 10k test sets

Utilizes recent VLP models like CLIP for image encoding

Abstract

Cross-modal image-recipe retrieval has gained significant attention in recent years. Most work focuses on improving cross-modal embeddings using unimodal encoders, that allow for efficient retrieval in large-scale databases, leaving aside cross-attention between modalities which is more computationally expensive. We propose a new retrieval framework, T-Food (Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval) that exploits the interaction between modalities in a novel regularization scheme, while using only unimodal encoders at test time for efficient retrieval. We also capture the intra-dependencies between recipe entities with a dedicated recipe encoder, and propose new variants of triplet losses with dynamic margins that adapt to the difficulty of the task. Finally, we leverage the power of the recent Vision and Language Pretraining (VLP) models such as CLIP for the image encoder. Our approach outperforms existing approaches by a large margin on the Recipe1M dataset. Specifically, we achieve absolute improvements of 8.1 % (72.6 R@1) and +10.9 % (44.6 R@1) on the 1k and 10k test sets respectively. The code is available here:https://github.com/mshukor/TFood