Modality-Agnostic Attention Fusion for visual search with text feedback

TL;DR

The paper introduces MAAF, a modality-agnostic attention fusion model for visual search with natural language feedback, outperforming existing methods on multiple datasets and establishing new benchmarks with rich language inputs.

Contribution

It proposes a novel attention fusion approach that effectively combines image and text features for visual search, and introduces new challenging benchmarks with detailed analysis.

Findings

Outperforms existing methods on Fashion IQ and CSS datasets.

Achieves competitive results on Fashion200k dataset.

Outperforms baselines on new benchmarks with rich language inputs.

Abstract

Image retrieval with natural language feedback offers the promise of catalog search based on fine-grained visual features that go beyond objects and binary attributes, facilitating real-world applications such as e-commerce. Our Modality-Agnostic Attention Fusion (MAAF) model combines image and text features and outperforms existing approaches on two visual search with modifying phrase datasets, Fashion IQ and CSS, and performs competitively on a dataset with only single-word modifications, Fashion200k. We also introduce two new challenging benchmarks adapted from Birds-to-Words and Spot-the-Diff, which provide new settings with rich language inputs, and we show that our approach without modification outperforms strong baselines. To better understand our model, we conduct detailed ablations on Fashion IQ and provide visualizations of the surprising phenomenon of words avoiding "attending" to the image region they refer to.