Visual Natural Language Query Auto-Completion for Estimating Instance Probabilities

TL;DR

This paper introduces a novel task of query auto-completion that combines visual and textual data to estimate instance probabilities, improving ranking accuracy over language-only methods.

Contribution

It proposes a new vision-language query auto-completion framework using fine-tuned BERT embeddings for better instance probability estimation.

Findings

Vision and language auto-completion outperforms language-only methods.

Fine-tuning BERT embeddings improves instance ranking.

The approach is agnostic to segmentation or attention mechanisms.

Abstract

We present a new task of query auto-completion for estimating instance probabilities. We complete a user query prefix conditioned upon an image. Given the complete query, we fine tune a BERT embedding for estimating probabilities of a broad set of instances. The resulting instance probabilities are used for selection while being agnostic to the segmentation or attention mechanism. Our results demonstrate that auto-completion using both language and vision performs better than using only language, and that fine tuning a BERT embedding allows to efficiently rank instances in the image. In the spirit of reproducible research we make our data, models, and code available.