Learning Colour Representations of Search Queries

TL;DR

This paper explores how colour information in search queries can be modeled using neural networks and click data to improve image ranking relevance.

Contribution

It introduces a neural network-based method to encode search queries into colour space and integrates this with relevance ranking models.

Findings

Colour-based query embeddings improve ranking performance.

Query-image colour distance enhances relevance scoring.

The approach captures implicit and explicit colour notions in queries.

Abstract

Image search engines rely on appropriately designed ranking features that capture various aspects of the content semantics as well as the historic popularity. In this work, we consider the role of colour in this relevance matching process. Our work is motivated by the observation that a significant fraction of user queries have an inherent colour associated with them. While some queries contain explicit colour mentions (such as 'black car' and 'yellow daisies'), other queries have implicit notions of colour (such as 'sky' and 'grass'). Furthermore, grounding queries in colour is not a mapping to a single colour, but a distribution in colour space. For instance, a search for 'trees' tends to have a bimodal distribution around the colours green and brown. We leverage historical clickthrough data to produce a colour representation for search queries and propose a recurrent neural network architecture to encode unseen queries into colour space. We also show how this embedding can be learnt alongside a cross-modal relevance ranker from impression logs where a subset of the result images were clicked. We demonstrate that the use of a query-image colour distance feature leads to an improvement in the ranker performance as measured by users' preferences of clicked versus skipped images.