GLOW : Global Weighted Self-Attention Network for Web Search

TL;DR

GLOW introduces a global weighted self-attention mechanism that incorporates corpus-wide statistics into deep matching models, significantly improving web document retrieval performance over BERT by capturing global importance and whole word attention.

Contribution

The paper proposes GLOW, a novel self-attention network that fuses global corpus knowledge into deep matching models and introduces whole word weight sharing for improved web search relevance.

Findings

GLOW outperforms BERT and baselines on public datasets.

GLOW effectively captures topical and semantic information.

The model maintains BERT's complexity while enhancing performance.

Abstract

Deep matching models aim to facilitate search engines retrieving more relevant documents by mapping queries and documents into semantic vectors in the first-stage retrieval. When leveraging BERT as the deep matching model, the attention score across two words are solely built upon local contextualized word embeddings. It lacks prior global knowledge to distinguish the importance of different words, which has been proved to play a critical role in information retrieval tasks. In addition to this, BERT only performs attention across sub-words tokens which weakens whole word attention representation. We propose a novel Global Weighted Self-Attention (GLOW) network for web document search. GLOW fuses global corpus statistics into the deep matching model. By adding prior weights into attention generation from global information, like BM25, GLOW successfully learns weighted attention scores jointly with query matrix $Q$ and key matrix $K$. We also present an efficient whole word weight sharing solution to bring prior whole word knowledge into sub-words level attention. It aids Transformer to learn whole word level attention. To make our models applicable to complicated web search scenarios, we introduce combined fields representation to accommodate documents with multiple fields even with variable number of instances. We demonstrate GLOW is more efficient to capture the topical and semantic representation both in queries and documents. Intrinsic evaluation and experiments conducted on public data sets reveal GLOW to be a general framework for document retrieve task. It significantly outperforms BERT and other competitive baselines by a large margin while retaining the same model complexity with BERT.