Efficient Document Retrieval by End-to-End Refining and Quantizing BERT Embedding with Contrastive Product Quantization
Zexuan Qiu, Qinliang Su, Jianxing Yu, and Shijing Si

TL;DR
This paper introduces an end-to-end method that refines and quantizes BERT embeddings using contrastive product quantization for more accurate and efficient document retrieval, surpassing existing semantic hashing techniques.
Contribution
It proposes a novel end-to-end framework that transforms BERT embeddings and applies probabilistic product quantization with contrastive loss and mutual information maximization.
Findings
Significantly outperforms state-of-the-art baselines on three benchmarks.
Effectively captures semantic information with real-valued codewords.
Improves document retrieval accuracy and efficiency.
Abstract
Efficient document retrieval heavily relies on the technique of semantic hashing, which learns a binary code for every document and employs Hamming distance to evaluate document distances. However, existing semantic hashing methods are mostly established on outdated TFIDF features, which obviously do not contain lots of important semantic information about documents. Furthermore, the Hamming distance can only be equal to one of several integer values, significantly limiting its representational ability for document distances. To address these issues, in this paper, we propose to leverage BERT embeddings to perform efficient retrieval based on the product quantization technique, which will assign for every document a real-valued codeword from the codebook, instead of a binary code as in semantic hashing. Specifically, we first transform the original BERT embeddings via a learnable…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Image and Video Retrieval Techniques · Image Retrieval and Classification Techniques · Text and Document Classification Technologies
MethodsMulti-Head Attention · Attention Is All You Need · Linear Warmup With Linear Decay · Attention Dropout · Weight Decay · Dense Connections · Linear Layer · Layer Normalization · Residual Connection · Dropout
