Static Embeddings as Efficient Knowledge Bases?

TL;DR

This paper demonstrates that static embeddings can serve as efficient and effective knowledge bases, outperforming large pretrained language models in factual recall tasks with lower energy costs across multiple languages.

Contribution

The study shows that simple static embeddings, with restricted output spaces, outperform complex PLMs like BERT in factual knowledge retrieval, highlighting their efficiency and effectiveness.

Findings

Static embeddings outperform BERT in factual recall by 1.6% points.

Static embeddings use only 0.3% of BERT's energy for training.

Performance advantage observed across ten diverse languages.

Abstract

Recent research investigates factual knowledge stored in large pretrained language models (PLMs). Instead of structural knowledge base (KB) queries, masked sentences such as "Paris is the capital of [MASK]" are used as probes. The good performance on this analysis task has been interpreted as PLMs becoming potential repositories of factual knowledge. In experiments across ten linguistically diverse languages, we study knowledge contained in static embeddings. We show that, when restricting the output space to a candidate set, simple nearest neighbor matching using static embeddings performs better than PLMs. E.g., static embeddings perform 1.6% points better than BERT while just using 0.3% of energy for training. One important factor in their good comparative performance is that static embeddings are standardly learned for a large vocabulary. In contrast, BERT exploits its more sophisticated, but expensive ability to compose meaningful representations from a much smaller subword vocabulary.