BIG MOOD: Relating Transformers to Explicit Commonsense Knowledge

TL;DR

This paper presents a method called BERT Infused Graphs that combines contextual embeddings with commonsense knowledge base embeddings to improve question-answering accuracy, achieving top leaderboard scores without extra pretraining.

Contribution

It introduces a novel approach to integrate BERT with knowledge graph embeddings, including preprocessing, alignment, and contextualization techniques, enhancing BERT's performance on commonsense tasks.

Findings

Achieves higher accuracy than BERT alone.

Scores fifth on the shared task leaderboard.

Outperforms other models without additional pretraining.

Abstract

We introduce a simple yet effective method of integrating contextual embeddings with commonsense graph embeddings, dubbed BERT Infused Graphs: Matching Over Other embeDdings. First, we introduce a preprocessing method to improve the speed of querying knowledge bases. Then, we develop a method of creating knowledge embeddings from each knowledge base. We introduce a method of aligning tokens between two misaligned tokenization methods. Finally, we contribute a method of contextualizing BERT after combining with knowledge base embeddings. We also show BERTs tendency to correct lower accuracy question types. Our model achieves a higher accuracy than BERT, and we score fifth on the official leaderboard of the shared task and score the highest without any additional language model pretraining.