Zero-shot Slot Filling with DPR and RAG

TL;DR

This paper presents enhancements to retrieval-augmented language models for zero-shot slot filling, achieving top performance on key benchmarks by improving retrieval and generation components.

Contribution

The authors introduce strategies to improve RAG-based models for slot filling, resulting in state-of-the-art performance on KILT benchmark datasets.

Findings

KGI0 system achieved top-1 on T-REx and zsRE datasets.

Enhanced retrieval and generation components improved slot filling accuracy.

Model outperforms previous approaches on real-world benchmarks.

Abstract

The ability to automatically extract Knowledge Graphs (KG) from a given collection of documents is a long-standing problem in Artificial Intelligence. One way to assess this capability is through the task of slot filling. Given an entity query in form of [Entity, Slot, ?], a system is asked to `fill' the slot by generating or extracting the missing value from a relevant passage or passages. This capability is crucial to create systems for automatic knowledge base population, which is becoming in ever-increasing demand, especially in enterprise applications. Recently, there has been a promising direction in evaluating language models in the same way we would evaluate knowledge bases, and the task of slot filling is the most suitable to this intent. The recent advancements in the field try to solve this task in an end-to-end fashion using retrieval-based language models. Models like Retrieval Augmented Generation (RAG) show surprisingly good performance without involving complex information extraction pipelines. However, the results achieved by these models on the two slot filling tasks in the KILT benchmark are still not at the level required by real-world information extraction systems. In this paper, we describe several strategies we adopted to improve the retriever and the generator of RAG in order to make it a better slot filler. Our KGI0 system (available at https://github.com/IBM/retrieve-write-slot-filling) reached the top-1 position on the KILT leaderboard on both T-REx and zsRE dataset with a large margin.