Enhancing Fact Retrieval in PLMs through Truthfulness

TL;DR

This paper proposes a method using a helper model to assess truthfulness via hidden states in PLMs, significantly improving factual knowledge retrieval by up to 33%.

Contribution

It introduces a novel approach leveraging hidden states and a helper model to enhance fact retrieval in pre-trained language models.

Findings

Fact retrieval improved by up to 33%

Hidden states effectively assess truthfulness

Potential for better knowledge extraction from PLMs

Abstract

Pre-trained Language Models (PLMs) encode various facts about the world at their pre-training phase as they are trained to predict the next or missing word in a sentence. There has a been an interest in quantifying and improving the amount of facts that can be extracted from PLMs, as they have been envisioned to act as soft knowledge bases, which can be queried in natural language. Different approaches exist to enhance fact retrieval from PLM. Recent work shows that the hidden states of PLMs can be leveraged to determine the truthfulness of the PLMs' inputs. Leveraging this finding to improve factual knowledge retrieval remains unexplored. In this work, we investigate the use of a helper model to improve fact retrieval. The helper model assesses the truthfulness of an input based on the corresponding hidden states representations from the PLMs. We evaluate this approach on several masked PLMs and show that it enhances fact retrieval by up to 33\%. Our findings highlight the potential of hidden states representations from PLMs in improving their factual knowledge retrieval.