Analyzing Commonsense Emergence in Few-shot Knowledge Models

TL;DR

This paper investigates how large language models encode commonsense knowledge, revealing that few-shot fine-tuning helps models access pre-existing knowledge rather than learning it from scratch.

Contribution

It demonstrates that commonsense knowledge models adapt quickly in few-shot settings, showing fine-tuning acts as an interface to pre-trained knowledge rather than learning anew.

Findings

Models rapidly adapt with limited examples

Fine-tuning acts as an interface to pre-trained knowledge

Parameter analysis reveals how knowledge is accessed

Abstract

Recently, commonsense knowledge models - pretrained language models (LM) fine-tuned on knowledge graph (KG) tuples - showed that considerable amounts of commonsense knowledge can be encoded in the parameters of large language models. However, as parallel studies show that LMs are poor hypothesizers of declarative commonsense relationships on their own, it remains unclear whether this knowledge is learned during pretraining or from fine-tuning on KG examples. To investigate this question, we train commonsense knowledge models in few-shot settings to study the emergence of their commonsense representation abilities. Our results show that commonsense knowledge models can rapidly adapt from limited examples, indicating that KG fine-tuning serves to learn an interface to encoded knowledge learned during pretraining. Importantly, our analysis of absolute, angular, and distributional parameter changes during few-shot fine-tuning provides novel insights into how this interface is learned.