PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion

TL;DR

This paper introduces PALT, a parameter-lite transfer learning method for knowledge graph completion that fine-tunes only a small subset of parameters in pretrained language models, achieving competitive results without full model fine-tuning.

Contribution

The paper proposes a novel parameter-lite transfer learning approach for KG completion that requires tuning significantly fewer parameters than traditional methods.

Findings

Outperforms full fine-tuning approaches on KG benchmarks

Tuning only 1% of parameters achieves competitive results

Maintains original LM parameters fixed during transfer

Abstract

This paper presents a parameter-lite transfer learning approach of pretrained language models (LM) for knowledge graph (KG) completion. Instead of finetuning, which modifies all LM parameters, we only tune a few new parameters while keeping the original LM parameters fixed. We establish this via reformulating KG completion as a "fill-in-the-blank" task, and introducing a parameter-lite encoder on top of the original LMs. We show that, by tuning far fewer parameters than finetuning, LMs transfer non-trivially to most tasks and reach competitiveness with prior state-of-the-art approaches. For instance, we outperform the fully finetuning approaches on a KG completion benchmark by tuning only 1% of the parameters. The code and datasets are available at \url{https://github.com/yuanyehome/PALT}.