RepCali: High Efficient Fine-tuning Via Representation Calibration in Latent Space for Pre-trained Language Models

TL;DR

RepCali introduces a universal, plug-and-play representation calibration method in the latent space of pre-trained language models, significantly enhancing downstream task performance across diverse models and languages.

Contribution

The paper proposes RepCali, a novel calibration block for latent space, improving fine-tuning efficiency and effectiveness for all encoder-decoder PLMs with minimal complexity.

Findings

Significant performance improvements on 8 tasks across 25 models.

Universality and ease of integration of RepCali with various PLMs.

Outperforms existing fine-tuning baselines in benchmark tests.

Abstract

Fine-tuning pre-trained language models (PLMs) has become a dominant paradigm in applying PLMs to downstream tasks. However, with limited fine-tuning, PLMs still struggle with the discrepancies between the representation obtained from the PLMs' encoder and the optimal input to the PLMs' decoder. This paper tackles this challenge by learning to calibrate the representation of PLMs in the latent space. In the proposed representation calibration method (RepCali), we integrate a specific calibration block to the latent space after the encoder and use the calibrated output as the decoder input. The merits of the proposed RepCali include its universality to all PLMs with encoder-decoder architectures, its plug-and-play nature, and ease of implementation. Extensive experiments on 25 PLM-based models across 8 tasks (including both English and Chinese datasets) demonstrate that the proposed RepCali offers desirable enhancements to PLMs (including LLMs) and significantly improves the performance of downstream tasks. Comparison experiments across 4 benchmark tasks indicate that RepCali is superior to the representative fine-tuning baselines.