Task-guided Disentangled Tuning for Pretrained Language Models

TL;DR

Task-guided Disentangled Tuning (TDT) improves the adaptation of pretrained language models to specific NLP tasks by disentangling task-relevant signals, leading to better performance especially in low-data scenarios.

Contribution

The paper introduces TDT, a novel method that enhances PLMs' generalization by disentangling task-specific signals using a learnable confidence model and regularization.

Findings

TDT outperforms standard fine-tuning on GLUE and CLUE benchmarks.

TDT demonstrates robustness across different PLMs and tasks.

Disentangling task signals improves low-data regime performance.

Abstract

Pretrained language models (PLMs) trained on large-scale unlabeled corpus are typically fine-tuned on task-specific downstream datasets, which have produced state-of-the-art results on various NLP tasks. However, the data discrepancy issue in domain and scale makes fine-tuning fail to efficiently capture task-specific patterns, especially in the low data regime. To address this issue, we propose Task-guided Disentangled Tuning (TDT) for PLMs, which enhances the generalization of representations by disentangling task-relevant signals from the entangled representations. For a given task, we introduce a learnable confidence model to detect indicative guidance from context, and further propose a disentangled regularization to mitigate the over-reliance problem. Experimental results on GLUE and CLUE benchmarks show that TDT gives consistently better results than fine-tuning with different PLMs, and extensive analysis demonstrates the effectiveness and robustness of our method. Code is available at https://github.com/lemon0830/TDT.