Enhancing Cross-task Transfer of Large Language Models via Activation Steering

TL;DR

This paper introduces CAST, a novel method that improves cross-task transfer in large language models by steering internal activations, enhancing performance in low-resource and cross-lingual tasks without additional parameter updates.

Contribution

The paper proposes a new activation steering framework that leverages latent space manipulation for effective transfer across tasks without parameter fine-tuning.

Findings

Outperforms baseline methods in cross-domain transfer tasks

Demonstrates superior scalability and efficiency

Effective in low-resource and cross-lingual scenarios

Abstract

Large language models (LLMs) have shown impressive abilities in leveraging pretrained knowledge through prompting, but they often struggle with unseen tasks, particularly in data-scarce scenarios. While cross-task in-context learning offers a direct solution for transferring knowledge across tasks, it still faces critical challenges in terms of robustness, scalability, and efficiency. In this paper, we investigate whether cross-task transfer can be achieved via latent space steering without parameter updates or input expansion. Through an analysis of activation patterns in the latent space of LLMs, we observe that the enhanced activations induced by in-context examples have consistent patterns across different tasks. Inspired by these findings, we propose CAST, a novel Cross-task Activation Steering Transfer framework that enables effective transfer by manipulating the model's internal activation states. Our approach first selects influential and diverse samples from high-resource tasks, then utilizes their contrastive representation-enhanced activations to adapt LLMs to low-resource tasks. Extensive experiments across both cross-domain and cross-lingual transfer settings show that our method outperforms competitive baselines and demonstrates superior scalability and lower computational costs.