DeCoVec: Building Decoding Space based Task Vector for Large Language Models via In-Context Learning

TL;DR

DeCoVec is a training-free, non-invasive method that constructs task vectors in the decoding space of large language models using in-context learning, improving task performance without fine-tuning.

Contribution

It introduces a novel approach to steer LLMs by constructing task vectors directly in the decoding space via ICL, avoiding fine-tuning or internal state manipulation.

Findings

DeCoVec outperforms standard few-shot baselines on multiple benchmarks.

It improves accuracy by up to +5.50 points.

It reduces generation degeneration and logical flaws.

Abstract

Task vectors, representing directions in model or activation spaces that encode task-specific behaviors, have emerged as a promising tool for steering large language models (LLMs). However, existing approaches typically require fine-tuning or invasive manipulation of internal states, limiting their flexibility and scalability. We propose \textsc{DeCoVec} (Decoding Space based Task Vector), a training-free and non-invasive framework that constructs task vectors directly in the \textit{decoding space} by leveraging in-context learning (ICL). Specifically, \textsc{DeCoVec} captures the task essence as the difference between the output logit distributions of few-shot and zero-shot prompts, then steers generation by injecting this vector into the decoding process. Experiments across seven LLMs (0.5B--9B) on TruthfulQA, Math-500, and AQUA-RAT show that \textsc{DeCoVec} consistently outperforms standard few-shot baselines, with gains up to +5.50 average accuracy. Further analysis demonstrates that \textsc{DeCoVec} effectively suppresses generation degeneration and logical flaws while exhibiting strong robustness to demonstration ordering, all without incurring additional input token costs. Our method offers a training-free and non-invasive solution for LLM steering without requiring weight updates or auxiliary models.