Can abstract concepts from LLM improve SLM performance?

TL;DR

This paper explores how high-level abstract concepts extracted from large language models can be transferred to smaller models during inference, significantly improving their performance without extensive retraining.

Contribution

It introduces a method for transferring steering vectors from large to small models and demonstrates inference-time scaling to boost accuracy across various tasks.

Findings

Effective transfer of concepts improves small model performance.

Inference-time scaling enhances accuracy by 7-15%.

Method is applicable across different model families.

Abstract

Large language models (LLMs) excel at diverse tasks, but their deployment on resource-constrained devices remains challenging. Existing methods like quantization, pruning, and distillation can reduce memory footprint but often demand extensive experimentation and careful infrastructure design. Leveraging existing techniques for extracting high-level concepts (represented as steering vectors) from larger models, we investigate their transferability to smaller language models (SLM) during inference. We demonstrate through extensive experimentation that these concepts can be effectively transferred to smaller models, irrespective of their family (e.g., Phi, Llama, Qwen), leading to performance improvements across a wide range of tasks. Furthermore, we introduce inference-time scaling to enhance performance by dynamically adjusting the steering intensity which has resulted in a 7-15\% of accuracy improvement for Qwen3-0.6B.