Unveiling Instruction-Specific Neurons & Experts: An Analytical Framework for LLM's Instruction-Following Capabilities

TL;DR

This paper introduces HexaInst and SPARCOM, an analytical framework to understand how fine-tuning reconfigures LLMs by isolating instruction-specific neurons and experts, revealing their roles in instruction-following capabilities.

Contribution

It presents a new dataset and a novel analytical framework to identify and analyze instruction-specific sparse components in LLMs, advancing understanding of their internal mechanisms.

Findings

Instruction-specific components exhibit functional generality and uniqueness.

Fine-tuning significantly alters sparse components related to instruction execution.

The study highlights the critical role of these components in LLM instruction-following.

Abstract

The finetuning of Large Language Models (LLMs) has significantly advanced their instruction-following capabilities, yet the underlying computational mechanisms driving these improvements remain poorly understood. This study systematically examines how fine-tuning reconfigures LLM computations by isolating and analyzing instruction-specific sparse components, i.e., neurons in dense models and both neurons and experts in Mixture-of-Experts (MoE) architectures. In particular, we introduce HexaInst, a carefully curated and balanced instructional dataset spanning six distinct categories, and propose SPARCOM, a novel analytical framework comprising three key contributions: (1) a method for identifying these sparse components, (2) an evaluation of their functional generality and uniqueness, and (3) a systematic comparison of their alterations. Through experiments, we demonstrate functional generality, uniqueness, and the critical role of these components in instruction execution. By elucidating the relationship between fine-tuning-induced adaptations and sparse computational substrates, this work provides deeper insights into how LLMs internalize instruction-following behavior for the trustworthy LLM community.