FineScope : SAE-guided Data Selection Enables Domain Specific LLM Pruning and Finetuning

TL;DR

FineScope is a novel framework that uses SAE-guided data selection and structured pruning to create compact, domain-specific LLMs that outperform larger models in specialized tasks, with improved efficiency and accuracy.

Contribution

Introducing FineScope, a framework combining SAE-based data selection, structured pruning, and self-data distillation for effective domain-specific LLM adaptation.

Findings

FineScope outperforms several state-of-the-art LLMs in domain-specific tasks.

Pruned models with SAE-curated datasets regain significant performance after fine-tuning.

Applying SAE-curated datasets to pretrained LLMs improves domain accuracy without pruning.

Abstract

Training large language models (LLMs) from scratch requires significant computational resources, driving interest in developing smaller, domain-specific LLMs that maintain both efficiency and strong task performance. Medium-sized models such as LLaMA, llama} have served as starting points for domain-specific adaptation, but they often suffer from accuracy degradation when tested on specialized datasets. We introduce FineScope, a framework for deriving compact, domain-optimized LLMs from larger pretrained models. FineScope leverages the Sparse Autoencoder (SAE) framework, inspired by its ability to produce interpretable feature representations, to extract domain-specific subsets from large datasets. We apply structured pruning with domain-specific constraints, ensuring that the resulting pruned models retain essential knowledge for the target domain. To further enhance performance, these pruned models undergo self-data distillation, leveraging SAE-curated datasets to restore key domain-specific information lost during pruning. Extensive experiments and ablation studies demonstrate that FineScope achieves highly competitive performance, outperforming several large-scale state-of-the-art LLMs in domain-specific tasks. Additionally, our results show that FineScope enables pruned models to regain a substantial portion of their original performance when fine-tuned with SAE-curated datasets. Furthermore, applying these datasets to fine-tune pretrained LLMs without pruning also improves their domain-specific accuracy, highlighting the robustness of our approach.