SPEAR-MM: Selective Parameter Evaluation and Restoration via Model Merging for Efficient Financial LLM Adaptation

TL;DR

SPEAR-MM is a framework that selectively preserves general reasoning in financial LLMs during domain adaptation, balancing performance retention and specialization with reduced computational costs.

Contribution

It introduces a novel method combining post-hoc analysis and model merging to maintain core capabilities while adapting LLMs to financial tasks.

Findings

Achieves 91.2% retention of general reasoning capabilities.

Maintains 94% of domain adaptation gains.

Reduces computational costs by 90%.

Abstract

Large language models (LLMs) adapted to financial domains often suffer from catastrophic forgetting of general reasoning capabilities essential for customer interactions and complex financial analysis. We introduce Selective Parameter Evaluation and Restoration via Model Merging (SPEAR-MM), a practical framework that preserves critical capabilities while enabling domain adaptation. Our method approximates layer-wise impact on external benchmarks through post-hoc analysis, then selectively freezes or restores transformer layers via spherical interpolation merging. Applied to LLaMA-3.1-8B for financial tasks, SPEAR-MM achieves 91.2% retention of general capabilities versus 69.7% for standard continual pretraining, while maintaining 94% of domain adaptation gains. The approach provides interpretable trade-off control and reduces computational costs by 90% crucial for resource-constrained financial institutions.