Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance

TL;DR

This paper explores methods for merging domain-specific continual pretraining models to create versatile financial language models, demonstrating improved performance and emergent skills through a comprehensive evaluation framework.

Contribution

It introduces the first systematic analysis of CPT model merging, proposing three merging methods and a three-stage evaluation framework for multi-skill LLMs.

Findings

Merging experts with base models recovers lost general knowledge.

Merging experts enhances performance and enables emergent cross-domain skills.

TIES method is more robust than Task Arithmetic.

Abstract

While LLMs excel at general tasks, they struggle in specialized domains like finance, requiring diverse skills in domain knowledge, mathematical reasoning, and multilingual processing. Merging domain-specific Continual Pre-training (CPT) "experts" offers a practical alternative to costly and unstable multi-skill training. However, unlike established Supervised Fine-Tuning (SFT) model-based merging, CPT model merging remains largely unexplored. We address this gap by creating financial LLMs from experts in finance, math, and Japanese. We propose a three-stage evaluation focusing on knowledge recovery, complementarity, and emergence, and assess three merging methods (Task Arithmetic, TIES, and DARE-TIES) on a comprehensive financial benchmark curated from 18 tasks across 8 established datasets. Results show that merging an expert with its base model recovers general knowledge lost during CPT, while merging experts improves performance and can yield emergent cross-domain skills. Among the methods, Task Arithmetic performs strongly but is hyperparameter-sensitive, whereas TIES is more robust. Our findings also suggest that while model similarity correlates with merging success, emergent skills depend on more complex factors. This work presents the first foundational analysis of CPT model merging, establishing a principled framework and providing clear guidance for building multi-skill LLMs from existing assets.