FEVO: Financial Knowledge Expansion and Reasoning Evolution for Large Language Models

TL;DR

FEVO is a multi-stage framework that enhances large language models with financial domain knowledge and reasoning skills, achieving state-of-the-art results in financial benchmarks.

Contribution

The paper introduces FEVO, a novel multi-stage training framework combining continued pre-training, supervised fine-tuning, and reinforcement learning for financial domain expertise in LLMs.

Findings

FEVO-R32B outperforms larger models on five financial benchmarks.

FEVO significantly improves reasoning and domain knowledge in LLMs.

The framework effectively integrates domain knowledge with structured reasoning.

Abstract

Advancements in reasoning for large language models (LLMs) have lead to significant performance improvements for LLMs in various fields such as mathematics and programming. However, research applying these advances to the financial domain, where considerable domain-specific knowledge is necessary to complete tasks, remains limited. To address this gap, we introduce FEVO (Financial Evolution), a multi-stage enhancement framework developed to enhance LLM performance in the financial domain. FEVO systemically enhances LLM performance by using continued pre-training (CPT) to expand financial domain knowledge, supervised fine-tuning (SFT) to instill structured, elaborate reasoning patterns, and reinforcement learning (RL) to further integrate the expanded financial domain knowledge with the learned structured reasoning. To ensure effective and efficient training, we leverage frontier reasoning models and rule-based filtering to curate FEVO-Train, high-quality datasets specifically designed for the different post-training phases. Using our framework, we train the FEVO series of models - C32B, S32B, R32B - from Qwen2.5-32B and evaluate them on seven benchmarks to assess financial and general capabilities, with results showing that FEVO-R32B achieves state-of-the-art performance on five financial benchmarks against much larger models as well as specialist models. More significantly, FEVO-R32B demonstrates markedly better performance than FEVO-R32B-0 (trained from Qwen2.5-32B-Instruct using only RL), thus validating the effectiveness of financial domain knowledge expansion and structured, logical reasoning distillation