A Comparative Analysis of Instruction Fine-Tuning LLMs for Financial Text Classification

TL;DR

This paper evaluates instruction fine-tuning of smaller LLMs for financial text classification, demonstrating improved performance and robustness, especially when combined with model merging techniques for zero-shot tasks.

Contribution

It introduces a comprehensive comparison of instruction fine-tuning on smaller LLMs for financial NLP tasks and proposes model merging to enhance zero-shot capabilities.

Findings

Instruction fine-tuning improves financial text classification performance.

Model merging enhances zero-shot generalization beyond original models.

Instruction-tuned models are more robust to degradation on unseen tasks.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities across diverse Natural Language Processing (NLP) tasks, including language understanding, reasoning, and generation. However, general-domain LLMs often struggle with financial tasks due to the technical and specialized nature of financial texts. This study investigates the efficacy of instruction fine-tuning smaller-scale LLMs, including Mistral-7B, Llama3-8B, and Phi3-mini, to enhance their performance in financial text classification tasks. We fine-tuned both instruction-tuned and base models across four financial classification tasks, achieving significant improvements in task-specific performance. Furthermore, we evaluated the zero-shot capabilities of these fine-tuned models on three unseen complex financial tasks, including argument classification, deal completeness classification, and causal classification. Our results indicate while base model fine-tuning led to greater degradation, instruction-tuned models maintained more robust performance. To address this degradation, we employed model merging techniques, integrating single-task domain-specific fine-tuned models with the base model. Using this merging method resulted in significant enhancements in zero-shot performance, even exceeding the original model's accuracy on certain datasets. Our findings underscore the effectiveness of instruction fine-tuning and model merging for adapting LLMs to specialized financial text classification tasks.