THaLLE-ThaiLLM: Domain-Specialized Small LLMs for Finance and Thai -- Technical Report

TL;DR

This paper explores the use of model merging to efficiently develop multi-capability small LLMs tailored for finance and Thai language, demonstrating improved performance across multiple benchmarks.

Contribution

It introduces a resource-efficient method of creating multi-capability LLMs through model merging, specifically for Thai language and financial domains.

Findings

Merging ThaiLLM-8B with Qwen-8B improves Thai language task performance.

Combining ThaiLLM-8B and THaLLE-CFA-8B with Qwen-8B enhances general and financial domain capabilities.

Model merging is viable for developing high-performance, multi-capability LLMs.

Abstract

Large Language Models (LLMs) have demonstrated significant potential across various domains, particularly in banking and finance, where they can automate complex tasks and enhance decision-making at scale. Due to privacy, security, and regulatory concerns, organizations often prefer on-premise deployment of LLMs. The ThaiLLM initiative aims to enhance Thai language capabilities in open-LLMs, enabling Thai industry to leverage advanced language models. However, organizations often face a trade-off between deploying multiple specialized models versus the prohibitive expense of training a single multi-capability model. To address this, we explore model merging as a resource-efficient alternative for developing high-performance, multi-capability LLMs. We present results from two key experiments: first, merging Qwen-8B with ThaiLLM-8B demonstrates how ThaiLLM-8B enhances Thai general capabilities, showing an uplift of M3 and M6 O-NET exams over the general instruction-following Qwen-8B. Second, we merge Qwen-8B with both ThaiLLM-8B and THaLLE-CFA-8B. This combination results in further improvements in performance across both general and financial domains, by demonstrating an uplift in both M3 and M6 O-NET, Flare-CFA, and Thai-IC benchmarks. The report showcases the viability of model merging for efficiently creating multi-capability LLMs.