THaLLE-ThaiLLM: Domain-Specialized Small LLMs for Finance and Thai -- Technical Report
KBTG Labs: Anuruth Lertpiya, Danupat Khamnuansin, Kantapong Sucharitpongpan, Pornchanan Balee, Tawunrat Chalothorn, Thadpong Pongthawornkamol, Monchai Lertsutthiwong

TL;DR
This paper explores the use of model merging to efficiently develop multi-capability small LLMs tailored for finance and Thai language, demonstrating improved performance across multiple benchmarks.
Contribution
It introduces a resource-efficient method of creating multi-capability LLMs through model merging, specifically for Thai language and financial domains.
Findings
Merging ThaiLLM-8B with Qwen-8B improves Thai language task performance.
Combining ThaiLLM-8B and THaLLE-CFA-8B with Qwen-8B enhances general and financial domain capabilities.
Model merging is viable for developing high-performance, multi-capability LLMs.
Abstract
Large Language Models (LLMs) have demonstrated significant potential across various domains, particularly in banking and finance, where they can automate complex tasks and enhance decision-making at scale. Due to privacy, security, and regulatory concerns, organizations often prefer on-premise deployment of LLMs. The ThaiLLM initiative aims to enhance Thai language capabilities in open-LLMs, enabling Thai industry to leverage advanced language models. However, organizations often face a trade-off between deploying multiple specialized models versus the prohibitive expense of training a single multi-capability model. To address this, we explore model merging as a resource-efficient alternative for developing high-performance, multi-capability LLMs. We present results from two key experiments: first, merging Qwen-8B with ThaiLLM-8B demonstrates how ThaiLLM-8B enhances Thai general…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Machine Learning and Data Classification · Explainable Artificial Intelligence (XAI)
