Thai Financial Domain Adaptation of THaLLE -- Technical Report
KBTG Labs, Atthakorn Petchsod, Pornchanan Balee, Danupat Khamnuansin,, Anuruth Lertpiya, Chanatip Saetia, Tawunrat Chalothorn, Thadpong, Pongthawornkamol, Monchai Lertsutthiwong

TL;DR
This paper presents a specialized Thai financial language model trained on local exam data, employing advanced fine-tuning techniques to improve domain-specific performance and support financial advisory tasks in Thailand.
Contribution
The authors developed a Thai financial domain-specific LLM using innovative training methods like ReLoRA, CPT, and rsLoRA, tailored for Thai financial regulations and terminology.
Findings
Achieved 72%, 72%, and 84% scores on IC exam levels P1, P2, and P3.
Demonstrated effectiveness in Thai financial advisory tasks.
Showcased potential for localized financial NLP applications.
Abstract
Large Language Models (LLMs) excel in general tasks but struggle with domain-specific challenges, such as specialized terminology and localized regulations. Existing financial LLMs, like FinGPT and BloombergGPT, lack support for the Thai financial domain. We developed a Thai Financial LLM using the Investment Consultant (IC) exam dataset from the Stock Exchange of Thailand. To address dataset limitations, we applied data augmentation, ReLoRA for efficient training, Continued Pretraining (CPT) for domain knowledge, and Rank-Stabilized LoRA (rsLoRA) for fine-tuning. Supervised Fine-Tuning (SFT) simulated exam scenarios, while Direct Preference Optimization (DPO) refined the model using feedback. The model achieved scores of 72%, 72%, and 84% on IC exam levels P1, P2, and P3, respectively, demonstrating its effectiveness in Thai financial advisory tasks and its potential for specialized…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsFinancial Distress and Bankruptcy Prediction
