Intent Classification for Bank Chatbots through LLM Fine-Tuning

TL;DR

This paper compares fine-tuned SlovakBERT and multilingual LLMs for intent classification in banking chatbots, finding SlovakBERT superior in accuracy and false positive rates, establishing a new benchmark.

Contribution

It demonstrates that fine-tuned SlovakBERT outperforms multilingual models for intent classification in banking chatbots, providing a new benchmark for the task.

Findings

SlovakBERT achieves higher in-scope accuracy.

SlovakBERT has lower out-of-scope false positive rate.

SlovakBERT outperforms multilingual models in this application.

Abstract

This study evaluates the application of large language models (LLMs) for intent classification within a chatbot with predetermined responses designed for banking industry websites. Specifically, the research examines the effectiveness of fine-tuning SlovakBERT compared to employing multilingual generative models, such as Llama 8b instruct and Gemma 7b instruct, in both their pre-trained and fine-tuned versions. The findings indicate that SlovakBERT outperforms the other models in terms of in-scope accuracy and out-of-scope false positive rate, establishing it as the benchmark for this application.