Transformer-Based Contextualized Language Models Joint with Neural Networks for Natural Language Inference in Vietnamese

TL;DR

This paper explores combining contextualized language models with neural networks for Vietnamese Natural Language Inference, achieving an F1 score of 82.78%, and demonstrates the effectiveness of the joint approach over fine-tuning existing models.

Contribution

It introduces a novel joint model combining CLM and neural networks for Vietnamese NLI, showing improved performance and resource efficiency.

Findings

XLM-R achieved the highest F1 score of 82.78%.

The joint model outperforms fine-tuned PhoBERT, mBERT, and XLM-R.

The approach is simple and effective for resource-constrained applications.

Abstract

Natural Language Inference (NLI) is a task within Natural Language Processing (NLP) that holds value for various AI applications. However, there have been limited studies on Natural Language Inference in Vietnamese that explore the concept of joint models. Therefore, we conducted experiments using various combinations of contextualized language models (CLM) and neural networks. We use CLM to create contextualized work presentations and use Neural Networks for classification. Furthermore, we have evaluated the strengths and weaknesses of each joint model and identified the model failure points in the Vietnamese context. The highest F1 score in this experiment, up to 82.78% in the benchmark dataset (ViNLI). By conducting experiments with various models, the most considerable size of the CLM is XLM-R (355M). That combination has consistently demonstrated superior performance compared to fine-tuning strong pre-trained language models like PhoBERT (+6.58%), mBERT (+19.08%), and XLM-R (+0.94%) in terms of F1-score. This article aims to introduce a novel approach or model that attains improved performance for Vietnamese NLI. Overall, we find that the joint approach of CLM and neural networks is simple yet capable of achieving high-quality performance, which makes it suitable for applications that require efficient resource utilization.