CMLFormer: A Dual Decoder Transformer with Switching Point Learning for Code-Mixed Language Modeling

TL;DR

CMLFormer is a novel dual-decoder Transformer model designed specifically for code-mixed language modeling, effectively capturing switching points and cross-lingual structures through specialized pre-training objectives.

Contribution

It introduces a dual-decoder Transformer with synchronized cross-attention and multi-task pre-training tailored for code-mixed languages, addressing structural challenges of language switching.

Findings

Improves F1 score, precision, and accuracy on HASOC-2021 benchmark.

Effectively identifies and attends to switching points in code-mixed text.

Demonstrates architecture's effectiveness for modeling code-mixed languages.

Abstract

Code-mixed languages, characterized by frequent within-sentence language transitions, present structural challenges that standard language models fail to address. In this work, we propose CMLFormer, an enhanced multi-layer dual-decoder Transformer with a shared encoder and synchronized decoder cross-attention, designed to model the linguistic and semantic dynamics of code-mixed text. CMLFormer is pre-trained on an augmented Hinglish corpus with switching point and translation annotations with multiple new objectives specifically aimed at capturing switching behavior, cross-lingual structure, and code-mixing complexity. Our experiments show that CMLFormer improves F1 score, precision, and accuracy over other approaches on the HASOC-2021 benchmark under select pre-training setups. Attention analyses further show that it can identify and attend to switching points, validating its sensitivity to code-mixed structure. These results demonstrate the effectiveness of CMLFormer's architecture and multi-task pre-training strategy for modeling code-mixed languages.