Linguistic Fingerprint in Transformer Models: How Language Variation Influences Parameter Selection in Irony Detection

TL;DR

This study investigates how linguistic variations in English influence transformer models for irony detection, revealing significant parameter sharing and structural similarities across models trained on different language variants.

Contribution

It demonstrates that transformer models trained on different English variations share at least 60% of their parameters, highlighting the importance of parameter values in capturing linguistic nuances.

Findings

Optimal subnetworks share at least 60% of parameters.

Linguistic variations exhibit structural similarities in models.

Parameter values are crucial for capturing language nuances.

Abstract

This paper explores the correlation between linguistic diversity, sentiment analysis and transformer model architectures. We aim to investigate how different English variations impact transformer-based models for irony detection. To conduct our study, we used the EPIC corpus to extract five diverse English variation-specific datasets and applied the KEN pruning algorithm on five different architectures. Our results reveal several similarities between optimal subnetworks, which provide insights into the linguistic variations that share strong resemblances and those that exhibit greater dissimilarities. We discovered that optimal subnetworks across models share at least 60% of their parameters, emphasizing the significance of parameter values in capturing and interpreting linguistic variations. This study highlights the inherent structural similarities between models trained on different variants of the same language and also the critical role of parameter values in capturing these nuances.