UZH_CLyp at SemEval-2023 Task 9: Head-First Fine-Tuning and ChatGPT Data Generation for Cross-Lingual Learning in Tweet Intimacy Prediction

TL;DR

This paper presents a cross-lingual tweet intimacy prediction method using Head-First Fine-Tuning and ChatGPT-generated data, achieving high multilingual performance and addressing data scarcity issues.

Contribution

It introduces HeFiT, a novel fine-tuning approach, and demonstrates the effectiveness of synthetic data for low-resource multilingual tweet analysis.

Findings

HeFiT stabilizes training and improves results.

Synthetic ChatGPT data enhances cross-lingual performance.

Inconsistencies in annotations affect cross-lingual transfer.

Abstract

This paper describes the submission of UZH_CLyp for the SemEval 2023 Task 9 "Multilingual Tweet Intimacy Analysis". We achieved second-best results in all 10 languages according to the official Pearson's correlation regression evaluation measure. Our cross-lingual transfer learning approach explores the benefits of using a Head-First Fine-Tuning method (HeFiT) that first updates only the regression head parameters and then also updates the pre-trained transformer encoder parameters at a reduced learning rate. Additionally, we study the impact of using a small set of automatically generated examples (in our case, from ChatGPT) for low-resource settings where no human-labeled data is available. Our study shows that HeFiT stabilizes training and consistently improves results for pre-trained models that lack domain adaptation to tweets. Our study also shows a noticeable performance increase in cross-lingual learning when synthetic data is used, confirming the usefulness of current text generation systems to improve zero-shot baseline results. Finally, we examine how possible inconsistencies in the annotated data contribute to cross-lingual interference issues.