TL;DR
This paper evaluates various Transformer-based models for sentiment analysis on Spanish-English code-mixed social media data, introducing a two-step fine-tuning approach that improves performance.
Contribution
The paper compares monolingual and multilingual models and proposes a novel two-step fine-tuning method for better sentiment classification.
Findings
XLM-RoBERTa achieved the highest weighted F1-score of 0.537 on development data.
Two-step fine-tuning outperforms standard fine-tuning.
Team ranked tenth overall in SemEval-2020 Task 9.
Abstract
In this paper, we present our approach for sentiment classification on Spanish-English code-mixed social media data in the SemEval-2020 Task 9. We investigate performance of various pre-trained Transformer models by using different fine-tuning strategies. We explore both monolingual and multilingual models with the standard fine-tuning method. Additionally, we propose a custom model that we fine-tune in two steps: once with a language modeling objective, and once with a task-specific objective. Although two-step fine-tuning improves sentiment classification performance over the base model, the large multilingual XLM-RoBERTa model achieves best weighted F1-score with 0.537 on development data and 0.739 on test data. With this score, our team jupitter placed tenth overall in the competition.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsLinear Layer · Absolute Position Encodings · Position-Wise Feed-Forward Layer · Multi-Head Attention · Label Smoothing · Adam · Dropout · Softmax · Layer Normalization · Dense Connections
