OYXOY: A Modern NLP Test Suite for Modern Greek
Konstantinos Kogkalidis, Stergios Chatzikyriakidis, Eirini, Chrysovalantou Giannikouri, Vassiliki Katsouli, Christina Klironomou,, Christina Koula, Dimitris Papadakis, Thelka Pasparaki, Erofili Psaltaki,, Efthymia Sakellariou, Hara Soupiona

TL;DR
This paper introduces OYXOY, a comprehensive NLP evaluation suite for Modern Greek, including novel inference datasets and cost-effective methods for resource creation, to advance Greek NLP research.
Contribution
It presents the first all-label inference dataset for Greek and a cost-efficient approach using ChatGPT to create multiple NLP evaluation tasks from dictionaries.
Findings
The inference dataset captures all possible labels, addressing ambiguity and polysemy.
Experiments show the tasks are challenging for current state-of-the-art models.
The proposed methods facilitate resource development for under-resourced languages.
Abstract
This paper serves as a foundational step towards the development of a linguistically motivated and technically relevant evaluation suite for Greek NLP. We initiate this endeavor by introducing four expert-verified evaluation tasks, specifically targeted at natural language inference, word sense disambiguation (through example comparison or sense selection) and metaphor detection. More than language-adapted replicas of existing tasks, we contribute two innovations which will resonate with the broader resource and evaluation community. Firstly, our inference dataset is the first of its kind, marking not just \textit{one}, but rather \textit{all} possible inference labels, accounting for possible shifts due to e.g. ambiguity or polysemy. Secondly, we demonstrate a cost-efficient method to obtain datasets for under-resourced languages. Using ChatGPT as a language-neutral parser, we…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Text Readability and Simplification
