Limpeh ga li gong: Challenges in Singlish Annotations

TL;DR

This paper explores the challenges of POS tagging in Singlish, a colloquial language from Singapore, highlighting the difficulties in annotation and the limited accuracy of current automatic taggers, thereby paving the way for future research.

Contribution

It introduces a new Singlish dataset with human-annotated POS tags and analyzes the limitations of existing automatic tagging methods on this language.

Findings

Automatic taggers achieve only ~80% accuracy on Singlish.

Singlish's unique features cause annotation and tagging challenges.

The dataset reveals significant variability and complexity in Singlish language use.

Abstract

Singlish, or Colloquial Singapore English, is a language formed from oral and social communication within multicultural Singapore. In this work, we work on a fundamental Natural Language Processing (NLP) task: Parts-Of-Speech (POS) tagging of Singlish sentences. For our analysis, we build a parallel Singlish dataset containing direct English translations and POS tags, with translation and POS annotation done by native Singlish speakers. Our experiments show that automatic transition- and transformer- based taggers perform with only $\sim 80\%$ accuracy when evaluated against human-annotated POS labels, suggesting that there is indeed room for improvement on computation analysis of the language. We provide an exposition of challenges in Singlish annotation: its inconsistencies in form and semantics, the highly context-dependent particles of the language, its structural unique expressions, and the variation of the language on different mediums. Our task definition, resultant labels and results reflects the challenges in analysing colloquial languages formulated from a variety of dialects, and paves the way for future studies beyond POS tagging.