Token-free Models for Sarcasm Detection

TL;DR

This paper demonstrates that token-free models like ByT5 and CANINE outperform token-based models in sarcasm detection tasks across social media and news headlines, achieving new state-of-the-art accuracy.

Contribution

It provides the first comprehensive evaluation of token-free models for sarcasm detection, showing their superiority over traditional token-based approaches.

Findings

ByT5-small and CANINE outperform token-based models in accuracy.

Token-free models achieve new state-of-the-art results.

Token-free models are more robust in noisy, informal domains.

Abstract

Tokenization is a foundational step in most natural language processing (NLP) pipelines, yet it introduces challenges such as vocabulary mismatch and out-of-vocabulary issues. Recent work has shown that models operating directly on raw text at the byte or character level can mitigate these limitations. In this paper, we evaluate two token-free models, ByT5 and CANINE, on the task of sarcasm detection in both social media (Twitter) and non-social media (news headlines) domains. We fine-tune and benchmark these models against token-based baselines and state-of-the-art approaches. Our results show that ByT5-small and CANINE outperform token-based counterparts and achieve new state-of-the-art performance, improving accuracy by 0.77% and 0.49% on the News Headlines and Twitter Sarcasm datasets, respectively. These findings underscore the potential of token-free models for robust NLP in noisy and informal domains such as social media.