Augmenting Lateral Thinking in Language Models with Humor and Riddle Data for the BRAINTEASER Task

TL;DR

This paper explores enhancing language models' lateral thinking abilities in the BRAINTEASER task by augmenting training data with humor and riddle datasets, leading to significant performance improvements.

Contribution

The study introduces a novel data augmentation approach using humor and riddles to improve lateral reasoning in language models for the BRAINTEASER task.

Findings

Achieved 92.5% accuracy on Sentence Puzzle subtask.

Framing as multiple-choice improves accuracy by 10%.

Data augmentation benefits sentence-level reasoning more.

Abstract

The SemEval 2024 BRAINTEASER task challenges language models to perform lateral thinking -- a form of creative, non-linear reasoning that remains underexplored in NLP. The task comprises two subtasks, Sentence Puzzle and Word Puzzle, requiring models to defy conventional commonsense associations. We present a system that fine-tunes DeBERTaV3 using HuggingFace's AutoModelForMultipleChoice architecture. We augment the provided training data with two additional sources: (1) a humor-style question-answering dataset generated via GPT-4 prompting, and (2) the RiddleSense dataset. This data augmentation strategy is motivated by the observation that humor and riddles share the lateral reasoning structure required by the task. Our best system achieves 92.5\% overall accuracy on the Sentence Puzzle subtask and 80.2\% on the Word Puzzle subtask, ranking 6th out of 31 teams and 10th out of 23 teams, respectively. We further show that the choice of task formulation matters: framing the problem as multiple-choice rather than sequence classification yields a 10-point accuracy improvement with the same base model. Our analysis reveals that data augmentation with humor and riddle data is particularly effective for sentence-level lateral reasoning, while word-level puzzles remain a harder challenge.