iREL at SemEval-2024 Task 9: Improving Conventional Prompting Methods for Brain Teasers

TL;DR

This paper presents a novel prompting strategy to enhance pre-trained language models' ability to solve brain teasers that require unconventional thinking, outperforming baselines but still lagging behind humans.

Contribution

It introduces static and dynamic few-shot prompting techniques along with a reasoning strategy to improve model performance on lateral thinking tasks.

Findings

Significant performance improvements over baseline models

Model approaches human-level performance but does not surpass it

Proves the effectiveness of prompting strategies for unconventional reasoning

Abstract

This paper describes our approach for SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense. The BRAINTEASER task comprises multiple-choice Question Answering designed to evaluate the models' lateral thinking capabilities. It consists of Sentence Puzzle and Word Puzzle subtasks that require models to defy default common-sense associations and exhibit unconventional thinking. We propose a unique strategy to improve the performance of pre-trained language models, notably the Gemini 1.0 Pro Model, in both subtasks. We employ static and dynamic few-shot prompting techniques and introduce a model-generated reasoning strategy that utilizes the LLM's reasoning capabilities to improve performance. Our approach demonstrated significant improvements, showing that it performed better than the baseline models by a considerable margin but fell short of performing as well as the human annotators, thus highlighting the efficacy of the proposed strategies.