RISCORE: Enhancing In-Context Riddle Solving in Language Models through Context-Reconstructed Example Augmentation

TL;DR

This paper introduces RISCORE, a novel automated prompting method that enhances language models' riddle-solving abilities by reconstructing contextually relevant examples, leading to significant performance improvements in reasoning tasks.

Contribution

The paper presents RISCORE, a new automated prompting technique that improves in-context reasoning for riddles by generating contextually reconstructed examples, outperforming traditional methods.

Findings

RISCORE improves LLM performance on reasoning riddles

Reconstructed examples enhance model understanding and reasoning

Method surpasses traditional exemplar selection strategies

Abstract

Riddle-solving requires advanced reasoning skills, pushing LLMs to engage in abstract thinking and creative problem-solving, often revealing limitations in their cognitive abilities. In this paper, we examine the riddle-solving capabilities of LLMs using a multiple-choice format, exploring how different prompting techniques impact performance on riddles that demand diverse reasoning skills. To enhance results, we introduce RISCORE (RIddle Solving with COntext REcontruciton) a novel fully automated prompting method that generates and utilizes contextually reconstructed sentence-based puzzles in conjunction with the original examples to create few-shot exemplars. Our experiments demonstrate that RISCORE significantly improves the performance of language models in both vertical and lateral thinking tasks, surpassing traditional exemplar selection strategies across a variety of few-shot settings.