Mothman at SemEval-2024 Task 9: An Iterative System for Chain-of-Thought Prompt Optimization

TL;DR

This paper presents an iterative prompt optimization system for large language models to enhance performance on lateral thinking tasks, demonstrated through the BrainTeaser shared task with improved results.

Contribution

It introduces a novel human-in-the-loop prompt engineering method that iteratively refines prompts for better model performance on creative, logic-based tasks.

Findings

Significant performance improvements on BrainTeaser dataset

Effective prompt optimization via human evaluation

Enhanced model ability to handle adversarial, lateral thinking tasks

Abstract

Extensive research exists on the performance of large language models on logic-based tasks, whereas relatively little has been done on their ability to generate creative solutions on lateral thinking tasks. The BrainTeaser shared task tests lateral thinking and uses adversarial datasets to prevent memorization, resulting in poor performance for out-of-the-box models. We propose a system for iterative, chain-of-thought prompt engineering which optimizes prompts using human evaluation. Using this shared task, we demonstrate our system's ability to significantly improve model performance by optimizing prompts and evaluate the input dataset.