Does the "most sinfully decadent cake ever" taste good? Answering Yes/No Questions from Figurative Contexts

TL;DR

This paper evaluates how well large language models understand figurative language in yes/no questions, introducing FigurativeQA dataset and showing that models like ChatGPT can improve by simplifying figurative contexts.

Contribution

The paper introduces the FigurativeQA dataset for testing QA models on figurative language and demonstrates that simplifying figurative contexts improves model performance.

Findings

Models like GPT-3 and ChatGPT perform better on literal contexts.

Simplifying figurative language into literal forms enhances QA accuracy.

ChatGPT with chain-of-thought prompting achieves the best results.

Abstract

Figurative language is commonplace in natural language, and while making communication memorable and creative, can be difficult to understand. In this work, we investigate the robustness of Question Answering (QA) models on figurative text. Yes/no questions, in particular, are a useful probe of figurative language understanding capabilities of large language models. We propose FigurativeQA, a set of 1000 yes/no questions with figurative and non-figurative contexts, extracted from the domains of restaurant and product reviews. We show that state-of-the-art BERT-based QA models exhibit an average performance drop of up to 15\% points when answering questions from figurative contexts, as compared to non-figurative ones. While models like GPT-3 and ChatGPT are better at handling figurative texts, we show that further performance gains can be achieved by automatically simplifying the figurative contexts into their non-figurative (literal) counterparts. We find that the best overall model is ChatGPT with chain-of-thought prompting to generate non-figurative contexts. Our work provides a promising direction for building more robust QA models with figurative language understanding capabilities.