Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?

TL;DR

This paper investigates reverse question answering (RQA) by testing large language models' ability to generate questions from answers, revealing insights into their reasoning, difficulty, and potential for improving LLM capabilities.

Contribution

It jointly evaluates QA and RQA in LLMs, compares their difficulty across answer types, and identifies challenges in generating valid multi-hop questions, advancing understanding of LLM reasoning.

Findings

LLMs are less accurate in RQA for numerical answers

LLMs are slightly more accurate in RQA for textual answers

RQA errors correlate with question difficulty and answer frequency

Abstract

Question answering (QA), giving correct answers to questions, is a popular task, but we test reverse question answering (RQA): for an input answer, give a question with that answer. Past work tests QA and RQA separately, but we test them jointly, comparing their difficulty, aiding benchmark design, and checking reasoning consistency. We run 16 LLMs on QA and RQA with trivia questions/answers, revealing: 1) Versus QA, LLMs are much less accurate in RQA for numerical answers, but slightly more accurate in RQA for textual answers; 2) LLMs often answer their own invalid questions from RQA accurately in QA, so RQA errors are not from knowledge gaps alone; 3) RQA errors correlate with question difficulty and inversely correlate with answer frequencies in the Dolma corpus; and 4) LLMs struggle to provide valid multi-hop questions. By finding question and answer types that lead to RQA errors, we suggest improvements for LLM reasoning.