Question Answering as Programming for Solving Time-Sensitive Questions

TL;DR

This paper introduces QAaP, a novel approach that reformulates time-sensitive question answering as programming, leveraging LLMs' coding abilities to improve answer accuracy under changing facts.

Contribution

The paper proposes a new method, QAaP, which uses LLMs to convert questions into code representations for better reasoning in time-sensitive QA tasks.

Findings

Achieved up to 14.5% improvement over strong baselines.

Demonstrated effectiveness of programming-based reformulation in time-sensitive QA.

Validated on multiple datasets with consistent performance gains.

Abstract

Question answering plays a pivotal role in human daily life because it involves our acquisition of knowledge about the world. However, due to the dynamic and ever-changing nature of real-world facts, the answer can be completely different when the time constraint in the question changes. Recently, Large Language Models (LLMs) have shown remarkable intelligence in question answering, while our experiments reveal that the aforementioned problems still pose a significant challenge to existing LLMs. This can be attributed to the LLMs' inability to perform rigorous reasoning based on surface-level text semantics. To overcome this limitation, rather than requiring LLMs to directly answer the question, we propose a novel approach where we reframe the $\textbf{Q}$uestion $\textbf{A}$nswering task $\textbf{a}$s $\textbf{P}$rogramming ($\textbf{QAaP}$). Concretely, by leveraging modern LLMs' superior capability in understanding both natural language and programming language, we endeavor to harness LLMs to represent diversely expressed text as well-structured code and select the best matching answer from multiple candidates through programming. We evaluate our QAaP framework on several time-sensitive question answering datasets and achieve decent improvement, up to $14.5$% over strong baselines. Our codes and data are available at https://github.com/TianHongZXY/qaap