Zero-Shot Question Answering over Financial Documents using Large Language Models

TL;DR

This paper presents a zero-shot prompting method for large language models to perform multi-hop numerical reasoning over financial documents by generating executable Python programs, significantly improving accuracy over baselines.

Contribution

The paper introduces a novel zero-shot prompt technique that guides LLMs to produce executable programs for complex reasoning, reducing reliance on few-shot examples.

Findings

Significant accuracy improvements over baseline models

Effective zero-shot reasoning with program generation

Potential for extracting complex numerical reasoning in finance

Abstract

We introduce a large language model (LLM) based approach to answer complex questions requiring multi-hop numerical reasoning over financial reports. While LLMs have exhibited remarkable performance on various natural language and reasoning tasks, complex reasoning problems often rely on few-shot prompts that require carefully crafted examples. In contrast, our approach uses novel zero-shot prompts that guide the LLM to encode the required reasoning into a Python program or a domain specific language. The generated program is then executed by a program interpreter, thus mitigating the limitations of LLM in performing accurate arithmetic calculations. We evaluate the proposed approach on three financial datasets using some of the recently developed generative pretrained transformer (GPT) models and perform comparisons with various zero-shot baselines. The experimental results demonstrate that our approach significantly improves the accuracy for all the LLMs over their respective baselines. We provide a detailed analysis of the results, generating insights to support our findings. The success of our approach demonstrates the enormous potential to extract complex domain specific numerical reasoning by designing zero-shot prompts to effectively exploit the knowledge embedded in LLMs.