GPT-3 Models are Few-Shot Financial Reasoners

TL;DR

This paper evaluates GPT-3's ability to perform financial reasoning with few-shot learning, emphasizing the importance of retrieval and logic components for accurate financial question answering.

Contribution

It demonstrates that GPT-3, with refined prompt engineering, can approach state-of-the-art performance in financial reasoning without fine-tuning, highlighting the continued need for retrieval and logic modules.

Findings

GPT-3 achieves near SOTA accuracy with prompt engineering

Retrieval and logic modules remain essential for financial QA

Fine-tuning is not necessary for strong performance in this task

Abstract

Financial analysis is an important tool for evaluating company performance. Practitioners work to answer financial questions to make profitable investment decisions, and use advanced quantitative analyses to do so. As a result, Financial Question Answering (QA) is a question answering task that requires deep reasoning about numbers. Furthermore, it is unknown how well pre-trained language models can reason in the financial domain. The current state-of-the-art requires a retriever to collect relevant facts about the financial question from the text and a generator to produce a valid financial program and a final answer. However, recently large language models like GPT-3 have achieved state-of-the-art performance on wide variety of tasks with just a few shot examples. We run several experiments with GPT-3 and find that a separate retrieval model and logic engine continue to be essential components to achieving SOTA performance in this task, particularly due to the precise nature of financial questions and the complex information stored in financial documents. With this understanding, our refined prompt-engineering approach on GPT-3 achieves near SOTA accuracy without any fine-tuning.