LLM Augmentations to support Analytical Reasoning over Multiple Documents

TL;DR

This paper explores augmenting large language models with dynamic evidence trees to improve their ability to support complex analytical reasoning tasks in intelligence analysis, highlighting current limitations and proposing enhancements.

Contribution

It introduces a novel architecture combining LLMs with memory modules called dynamic evidence trees for multi-threaded investigative reasoning.

Findings

LLMs alone are insufficient for complex intelligence analysis tasks.

The proposed architecture improves reasoning over multiple documents.

Recommendations for enhancing LLMs in intricate analytical applications.

Abstract

Building on their demonstrated ability to perform a variety of tasks, we investigate the application of large language models (LLMs) to enhance in-depth analytical reasoning within the context of intelligence analysis. Intelligence analysts typically work with massive dossiers to draw connections between seemingly unrelated entities, and uncover adversaries' plans and motives. We explore if and how LLMs can be helpful to analysts for this task and develop an architecture to augment the capabilities of an LLM with a memory module called dynamic evidence trees (DETs) to develop and track multiple investigation threads. Through extensive experiments on multiple datasets, we highlight how LLMs, as-is, are still inadequate to support intelligence analysts and offer recommendations to improve LLMs for such intricate reasoning applications.