DeFine: Decision-Making with Analogical Reasoning over Factor Profiles

TL;DR

DeFine is a modular framework that enhances decision-making in LLMs by constructing probabilistic factor profiles from complex scenarios and applying analogical reasoning to incorporate uncertainty systematically.

Contribution

It introduces a novel approach combining probabilistic factor profiles with analogical reasoning to improve decision-making under uncertainty in LLMs.

Findings

Effective in complex, uncertain scenarios like finance and consulting.

Separates uncertainty quantification from decision-making process.

Leverages past experiences to inform current decisions.

Abstract

LLMs are ideal for decision-making thanks to their ability to reason over long contexts. However, challenges arise when processing speech transcripts that describe complex scenarios, as they are verbose and include repetition, hedging, and vagueness. E.g., during a company's earnings call, an executive might project a positive revenue outlook to reassure investors, despite uncertainty regarding future earnings. It is crucial for LLMs to incorporate this uncertainty systematically when making decisions. In this paper, we introduce \textsc{DeFine}, a modular framework that constructs probabilistic factor profiles from complex scenarios. It then integrates these profiles with analogical reasoning, leveraging insights from similar past experiences to guide LLMs in making critical decisions in new situations. Our framework separates the tasks of quantifying uncertainty and incorporating it into LLM decision-making. This approach is particularly useful in areas such as consulting and financial deliberation, where making decisions under uncertainty is vital.