CoRE: Condition-based Reasoning for Identifying Outcome Variance in Complex Events

TL;DR

This paper introduces CoRE, a reasoning framework that identifies latent conditions influencing complex event outcomes, evaluating various language models' ability to reason about these conditions under different context scenarios.

Contribution

It presents a novel approach combining dataset annotations and reasoning tasks to analyze how models identify outcome-variant conditions in complex events.

Findings

Conditions improve reasoning when context is limited

Model performance varies significantly across sizes and intent

Larger models like GPT-4o are more cautious in uncertain situations

Abstract

Knowing which latent conditions lead to a particular outcome is useful for critically examining claims made about complex event outcomes. Identifying implied conditions and examining their influence on an outcome is challenging. We handle this by combining and augmenting annotations from two existing datasets consisting of goals and states, and explore the influence of conditions through our research questions and Condition-based Reasoning tasks. We examine open and closed LLMs of varying sizes and intent-alignment on our reasoning tasks and find that conditions are useful when not all context is available. Models differ widely in their ability to generate and identify outcome-variant conditions which affects their performance on outcome validation when conditions are used to replace missing context. Larger models like GPT-4o, are more cautious in such less constrained situations.