Few-shot Policy (de)composition in Conversational Question Answering

TL;DR

This paper introduces LDPC, a neuro-symbolic framework using large language models for policy compliance detection in conversations, achieving competitive results with minimal data and enhancing transparency and interpretability.

Contribution

The work presents a novel logical decomposition approach for policy compliance detection that leverages LLMs in a few-shot setting, improving reasoning transparency without task-specific fine-tuning.

Findings

Competitive performance on ShARC benchmark

Enhanced transparency and explainability in policy reasoning

Identified dataset ambiguities and reasoning challenges

Abstract

The task of policy compliance detection (PCD) is to determine if a scenario is in compliance with respect to a set of written policies. In a conversational setting, the results of PCD can indicate if clarifying questions must be asked to determine compliance status. Existing approaches usually claim to have reasoning capabilities that are latent or require a large amount of annotated data. In this work, we propose logical decomposition for policy compliance (LDPC): a neuro-symbolic framework to detect policy compliance using large language models (LLMs) in a few-shot setting. By selecting only a few exemplars alongside recently developed prompting techniques, we demonstrate that our approach soundly reasons about policy compliance conversations by extracting sub-questions to be answered, assigning truth values from contextual information, and explicitly producing a set of logic statements from the given policies. The formulation of explicit logic graphs can in turn help answer PCDrelated questions with increased transparency and explainability. We apply this approach to the popular PCD and conversational machine reading benchmark, ShARC, and show competitive performance with no task-specific finetuning. We also leverage the inherently interpretable architecture of LDPC to understand where errors occur, revealing ambiguities in the ShARC dataset and highlighting the challenges involved with reasoning for conversational question answering.