Scenarios and Approaches for Situated Natural Language Explanations

TL;DR

This paper introduces a benchmark dataset and evaluates how well large language models can generate context-specific natural language explanations tailored to different audience types, revealing insights into prompting techniques and model capabilities.

Contribution

The paper presents the Situation-Based Explanation dataset and analyzes the effectiveness of various prompting methods for generating tailored explanations with LLMs.

Findings

Language models can produce explanations aligned with target situations.

Explicit persona prompting is not necessary for situated explanations.

In-context learning prompts mainly help with demonstration templates, not inference.

Abstract

Large language models (LLMs) can be used to generate natural language explanations (NLE) that are adapted to different users' situations. However, there is yet to be a quantitative evaluation of the extent of such adaptation. To bridge this gap, we collect a benchmarking dataset, Situation-Based Explanation. This dataset contains 100 explanandums. Each explanandum is paired with explanations targeted at three distinct audience types-such as educators, students, and professionals-enabling us to assess how well the explanations meet the specific informational needs and contexts of these diverse groups e.g. students, teachers, and parents. For each "explanandum paired with an audience" situation, we include a human-written explanation. These allow us to compute scores that quantify how the LLMs adapt the explanations to the situations. On an array of pretrained language models with varying sizes, we examine three categories of prompting methods: rule-based prompting, meta-prompting, and in-context learning prompting. We find that 1) language models can generate prompts that result in explanations more precisely aligned with the target situations, 2) explicitly modeling an "assistant" persona by prompting "You are a helpful assistant..." is not a necessary prompt technique for situated NLE tasks, and 3) the in-context learning prompts only can help LLMs learn the demonstration template but can't improve their inference performance. SBE and our analysis facilitate future research towards generating situated natural language explanations.