Introducing Visual Scenes and Reasoning: A More Realistic Benchmark for Spoken Language Understanding

TL;DR

This paper introduces VRSLU, a new SLU dataset that incorporates visual context and explicit reasoning to improve real-world applicability, interpretability, and performance of spoken language understanding models.

Contribution

The paper presents VRSLU, a novel dataset integrating visual images and reasoning, and proposes LR-Instruct, a two-step instruction-based approach for better SLU performance and interpretability.

Findings

Visual information improves SLU accuracy.

Explicit reasoning enhances model interpretability.

Two-step instruction approach reduces reasoning bias.

Abstract

Spoken Language Understanding (SLU) consists of two sub-tasks: intent detection (ID) and slot filling (SF). Given its broad range of real-world applications, enhancing SLU for practical deployment is increasingly critical. Profile-based SLU addresses ambiguous user utterances by incorporating context awareness (CA), user profiles (UP), and knowledge graphs (KG) to support disambiguation, thereby advancing SLU research toward real-world applicability. However, existing SLU datasets still fall short in representing real-world scenarios. Specifically, (1) CA uses one-hot vectors for representation, which is overly idealized, and (2) models typically focuses solely on predicting intents and slot labels, neglecting the reasoning process that could enhance performance and interpretability. To overcome these limitations, we introduce VRSLU, a novel SLU dataset that integrates both Visual images and explicit Reasoning. For over-idealized CA, we use GPT-4o and FLUX.1-dev to generate images reflecting users' environments and statuses, followed by human verification to ensure quality. For reasoning, GPT-4o is employed to generate explanations for predicted labels, which are then refined by human annotators to ensure accuracy and coherence. Additionally, we propose an instructional template, LR-Instruct, which first predicts labels and then generates corresponding reasoning. This two-step approach helps mitigate the influence of reasoning bias on label prediction. Experimental results confirm the effectiveness of incorporating visual information and highlight the promise of explicit reasoning in advancing SLU.