Towards Ecologically Valid Research on Language User Interfaces

TL;DR

This paper discusses the importance of ecological validity in research on Language User Interfaces (LUIs), criticizing current benchmarks for lacking real-world relevance and proposing guidelines to improve research practices.

Contribution

It identifies five common deviations in existing benchmarks from ideal methodologies and offers recommendations to enhance ecological validity in LUI research.

Findings

Current benchmarks often lack real-world relevance.

Deviations from ideal methodologies impact research applicability.

Recommendations aim to improve ecological validity.

Abstract

Language User Interfaces (LUIs) could improve human-machine interaction for a wide variety of tasks, such as playing music, getting insights from databases, or instructing domestic robots. In contrast to traditional hand-crafted approaches, recent work attempts to build LUIs in a data-driven way using modern deep learning methods. To satisfy the data needs of such learning algorithms, researchers have constructed benchmarks that emphasize the quantity of collected data at the cost of its naturalness and relevance to real-world LUI use cases. As a consequence, research findings on such benchmarks might not be relevant for developing practical LUIs. The goal of this paper is to bootstrap the discussion around this issue, which we refer to as the benchmarks' low ecological validity. To this end, we describe what we deem an ideal methodology for machine learning research on LUIs and categorize five common ways in which recent benchmarks deviate from it. We give concrete examples of the five kinds of deviations and their consequences. Lastly, we offer a number of recommendations as to how to increase the ecological validity of machine learning research on LUIs.