Risks and NLP Design: A Case Study on Procedural Document QA

TL;DR

This paper examines risks in NLP applications through a case study on recipe question answering, highlighting potential harms and proposing risk-aware analysis to improve safety and performance.

Contribution

It introduces a risk-focused error analysis framework applied to procedural document QA, demonstrating how to identify and mitigate specific user harms in NLP systems.

Findings

Language model answers recipes as well as humans in zero-shot mode

Risk-oriented error analysis reveals potential safety issues

Framework informs safer NLP system design

Abstract

As NLP systems are increasingly deployed at scale, concerns about their potential negative impacts have attracted the attention of the research community, yet discussions of risk have mostly been at an abstract level and focused on generic AI or NLP applications. We argue that clearer assessments of risks and harms to users--and concrete strategies to mitigate them--will be possible when we specialize the analysis to more concrete applications and their plausible users. As an illustration, this paper is grounded in cooking recipe procedural document question answering (ProcDocQA), where there are well-defined risks to users such as injuries or allergic reactions. Our case study shows that an existing language model, applied in "zero-shot" mode, quantitatively answers real-world questions about recipes as well or better than the humans who have answered the questions on the web. Using a novel questionnaire informed by theoretical work on AI risk, we conduct a risk-oriented error analysis that could then inform the design of a future system to be deployed with lower risk of harm and better performance.