Teaching and Critiquing Conceptualization and Operationalization in NLP

TL;DR

This paper discusses the importance of clearly defining and operationalizing abstract concepts like interpretability and bias in NLP, proposing educational approaches to improve understanding and measurement.

Contribution

It introduces a seminar framework for teaching NLP students to critically examine the conceptualization and operationalization of key abstract concepts.

Findings

Enhanced student understanding of conceptual issues in NLP

Promoted critical discussion on measurement and definitions

Encouraged interdisciplinary perspectives in NLP education

Abstract

NLP researchers regularly invoke abstract concepts like "interpretability," "bias," "reasoning," and "stereotypes," without defining them. Each subfield has a shared understanding or conceptualization of what these terms mean and how we should treat them, and this shared understanding is the basis on which operational decisions are made: Datasets are built to evaluate these concepts, metrics are proposed to quantify them, and claims are made about systems. But what do they mean, what should they mean, and how should we measure them? I outline a seminar I created for students to explore these questions of conceptualization and operationalization, with an interdisciplinary reading list and an emphasis on discussion and critique.