Conceptual Engineering Using Large Language Models

TL;DR

This paper introduces a method that uses large language models to perform classification-based conceptual engineering, evaluated with data from Wikidata on definitions like PLANET and WOMAN, showing promising performance and interpretability.

Contribution

It presents a novel approach to conceptual engineering using LLMs for classification and rationale generation, advancing empirical methods in the field.

Findings

Classification procedures show good performance.

Generated rationales help identify issues in definitions.

Method offers insights into conceptual engineering targets.

Abstract

We describe a method, based on Jennifer Nado's proposal for classification procedures as targets of conceptual engineering, that implements such procedures by prompting a large language model. We apply this method, using data from the Wikidata knowledge graph, to evaluate stipulative definitions related to two paradigmatic conceptual engineering projects: the International Astronomical Union's redefinition of PLANET and Haslanger's ameliorative analysis of WOMAN. Our results show that classification procedures built using our approach can exhibit good classification performance and, through the generation of rationales for their classifications, can contribute to the identification of issues in either the definitions or the data against which they are being evaluated. We consider objections to this method, and discuss implications of this work for three aspects of theory and practice of conceptual engineering: the definition of its targets, empirical methods for their investigation, and their practical roles. The data and code used for our experiments, together with the experimental results, are available in a Github repository.