Defining and Evaluating Fair Natural Language Generation

TL;DR

This paper introduces a fairness framework for natural language generation, evaluates gender biases in leading models, and provides both theoretical and empirical insights into bias embedding in NLG systems.

Contribution

It presents a novel fairness framework for NLG and empirically assesses gender bias in state-of-the-art language models.

Findings

Existing models embed gender biases

Theoretical formulation of biases in NLG

Empirical evidence of bias in language models

Abstract

Our work focuses on the biases that emerge in the natural language generation (NLG) task of sentence completion. In this paper, we introduce a framework of fairness for NLG followed by an evaluation of gender biases in two state-of-the-art language models. Our analysis provides a theoretical formulation for biases in NLG and empirical evidence that existing language generation models embed gender bias.