Should attention be all we need? The epistemic and ethical implications of unification in machine learning

TL;DR

This paper critically examines the widespread adoption of attention-based models like transformers in machine learning, highlighting epistemic and ethical implications of their unification across diverse domains.

Contribution

It provides a nuanced analysis of the epistemic and ethical risks associated with the dominance of attention mechanisms in machine learning, challenging assumptions of their universal applicability.

Findings

Unification may limit methodological diversity and increase black-boxing.

Transformers' success does not necessarily transfer across all domains.

Centralization of power raises ethical concerns about marginalization.

Abstract

"Attention is all you need" has become a fundamental precept in machine learning research. Originally designed for machine translation, transformers and the attention mechanisms that underpin them now find success across many problem domains. With the apparent domain-agnostic success of transformers, many researchers are excited that similar model architectures can be successfully deployed across diverse applications in vision, language and beyond. We consider the benefits and risks of these waves of unification on both epistemic and ethical fronts. On the epistemic side, we argue that many of the arguments in favor of unification in the natural sciences fail to transfer over to the machine learning case, or transfer over only under assumptions that might not hold. Unification also introduces epistemic risks related to portability, path dependency, methodological diversity, and increased black-boxing. On the ethical side, we discuss risks emerging from epistemic concerns, further marginalizing underrepresented perspectives, the centralization of power, and having fewer models across more domains of application