Lost in Translation: The Algorithmic Gap Between LMs and the Brain

TL;DR

This paper explores the differences and similarities between language models and the human brain, emphasizing the need for biologically inspired features and internal process analysis to improve AI and cognitive understanding.

Contribution

It highlights the importance of examining internal processes and neuroscience insights to develop more brain-like language models, bridging the gap between AI and human cognition.

Findings

Insights from neuroscience can inform model design.

Scaling laws relate to cognitive plausibility.

Internal process analysis reveals key differences.

Abstract

Language Models (LMs) have achieved impressive performance on various linguistic tasks, but their relationship to human language processing in the brain remains unclear. This paper examines the gaps and overlaps between LMs and the brain at different levels of analysis, emphasizing the importance of looking beyond input-output behavior to examine and compare the internal processes of these systems. We discuss how insights from neuroscience, such as sparsity, modularity, internal states, and interactive learning, can inform the development of more biologically plausible language models. Furthermore, we explore the role of scaling laws in bridging the gap between LMs and human cognition, highlighting the need for efficiency constraints analogous to those in biological systems. By developing LMs that more closely mimic brain function, we aim to advance both artificial intelligence and our understanding of human cognition.