A narrowing of AI research?

TL;DR

This paper analyzes the evolution of AI research diversity using semantic analysis of arXiv data, revealing recent stagnation and a private sector focus on deep learning at the expense of broader societal and methodological diversity.

Contribution

It provides empirical evidence of diversity stagnation in AI research and highlights the influence of private sector dominance and specialization.

Findings

Research diversity has stagnated in recent years.

Private sector AI research is less diverse and more influential.

Private sector focuses on deep learning, neglecting societal and ethical aspects.

Abstract

The arrival of deep learning techniques able to infer patterns from large datasets has dramatically improved the performance of Artificial Intelligence (AI) systems. Deep learning's rapid development and adoption, in great part led by large technology companies, has however created concerns about a premature narrowing in the technological trajectory of AI research despite its weaknesses, which include lack of robustness, high environmental costs, and potentially unfair outcomes. We seek to improve the evidence base with a semantic analysis of AI research in arXiv, a popular pre-prints database. We study the evolution of the thematic diversity of AI research, compare the thematic diversity of AI research in academia and the private sector and measure the influence of private companies in AI research through the citations they receive and their collaborations with other institutions. Our results suggest that diversity in AI research has stagnated in recent years, and that AI research involving the private sector tends to be less diverse and more influential than research in academia. We also find that private sector AI researchers tend to specialise in data-hungry and computationally intensive deep learning methods at the expense of research involving other AI methods, research that considers the societal and ethical implications of AI, and applications in sectors like health. Our results provide a rationale for policy action to prevent a premature narrowing of AI research that could constrain its societal benefits, but we note the informational, incentive and scale hurdles standing in the way of such interventions.