Distributed and Decentralised Training: Technical Governance Challenges in a Shifting AI Landscape

TL;DR

This paper explores the emerging shift from centralized to distributed and decentralized AI training, analyzing governance challenges, potential risks, and benefits to inform more precise policy development.

Contribution

It clarifies the distinctions between distributed and decentralized training, discusses their governance implications, and highlights policy considerations for these evolving AI training paradigms.

Findings

Distributed and decentralized training pose unique governance challenges.

Decentralized AI can enhance privacy and reduce power concentration.

Policy tools like export controls remain relevant despite new training paradigms.

Abstract

Advances in low-communication training algorithms are enabling a shift from centralised model training to compute setups that are either distributed across multiple clusters or decentralised via community-driven contributions. This paper distinguishes these two scenarios - distributed and decentralised training - which are little understood and often conflated in policy discourse. We discuss how they could impact technical AI governance through an increased risk of compute structuring, capability proliferation, and the erosion of detectability and shutdownability. While these trends foreshadow a possible new paradigm that could challenge key assumptions of compute governance, we emphasise that certain policy levers, like export controls, remain relevant. We also acknowledge potential benefits of decentralised AI, including privacy-preserving training runs that could unlock access to more data, and mitigating harmful power concentration. Our goal is to support more precise policymaking around compute, capability proliferation, and decentralised AI development.