Functional or imperative? On pleasant semantics for differentiable programming languages

TL;DR

This paper investigates the semantic differences between functional and imperative programming styles in machine learning, aiming to reconcile their use and improve language design for better optimization and usability.

Contribution

It analyzes the reasons behind the preference for imperative and functional styles and proposes ways to integrate their advantages in future language designs.

Findings

Imperative languages offer flexibility and ease of use for practitioners.

Functional languages provide better optimization and semantic clarity.

A hybrid approach could combine benefits of both styles.

Abstract

In machine learning (ML), researchers and engineers seem to be at odds. System implementers would prefer models to be declarative, with detailed type information and semantic restrictions that allow models to be optimised, rearranged and parallelised. Yet practitioners show an overwhelming preference for dynamic, imperative languages with mutable state, and much engineering effort is spent bridging the resulting semantic divide. Is there a fundamental conflict? This article explores why imperative and functional styles are used, and how future language designs might get the best of both worlds.