PyGlove: Efficiently Exchanging ML Ideas as Code

TL;DR

PyGlove introduces a symbolic, patch-based approach for sharing machine learning ideas efficiently across teams, significantly reducing code complexity and fostering rapid adoption of innovations.

Contribution

The paper extends PyGlove to enable scalable, rule-based sharing of ML ideas, facilitating network effects and reducing development effort.

Findings

Achieved 80% reduction in code size in a large codebase

Enabled quick sharing and adoption of ML ideas among teams

Demonstrated effectiveness of symbolic patches in organizing ML development

Abstract

The increasing complexity and scale of machine learning (ML) has led to the need for more efficient collaboration among multiple teams. For example, when a research team invents a new architecture like "ResNet," it is desirable for multiple engineering teams to adopt it. However, the effort required for each team to study and understand the invention does not scale well with the number of teams or inventions. In this paper, we present an extension of our PyGlove library to easily and scalably share ML ideas. PyGlove represents ideas as symbolic rule-based patches, enabling researchers to write down the rules for models they have not seen. For example, an inventor can write rules that will "add skip-connections." This permits a network effect among teams: at once, any team can issue patches to all other teams. Such a network effect allows users to quickly surmount the cost of adopting PyGlove by writing less code quicker, providing a benefit that scales with time. We describe the new paradigm of organizing ML through symbolic patches and compare it to existing approaches. We also perform a case study of a large codebase where PyGlove led to an 80% reduction in the number of lines of code.