BOND: License to Train with Black-Box Functions

TL;DR

BOND is a novel perturbative method for estimating gradients of black-box functions, enabling scalable training of models with non-autodifferentiable modules and expanding the potential of hybrid digital-analog systems.

Contribution

We introduce BOND, a new adaptive bounding technique for gradient estimation that improves accuracy and scalability in black-box optimization scenarios.

Findings

BOND accurately estimates gradients for black-box functions.

Incorporating fixed modules can enhance model performance.

The method offers insights into adaptive optimizer dynamics.

Abstract

We introduce Bounded Numerical Differentiation (BOND), a perturbative method for estimating the gradients of black-box functions. BOND is distinguished by its formulation, which adaptively bounds perturbations to ensure accurate sign estimation, and by its implementation, which operates at black-box interfaces. This enables BOND to be more accurate and scalable compared to existing methods, facilitating end-to-end training of architectures that incorporate non-autodifferentiable modules. We observe that these modules, implemented in our experiments as frozen networks, can enhance model performance without increasing the number of trainable parameters. Our findings highlight the potential of leveraging fixed transformations to expand model capacity, pointing to hybrid analogue - digital devices as a path to scaling networks, and provides insights into the dynamics of adaptive optimizers.