The Cost of Avoiding Backpropagation

TL;DR

This paper compares backpropagation, forward-mode automatic differentiation, and zero-order optimization, revealing that BP with checkpointing remains superior in accuracy, speed, and efficiency for training large models under memory constraints.

Contribution

It provides the first comprehensive theoretical and empirical comparison of these methods, highlighting the limitations of FmAD and ZO and reaffirming BP with checkpointing as the best approach.

Findings

BP with checkpointing outperforms FmAD and ZO in accuracy and speed

FAD and ZO incur higher costs in accuracy and computation

Results favor BP with checkpointing for memory-constrained training

Abstract

Forward-mode automatic differentiation (FmAD) and zero-order (ZO) optimization have been proposed as memory-efficient alternatives to backpropagation (BP) for gradient computation, especially in low-resource settings. However, their practical benefits remain unclear due to two key gaps: a lack of comparison against memory-efficient BP variants, such as activation checkpointing, and a lack of a unified theoretical analysis. This work presents a comprehensive theoretical and empirical comparison of BP, FmAD, and ZO methods. Our theoretical analysis shows that while FmAD, and ZO can reduce memory usage, they incur significant costs in accuracy, convergence speed, and computation compared to BP with checkpointing. These drawbacks worsen with larger models or constrained perturbation budgets. Empirical experiments on large language and vision-language models show that BP with checkpointing outperforms FmAD and ZO variants, including those enhanced with variance reduction, achieving up to 31.1% higher accuracy, 34.8% faster convergence, and 3.8x fewer computations at comparable memory usage. Our results highlight fundamental limitations of FmAD and ZO, and reaffirm BP with checkpointing as the most effective strategy for model training under memory-constrained settings. Our code is available at https://github.com/Astuary/The_Cost_of_Avoiding_Backpropagation.