# Direct Nonlinear Acceleration

## TL;DR

This paper introduces direct nonlinear acceleration (DNA), a new method that improves optimization speed by directly minimizing the function value at extrapolated points, outperforming previous RNA methods in various datasets.

## Contribution

The paper proposes a novel direct nonlinear acceleration technique that minimizes the function value at extrapolated points, offering better performance than existing RNA methods.

## Key findings

- DNA significantly outperforms RNA on synthetic datasets
- DNA accelerates neural network training effectively
- Computational cost of DNA is comparable to RNA

## Abstract

Optimization acceleration techniques such as momentum play a key role in state-of-the-art machine learning algorithms. Recently, generic vector sequence extrapolation techniques, such as regularized nonlinear acceleration (RNA) of Scieur et al., were proposed and shown to accelerate fixed point iterations. In contrast to RNA which computes extrapolation coefficients by (approximately) setting the gradient of the objective function to zero at the extrapolated point, we propose a more direct approach, which we call direct nonlinear acceleration (DNA). In DNA, we aim to minimize (an approximation of) the function value at the extrapolated point instead. We adopt a regularized approach with regularizers designed to prevent the model from entering a region in which the functional approximation is less precise. While the computational cost of DNA is comparable to that of RNA, our direct approach significantly outperforms RNA on both synthetic and real-world datasets. While the focus of this paper is on convex problems, we obtain very encouraging results in accelerating the training of neural networks.