Adaptive Backtracking Line Search
Joao V. Cavalcanti, Laurent Lessard, Ashia C. Wilson
TL;DR
This paper introduces an adaptive backtracking line search method that dynamically adjusts step-sizes based on violation degree, leading to faster optimization without extra computational cost, supported by theoretical guarantees and empirical results.
Contribution
The paper presents a novel adaptive adjustment rule for backtracking line search that improves speed while maintaining theoretical convergence guarantees.
Findings
Significantly faster optimization on real datasets.
No additional computational burden compared to regular backtracking.
Maintains convergence guarantees for convex and nonconvex problems.
Abstract
Backtracking line search is foundational in numerical optimization. The basic idea is to adjust the step-size of an algorithm by a constant factor until some chosen criterion (e.g. Armijo, Descent Lemma) is satisfied. We propose a novel way to adjust step-sizes, replacing the constant factor used in regular backtracking with one that takes into account the degree to which the chosen criterion is violated, with no additional computational burden. This light-weight adjustment leads to significantly faster optimization, which we confirm by performing a variety of experiments on over fifteen real world datasets. For convex problems, we prove adaptive backtracking requires no more adjustments to produce a feasible step-size than regular backtracking does. For nonconvex smooth problems, we prove adaptive backtracking enjoys the same guarantees of regular backtracking. Furthermore, we prove…
Peer Reviews
Decision·ICLR 2025 Poster
Backtracking line search is a fundamental task in general optimization algorithms. The numerical simulations show promising potential for the adaptive method.
As my work primarily focuses on the theoretical aspects of optimization, my comments and concerns mainly relate to the theoretical analysis of the proposed adaptive approach. Broadly speaking, the theorems presented in Section 4 provide limited insight into the advantages of the new approach compared to the conventional backtracking line search in Algorithm 1. I will elaborate on this and discuss other concerns below. * My main technical concern is with the lower bound of the adaptive factor, w
- Usually papers on backtracking line-search tend to focus on effects of non-monotonic searches or on changing the condition of the line-search. This paper proposes a seemingly simple modification to the choice of the backtracking factor that preserves most of the theoretical guarantees. This seems like one of those simple observations that is of interest to the optimization community in general. In fact, I think the contribution is good exactly because of the simplicity: there seems to be a eas
I believe the main area of improvement of this paper is its discussion previous work, which is even more important in a paper about such a classical topic. I will detail two aspects of the paper that I believe could be improved. - **Discussion of related research work**: Backtracking line-search is a classical topic with a vast amount of work on it (even if most of it is not necessarily on the backtracking factor). Yet, besides references to classical books and the original works on BLS, there
The paper is very nicely written, the intuitition behind the proposed adaptive strategy is well described and the numerical tests are convincing. I liked the choice of the authors of presenting first the numerical results raather than the theoretical ones, since the reader is really triggered to keep reading and understanding more. The exhaustive list of experiments reported in the appendix is very convincing and shows that the proposed strategy performs indeed very well, as supported by the tho
The only "weaknesses" I could highlight relate to some (minor) lack of referencing to existing work and/or to some refinement that could be made to guarantee that the proposed strategy maintains (as it seems) the convergence speed of stnadard algorithm also in more 'regular' scenario (e.g., strong convexity). Some minor imprecisions can be easily fixed as I will suggest below. Upon the modifications suggested (at all, or even partially), I think I would be happy to change my note to 10 as I thi
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsRobotic Mechanisms and Dynamics · Parallel Computing and Optimization Techniques