# Towards Better Generalization: BP-SVRG in Training Deep Neural Networks

## TL;DR

This paper explores how training techniques like mini-batching and learning rate decay affect the generalization of SVRG in deep neural networks, proposing BP-SVRG which outperforms B-SVRG and SGD in some cases.

## Contribution

It introduces BatchPlus-SVRG (BP-SVRG), a novel variant of SVRG that improves generalization by leveraging insights from gradient norms and landscape flatness.

## Key findings

- BP-SVRG outperforms B-SVRG in generalization.
- BP-SVRG can outperform SGD in certain deep learning scenarios.
- Gradient norm metrics correlate with model generalization.

## Abstract

Stochastic variance-reduced gradient (SVRG) is a classical optimization method. Although it is theoretically proved to have better convergence performance than stochastic gradient descent (SGD), the generalization performance of SVRG remains open. In this paper we investigate the effects of some training techniques, mini-batching and learning rate decay, on the generalization performance of SVRG, and verify the generalization performance of Batch-SVRG (B-SVRG). In terms of the relationship between optimization and generalization, we believe that the average norm of gradients on each training sample as well as the norm of average gradient indicate how flat the landscape is and how well the model generalizes. Based on empirical observations of such metrics, we perform a sign switch on B-SVRG and derive a practical algorithm, BatchPlus-SVRG (BP-SVRG), which is numerically shown to enjoy better generalization performance than B-SVRG, even SGD in some scenarios of deep neural networks.