# Improving Model Training by Periodic Sampling over Weight Distributions

## TL;DR

This paper introduces a novel periodic sampling technique for model weights that enhances convergence speed, robustness, and performance across various vision tasks with minimal additional computation.

## Contribution

The paper presents a new weight sampling method that improves convergence and robustness, independent of model type or optimization algorithm, applicable to multiple vision tasks.

## Key findings

- Faster convergence with improved robustness.
- Performance improvements are approximately monotonic.
- Method is effective across different vision problems.

## Abstract

In this paper, we explore techniques centered around periodic sampling of model weights that provide convergence improvements on gradient update methods (vanilla \acs{SGD}, Momentum, Adam) for a variety of vision problems (classification, detection, segmentation). Importantly, our algorithms provide better, faster and more robust convergence and training performance with only a slight increase in computation time. Our techniques are independent of the neural network model, gradient optimization methods or existing optimal training policies and converge in a less volatile fashion with performance improvements that are approximately monotonic. We conduct a variety of experiments to quantify these improvements and identify scenarios where these techniques could be more useful.