Self-Similar Epochs: Value in Arrangement

TL;DR

This paper proposes self-similar arrangements of training data for matrix factorization, which preserve data structure and accelerate training by 3-37%, offering a new enhancement to stochastic gradient descent.

Contribution

It introduces a novel data arrangement method that maintains data similarity structures, improving training efficiency in matrix factorization tasks.

Findings

Training acceleration of 3-37% observed.

Self-similar arrangements preserve data structure.

Method shows promise for enhancing SGD efficiency.

Abstract

Optimization of machine learning models is commonly performed through stochastic gradient updates on randomly ordered training examples. This practice means that sub-epochs comprise of independent random samples of the training data that may not preserve informative structure present in the full data. We hypothesize that the training can be more effective with {\em self-similar} arrangements that potentially allow each epoch to provide benefits of multiple ones. We study this for "matrix factorization" -- the common task of learning metric embeddings of entities such as queries, videos, or words from example pairwise associations. We construct arrangements that preserve the weighted Jaccard similarities of rows and columns and experimentally observe training acceleration of 3\%-37\% on synthetic and recommendation datasets. Principled arrangements of training examples emerge as a novel and potentially powerful enhancement to SGD that merits further exploration.