One Pass ImageNet

TL;DR

This paper introduces the One Pass ImageNet (OPIN) problem, exploring how deep learning models perform when trained on streaming data with limited memory, highlighting the challenges and potential solutions for resource-efficient learning.

Contribution

The paper formulates the OPIN problem, demonstrating the performance gap in single-pass training, and proposes leveraging continual learning techniques to improve accuracy under memory constraints.

Findings

Single-pass training on ImageNet causes significant accuracy drop.

Memory-efficient techniques from continual learning can mitigate performance loss.

OPIN provides a new benchmark for resource-efficient deep learning.

Abstract

We present the One Pass ImageNet (OPIN) problem, which aims to study the effectiveness of deep learning in a streaming setting. ImageNet is a widely known benchmark dataset that has helped drive and evaluate recent advancements in deep learning. Typically, deep learning methods are trained on static data that the models have random access to, using multiple passes over the dataset with a random shuffle at each epoch of training. Such data access assumption does not hold in many real-world scenarios where massive data is collected from a stream and storing and accessing all the data becomes impractical due to storage costs and privacy concerns. For OPIN, we treat the ImageNet data as arriving sequentially, and there is limited memory budget to store a small subset of the data. We observe that training a deep network in a single pass with the same training settings used for multi-epoch training results in a huge drop in prediction accuracy. We show that the performance gap can be significantly decreased by paying a small memory cost and utilizing techniques developed for continual learning, despite the fact that OPIN differs from typical continual problem settings. We propose using OPIN to study resource-efficient deep learning.