One Pass ImageNet
Huiyi Hu, Ang Li, Daniele Calandriello, Dilan Gorur

TL;DR
This paper introduces the One Pass ImageNet (OPIN) problem, exploring how deep learning models perform when trained on streaming data with limited memory, highlighting the challenges and potential solutions for resource-efficient learning.
Contribution
The paper formulates the OPIN problem, demonstrating the performance gap in single-pass training, and proposes leveraging continual learning techniques to improve accuracy under memory constraints.
Findings
Single-pass training on ImageNet causes significant accuracy drop.
Memory-efficient techniques from continual learning can mitigate performance loss.
OPIN provides a new benchmark for resource-efficient deep learning.
Abstract
We present the One Pass ImageNet (OPIN) problem, which aims to study the effectiveness of deep learning in a streaming setting. ImageNet is a widely known benchmark dataset that has helped drive and evaluate recent advancements in deep learning. Typically, deep learning methods are trained on static data that the models have random access to, using multiple passes over the dataset with a random shuffle at each epoch of training. Such data access assumption does not hold in many real-world scenarios where massive data is collected from a stream and storing and accessing all the data becomes impractical due to storage costs and privacy concerns. For OPIN, we treat the ImageNet data as arriving sequentially, and there is limited memory budget to store a small subset of the data. We observe that training a deep network in a single pass with the same training settings used for multi-epoch…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Adversarial Robustness in Machine Learning · Anomaly Detection Techniques and Applications
