Online Deep Learning: Learning Deep Neural Networks on the Fly

TL;DR

This paper introduces a novel online deep learning framework that enables deep neural networks to learn incrementally from streaming data, addressing the challenges of non-convex optimization and model adaptation in real-time scenarios.

Contribution

It proposes a new Hedge Backpropagation method for effective online updating of deep neural networks with adaptive depth.

Findings

HBP effectively updates DNN parameters in online settings

The framework performs well on large-scale datasets with concept drift

It demonstrates robustness in both stationary and drifting data scenarios

Abstract

Deep Neural Networks (DNNs) are typically trained by backpropagation in a batch learning setting, which requires the entire training data to be made available prior to the learning task. This is not scalable for many real-world scenarios where new data arrives sequentially in a stream form. We aim to address an open challenge of "Online Deep Learning" (ODL) for learning DNNs on the fly in an online setting. Unlike traditional online learning that often optimizes some convex objective function with respect to a shallow model (e.g., a linear/kernel-based hypothesis), ODL is significantly more challenging since the optimization of the DNN objective function is non-convex, and regular backpropagation does not work well in practice, especially for online learning settings. In this paper, we present a new online deep learning framework that attempts to tackle the challenges by learning DNN models of adaptive depth from a sequence of training data in an online learning setting. In particular, we propose a novel Hedge Backpropagation (HBP) method for online updating the parameters of DNN effectively, and validate the efficacy of our method on large-scale data sets, including both stationary and concept drifting scenarios.