Semi-Supervised Online Learning on the Edge by Transforming Knowledge from Teacher Models

TL;DR

This paper introduces Knowledge Transformation, a hybrid method combining knowledge distillation, active learning, and causal reasoning, to enable online edge machine learning models to learn from unseen future data by generating pseudo-labels from teacher models.

Contribution

It proposes a novel hybrid approach, Knowledge Transformation, for online edge ML that effectively utilizes teacher models to generate pseudo-labels for unseen data, addressing a key challenge in the field.

Findings

Stable teacher models lead to optimal student performance.

KT benefits scenarios with generic teacher tasks or expensive labels.

Simulation results validate KT's effectiveness in online edge learning.

Abstract

Edge machine learning (Edge ML) enables training ML models using the vast data distributed across network edges. However, many existing approaches assume static models trained centrally and then deployed, making them ineffective against unseen data. To address this, Online Edge ML allows models to be trained directly on edge devices and updated continuously with new data. This paper explores a key challenge of Online Edge ML: "How to determine labels for truly future, unseen data points". We propose Knowledge Transformation (KT), a hybrid method combining Knowledge Distillation, Active Learning, and causal reasoning. In short, KT acts as the oracle in active learning by transforming knowledge from a teacher model to generate pseudo-labels for training a student model. To verify the validity of the method, we conducted simulation experiments with two setups: (1) using a less stable teacher model and (2) a relatively more stable teacher model. Results indicate that when a stable teacher model is given, the student model can eventually reach its expected maximum performance. KT is potentially beneficial for scenarios that meet the following circumstances: (1) when the teacher's task is generic, which means existing pre-trained models might be adequate for its task, so there will be no need to train the teacher model from scratch; and/or (2) when the label for the student's task is difficult or expensive to acquire.