Effective prevention of semantic drift as angular distance in memory-less continual deep neural networks

TL;DR

This paper introduces an angular distance-based method to evaluate semantic drift in memory-less continual deep neural networks, improving stability-plasticity balance and outperforming existing models.

Contribution

It proposes a novel angular distance measure for better node separation, enhancing continual learning performance in memory-less neural networks.

Findings

Outperforms state-of-the-art models on standard datasets

Maintains higher accuracy across tasks

Provides better separation of nodes for stability and plasticity

Abstract

Lifelong machine learning or continual learning models attempt to learn incrementally by accumulating knowledge across a sequence of tasks. Therefore, these models learn better and faster. They are used in various intelligent systems that have to interact with humans or any dynamic environment e.g., chatbots and self-driving cars. Memory-less approach is more often used with deep neural networks that accommodates incoming information from tasks within its architecture. It allows them to perform well on all the seen tasks. These models suffer from semantic drift or the plasticity-stability dilemma. The existing models use Minkowski distance measures to decide which nodes to freeze, update or duplicate. These distance metrics do not provide better separation of nodes as they are susceptible to high dimensional sparse vectors. In our proposed approach, we use angular distance to evaluate the semantic drift in individual nodes that provide better separation of nodes and thus better balancing between stability and plasticity. The proposed approach outperforms state-of-the art models by maintaining higher accuracy on standard datasets.