Laws of Learning Dynamics and the Core of Learners

TL;DR

This paper establishes fundamental laws of learning dynamics, introduces an entropy-based lifelong ensemble method, and demonstrates its effectiveness in defending against adversarial attacks on CIFAR-10.

Contribution

It formulates core laws of learning dynamics and proposes a novel entropy-based ensemble approach for robust lifelong learning.

Findings

The entropy-based ensemble improves accuracy over naive methods.

The method shows significant gains under strong adversarial perturbations.

It effectively defends against transfer-based adversarial attacks.

Abstract

We formulate the fundamental laws governing learning dynamics, namely the conservation law and the decrease of total entropy. Within this framework, we introduce an entropy-based lifelong ensemble learning method. We evaluate its effectiveness by constructing an immunization mechanism to defend against transfer-based adversarial attacks on the CIFAR-10 dataset. Compared with a naive ensemble formed by simply averaging models specialized on clean and adversarial samples, the resulting logifold achieves higher accuracy in most test cases, with particularly large gains under strong perturbations.