Stress and Adaptation: Applying Anna Karenina Principle in Deep Learning for Image Classification

TL;DR

This paper introduces an Anna Karenina Principle for deep learning, showing that more generalizable models have similar internal representations, while less generalizable ones vary more, supported by theoretical and experimental evidence.

Contribution

It proposes a novel AKP framework for deep neural networks, linking model generalizability to internal representation similarity, with theoretical proof and experimental validation.

Findings

Generalizable models have similar internal representations.

Perturbations cause less generalizable models to vary more.

Theoretical proof supports the similarity of happy models.

Abstract

Image classification with deep neural networks has reached state-of-art with high accuracy. This success is attributed to good internal representation features that bypasses the difficulties of the non-convex optimization problems. We have little understanding of these internal representations, let alone quantifying them. Recent research efforts have focused on alternative theories and explanations of the generalizability of these deep networks. We propose the alternative perturbation of deep models during their training induces changes that lead to transitions to different families. The result is an Anna Karenina Principle AKP for deep learning, in which less generalizable models unhappy families vary more in their representation than more generalizable models happy families paralleling Leo Tolstoy dictum that all happy families look alike, each unhappy family is unhappy in its own way. Anna Karenina principle has been found in systems in a wide range: from the surface of endangered corals exposed to harsh weather to the lungs of patients suffering from fatal diseases of AIDs. In our paper, we have generated artificial perturbations to our model by hot-swapping the activation and loss functions during the training. In this paper, we build a model to classify cancer cells from non-cancer ones. We give theoretical proof that the internal representations of generalizable happy models are similar in the asymptotic limit. Our experiments verify similar representations of generalizable models.