Knowledge Consistency between Neural Networks and Beyond

TL;DR

This paper introduces a generic framework for analyzing and utilizing knowledge consistency between neural networks, providing insights into their representations and improving their performance through refinement.

Contribution

It proposes a novel, task-agnostic method to measure and analyze knowledge consistency across neural networks at different levels of fuzziness.

Findings

Knowledge consistency helps diagnose neural network representations.

It explains the success of techniques like knowledge distillation.

Refining networks using knowledge consistency boosts performance.

Abstract

This paper aims to analyze knowledge consistency between pre-trained deep neural networks. We propose a generic definition for knowledge consistency between neural networks at different fuzziness levels. A task-agnostic method is designed to disentangle feature components, which represent the consistent knowledge, from raw intermediate-layer features of each neural network. As a generic tool, our method can be broadly used for different applications. In preliminary experiments, we have used knowledge consistency as a tool to diagnose representations of neural networks. Knowledge consistency provides new insights to explain the success of existing deep-learning techniques, such as knowledge distillation and network compression. More crucially, knowledge consistency can also be used to refine pre-trained networks and boost performance.