Exploring new ways: Enforcing representational dissimilarity to learn new features and reduce error consistency

TL;DR

This paper introduces a novel approach to training ensemble models by promoting dissimilar intermediate representations, which reduces correlated errors and improves overall ensemble accuracy.

Contribution

It proposes using representational dissimilarity measures during training to create more diverse models with disjoint failure modes, unlike previous methods focusing on output decorrelation.

Findings

Dissimilar intermediate representations lead to less correlated predictions.

Promoting dissimilarity slightly reduces error consistency.

Ensembles with dissimilar features achieve higher accuracy.

Abstract

Independently trained machine learning models tend to learn similar features. Given an ensemble of independently trained models, this results in correlated predictions and common failure modes. Previous attempts focusing on decorrelation of output predictions or logits yielded mixed results, particularly due to their reduction in model accuracy caused by conflicting optimization objectives. In this paper, we propose the novel idea of utilizing methods of the representational similarity field to promote dissimilarity during training instead of measuring similarity of trained models. To this end, we promote intermediate representations to be dissimilar at different depths between architectures, with the goal of learning robust ensembles with disjoint failure modes. We show that highly dissimilar intermediate representations result in less correlated output predictions and slightly lower error consistency, resulting in higher ensemble accuracy. With this, we shine first light on the connection between intermediate representations and their impact on the output predictions.