Layer Ensembles
Illia Oleksiienko, Alexandros Iosifidis

TL;DR
This paper introduces a novel layer-based ensemble method for uncertainty estimation in neural networks, offering faster inference, reduced memory usage, and improved uncertainty quality over traditional Deep Ensembles.
Contribution
The paper proposes a new layer ensemble approach with an optimized inference procedure that reuses layer outputs, enhancing efficiency and uncertainty estimation quality.
Findings
Achieves up to 19x speed up in inference
Reduces memory usage quadratically
Provides higher uncertainty quality than Deep Ensembles
Abstract
Deep Ensembles, as a type of Bayesian Neural Networks, can be used to estimate uncertainty on the prediction of multiple neural networks by collecting votes from each network and computing the difference in those predictions. In this paper, we introduce a method for uncertainty estimation that considers a set of independent categorical distributions for each layer of the network, giving many more possible samples with overlapped layers than in the regular Deep Ensembles. We further introduce an optimized inference procedure that reuses common layer outputs, achieving up to 19x speed up and reducing memory usage quadratically. We also show that the method can be further improved by ranking samples, resulting in models that require less memory and time to run while achieving higher uncertainty quality than Deep Ensembles.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Machine Learning and Data Classification · Explainable Artificial Intelligence (XAI)
MethodsDeep Ensembles · SPEED: Separable Pyramidal Pooling EncodEr-Decoder for Real-Time Monocular Depth Estimation on Low-Resource Settings
