Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systems

TL;DR

This paper explores low precision ensembling, where ensemble models are derived from a single large model using low precision number systems, showing promising improvements without additional training.

Contribution

It introduces a training-free low precision ensembling method for large models, addressing scalability issues in ensemble techniques for deep learning.

Findings

Low precision ensembling improves model performance.

The method is effective without additional training.

Scalability benefits for large models.

Abstract

While ensembling deep neural networks has shown promise in improving generalization performance, scaling current ensemble methods for large models remains challenging. Given that recent progress in deep learning is largely driven by the scale, exemplified by the widespread adoption of large-scale neural network architectures, scalability emerges an increasingly critical issue for machine learning algorithms in the era of large-scale models. In this work, we first showcase the potential of low precision ensembling, where ensemble members are derived from a single model within low precision number systems in a training-free manner. Our empirical analysis demonstrates the effectiveness of our proposed low precision ensembling method compared to existing ensemble approaches.