Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model

TL;DR

This paper introduces Multi-Ticket Ensemble, finetuning diverse subnetworks from a single pretrained model to enhance ensemble performance through increased diversity among members.

Contribution

It proposes a novel ensemble method that leverages diverse subnetworks from one pretrained model, improving performance over standard ensembles.

Findings

Winning-ticket subnetworks are more diverse than dense networks.

Ensemble of subnetworks outperforms standard ensemble on certain tasks.

Diversity among ensemble members enhances overall performance.

Abstract

Ensembling is a popular method used to improve performance as a last resort. However, ensembling multiple models finetuned from a single pretrained model has been not very effective; this could be due to the lack of diversity among ensemble members. This paper proposes Multi-Ticket Ensemble, which finetunes different subnetworks of a single pretrained model and ensembles them. We empirically demonstrated that winning-ticket subnetworks produced more diverse predictions than dense networks, and their ensemble outperformed the standard ensemble on some tasks.