CODA: Constructivism Learning for Instance-Dependent Dropout Architecture Construction

TL;DR

This paper introduces CODA, a novel instance-dependent dropout architecture inspired by constructivism learning, which adaptively constructs dropout structures based on individual data instances, showing improved performance over existing methods.

Contribution

The paper proposes a new instance-dependent dropout method called CODA, utilizing Bayesian nonparametric models to differentiate dropout architectures per data instance, addressing a key limitation of prior approaches.

Findings

CODA outperforms state-of-the-art dropout techniques on five real-world datasets.

The method effectively differentiates dropout architectures for individual data instances.

Experimental results confirm the effectiveness of the proposed approach.

Abstract

Dropout is attracting intensive research interest in deep learning as an efficient approach to prevent overfitting. Recently incorporating structural information when deciding which units to drop out produced promising results comparing to methods that ignore the structural information. However, a major issue of the existing work is that it failed to differentiate among instances when constructing the dropout architecture. This can be a significant deficiency for many applications. To solve this issue, we propose Constructivism learning for instance-dependent Dropout Architecture (CODA), which is inspired from a philosophical theory, constructivism learning. Specially, based on the theory we have designed a better drop out technique, Uniform Process Mixture Models, using a Bayesian nonparametric method Uniform process. We have evaluated our proposed method on 5 real-world datasets and compared the performance with other state-of-the-art dropout techniques. The experimental results demonstrated the effectiveness of CODA.