Mixture of Expert/Imitator Networks: Scalable Semi-supervised Learning Framework

TL;DR

This paper introduces a scalable semi-supervised learning framework for text classification that uses a mixture of expert and imitator networks, improving performance with more unlabeled data.

Contribution

The paper presents a novel SSL method where imitator networks learn from expert networks to enhance text classification, demonstrating scalability and improved accuracy.

Findings

Consistent performance improvements over baseline DNNs.

Effective utilization of large unlabeled datasets.

Scalable approach suitable for various models.

Abstract

The current success of deep neural networks (DNNs) in an increasingly broad range of tasks involving artificial intelligence strongly depends on the quality and quantity of labeled training data. In general, the scarcity of labeled data, which is often observed in many natural language processing tasks, is one of the most important issues to be addressed. Semi-supervised learning (SSL) is a promising approach to overcoming this issue by incorporating a large amount of unlabeled data. In this paper, we propose a novel scalable method of SSL for text classification tasks. The unique property of our method, Mixture of Expert/Imitator Networks, is that imitator networks learn to "imitate" the estimated label distribution of the expert network over the unlabeled data, which potentially contributes a set of features for the classification. Our experiments demonstrate that the proposed method consistently improves the performance of several types of baseline DNNs. We also demonstrate that our method has the more data, better performance property with promising scalability to the amount of unlabeled data.