Big Learning
Yulai Cong, Miaoyun Zhao
TL;DR
Big learning is a comprehensive framework that leverages large-scale data and models to unify various data distributions and machine learning paradigms, promising flexible, trustworthy, and universal AI systems.
Contribution
The paper introduces big learning as a new paradigm that exploits large-scale data and models to unify diverse data distributions and learning tasks within a single universal framework.
Findings
Big learning underpins most existing foundation models.
It offers flexibility for complete and incomplete data training.
Experiments validate its effectiveness across tasks.
Abstract
Recent advances in big/foundation models reveal a promising path for deep learning, where the roadmap steadily moves from big data to big models to (the newly-introduced) big learning. Specifically, the big learning exhaustively exploits the information inherent in its large-scale complete/incomplete training data, by simultaneously modeling many/all joint/conditional/marginal data distributions across potentially diverse domains, with one universal foundation model. We reveal that big learning () underlies most existing foundation models, () is equipped with extraordinary flexibilities for complete/incomplete training data and trustworthy data tasks, () is capable of delivering all joint/conditional/marginal data capabilities with one universal model, and () unifies conventional machine learning paradigms and enables their flexible cooperations, manifested as a…
Peer Reviews
Decision·Submitted to ICLR 2024
The authors introduce a new concept call the Big Learning, which unifies many existing learning paradigms such as Mask LM. Based on this new learning concept, the author proposes advanced versions of GAN and maximum likelihood learning. The authors also run experiments and show the efficacy of the proposed methods.
I have mixed feelings about this paper. While the authors propose a new learning paradigm Big Learning that can unify some existing learning paradigms, it seems to me that this new learning paradigm is just a slightly more "advanced" version of self-supervised learning. Can authors highlight the main differences? Also, the description of experiments in Section 4 is not clear to me, e.g., what is the experiment setups for the ones shown in Fig 2, 3, 4?
This paper proposes to generalize a problem for masked/casual/autoregressive LM, supervised classification, generation, which could provide some structural perspective. The authors investigated the GAN network in this context and conduct experiments to test their method.
1. The presentation is poor, without the examples the definition of the big learning problem is unclear. The authors should define $x_T$ and $x_S$. Most of the examples only have one pair of $(T, S)$, not a collection. 2. This paper lacks of systematic quantitative comparisons between the proposed approach and existing methods on pretraining foundation models with large-scale data. 3. The claims are not sufficiently supported by the experiments: while the authors argue the potential of the lea
The authors try to propose a big learning framework, which contains most objectives of foundation models as special cases and potentially delivers all joint, conditional, and marginal sampling data capabilities.
Quality/Clarity: the paper is hard to follow. The title is too big and it is hard to know its contribution since it aggregates the existing approaches and wants to put everything under this framework. And if BigLearn-GAN is the contribution, then please compare it with the state of the art. Originality/significance: the idea is ok, which wants to put all models under this big learning framework. However, it only aggregates the current approaches, and these approaches are known and did before.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning and Data Classification · Machine Learning and Algorithms · Domain Adaptation and Few-Shot Learning
MethodsSelf-Learning