Continually self-improving AI

TL;DR

This paper explores methods for creating AI systems that can improve themselves continually by generating synthetic data, self-bootstrapping knowledge, and autonomously exploring new training algorithms, aiming to surpass current human-dependent limitations.

Contribution

It introduces three novel techniques: synthetic data generation for efficient knowledge update, self-generated data for pretraining, and automated search over learning algorithms to enable self-improvement.

Findings

Synthetic data enhances knowledge acquisition from limited sources.

Self-generated data can bootstrap pretraining without external models.

Automated algorithm search surpasses human-designed training paradigms.

Abstract

Modern language model-based AI systems are remarkably powerful, yet their capabilities remain fundamentally capped by their human creators in three key ways. First, although a model's weights can be updated via fine-tuning, acquiring new knowledge from small, specialized corpora after pretraining remains highly data-inefficient. Second, the training of these systems relies heavily on finite, human-generated data from across history. Third, the pipelines used to train AI models are confined by the algorithms that human researchers can discover and explore. This thesis takes a small step toward overcoming these inherent limitations, presenting three chapters aimed at breaking these dependencies to create continually self-improving AI. First, to overcome this data-efficiency barrier in knowledge acquisition, we propose a synthetic data approach that diversifies and amplifies small corpora into rich knowledge representations, enabling a model to effectively update its parameters from limited source material. Second, to reduce reliance on human data, we show that given a fixed amount of such data, the model can self-generate synthetic data to bootstrap its fundamental pretraining capabilities without distillation from any off-the-shelf, instruction-tuned LM. Finally, to transcend human-engineered training paradigms, we demonstrate that by scaling search during test time over the space of algorithms, AI can search over a larger space of learning algorithm configurations than human researchers can explore manually.