Zeta Distribution and Transfer Learning Problem

TL;DR

This paper investigates the connection between zeta distributions and transfer learning, proposing models inspired by natural processes and evolution to analyze learning bounds and the feasibility of AI.

Contribution

It introduces new models of transfer learning based on zeta distributions and analyzes their implications for AI development and natural process modeling.

Findings

Power-law models fit natural processes well.

Transfer learning effectiveness varies across models.

Evolution-inspired models suggest AI feasibility in nature.

Abstract

We explore the relations between the zeta distribution and algorithmic information theory via a new model of the transfer learning problem. The program distribution is approximated by a zeta distribution with parameter near $1$. We model the training sequence as a stochastic process. We analyze the upper temporal bound for learning a training sequence and its entropy rates, assuming an oracle for the transfer learning problem. We argue from empirical evidence that power-law models are suitable for natural processes. Four sequence models are proposed. Random typing model is like no-free lunch where transfer learning does not work. Zeta process independently samples programs from the zeta distribution. A model of common sub-programs inspired by genetics uses a database of sub-programs. An evolutionary zeta process samples mutations from Zeta distribution. The analysis of stochastic processes inspired by evolution suggest that AI may be feasible in nature, countering no-free lunch sort of arguments.