Peirce in the Machine: How Mixture of Experts Models Perform Hypothesis Construction

TL;DR

This paper demonstrates that mixture of experts models outperform Bayesian methods due to higher functional capacity, and frames them as a form of Peircian abductive reasoning for hypothesis construction.

Contribution

It proves that mixture of experts have greater capacity than Bayesian methods and links them to Peircian abductive reasoning, supported by experiments.

Findings

Mixture of experts outperform Bayesian methods in prediction.

Mixture of experts have greater functional capacity.

Models can be viewed as Peircian hypothesis construction.

Abstract

Mixture of experts is a prediction aggregation method in machine learning that aggregates the predictions of specialized experts. This method often outperforms Bayesian methods despite the Bayesian having stronger inductive guarantees. We argue that this is due to the greater functional capacity of mixture of experts. We prove that in a limiting case of mixture of experts will have greater capacity than equivalent Bayesian methods, which we vouchsafe through experiments on non-limiting cases. Finally, we conclude that mixture of experts is a type of abductive reasoning in the Peircian sense of hypothesis construction.