Mathematically Modeling the Lexicon Entropy of Emergent Language

TL;DR

This paper introduces FiLex, a mathematical model that predicts how various hyperparameters influence the entropy of emergent language in deep learning systems, validated across multiple environments.

Contribution

The paper presents FiLex, a novel stochastic process model that accurately predicts lexicon entropy based on hyperparameters in emergent language systems.

Findings

FiLex accurately predicts entropy in all tested environment-hyperparameter cases.

Different environments exhibit diverse hyperparameter-entropy relationships.

The model enables precise, testable predictions of emergent language properties.

Abstract

We formulate a stochastic process, FiLex, as a mathematical model of lexicon entropy in deep learning-based emergent language systems. Defining a model mathematically allows it to generate clear predictions which can be directly and decisively tested. We empirically verify across four different environments that FiLex predicts the correct correlation between hyperparameters (training steps, lexicon size, learning rate, rollout buffer size, and Gumbel-Softmax temperature) and the emergent language's entropy in 20 out of 20 environment-hyperparameter combinations. Furthermore, our experiments reveal that different environments show diverse relationships between their hyperparameters and entropy which demonstrates the need for a model which can make well-defined predictions at a precise level of granularity.