Adaptive n-ary Activation Functions for Probabilistic Boolean Logic

TL;DR

This paper introduces adaptive n-ary activation functions that approximate probabilistic Boolean logic, enabling neural networks to learn complex logical relations efficiently and relate logical complexity to parameter sparsity.

Contribution

It proposes a novel class of activation functions that model belief functions with adjustable arity, linking logical complexity to parameter efficiency and enabling gradient-based optimization.

Findings

Successfully learned arbitrary logic such as XOR and ternary disjunction in a single layer.

Represented belief tables with parameters directly related to logical arity.

Demonstrated the potential for reducing logical complexity via parameter sparsity.

Abstract

Balancing model complexity against the information contained in observed data is the central challenge to learning. In order for complexity-efficient models to exist and be discoverable in high dimensions, we require a computational framework that relates a credible notion of complexity to simple parameter representations. Further, this framework must allow excess complexity to be gradually removed via gradient-based optimization. Our n-ary, or n-argument, activation functions fill this gap by approximating belief functions (probabilistic Boolean logic) using logit representations of probability. Just as Boolean logic determines the truth of a consequent claim from relationships among a set of antecedent propositions, probabilistic formulations generalize predictions when antecedents, truth tables, and consequents all retain uncertainty. Our activation functions demonstrate the ability to learn arbitrary logic, such as the binary exclusive disjunction (p xor q) and ternary conditioned disjunction ( c ? p : q ), in a single layer using an activation function of matching or greater arity. Further, we represent belief tables using a basis that directly associates the number of nonzero parameters to the effective arity of the belief function, thus capturing a concrete relationship between logical complexity and efficient parameter representations. This opens optimization approaches to reduce logical complexity by inducing parameter sparsity.