Copula Representations and Error Surface Projections for the Exclusive Or Problem

TL;DR

This paper explores the XOR problem using copula functions and error surface visualizations to enhance understanding of neural network dynamics and cross-validation in a pedagogical manner.

Contribution

It introduces copula representations for XOR, compares error surface dynamics with different activation functions, and demonstrates cross-validation concepts visually.

Findings

Error surfaces differ significantly between RELU and tanh.

Copula representations extend XOR to real values.

Visualizations aid in understanding neural network training and validation.

Abstract

The exclusive or (xor) function is one of the simplest examples that illustrate why nonlinear feedforward networks are superior to linear regression for machine learning applications. We review the xor representation and approximation problems and discuss their solutions in terms of probabilistic logic and associative copula functions. After briefly reviewing the specification of feedforward networks, we compare the dynamics of learned error surfaces with different activation functions such as RELU and tanh through a set of colorful three-dimensional charts. The copula representations extend xor from Boolean to real values, thereby providing a convenient way to demonstrate the concept of cross-validation on in-sample and out-sample data sets. Our approach is pedagogical and is meant to be a machine learning prolegomenon.