Generalized Independent Component Analysis Over Finite Alphabets

TL;DR

This paper introduces a novel approach to independent component analysis over finite alphabets, solving Barlow's minimal redundancy problem efficiently with theoretical guarantees, and demonstrating broad applications in neural networks, coding, and network management.

Contribution

It provides the first efficient, constructive solutions to Barlow's minimal redundancy problem, generalizing ICA without prior generative assumptions.

Findings

NP-hard problem can be solved with branch and bound algorithms.

Linear programming approximations yield tight solutions.

Applications extend to neural networks, coding, and network management.

Abstract

Independent component analysis (ICA) is a statistical method for transforming an observable multidimensional random vector into components that are as statistically independent as possible from each other.Usually the ICA framework assumes a model according to which the observations are generated (such as a linear transformation with additive noise). ICA over finite fields is a special case of ICA in which both the observations and the independent components are over a finite alphabet. In this work we consider a generalization of this framework in which an observation vector is decomposed to its independent components (as much as possible) with no prior assumption on the way it was generated. This generalization is also known as Barlow's minimal redundancy representation problem and is considered an open problem. We propose several theorems and show that this NP hard problem can be accurately solved with a branch and bound search tree algorithm, or tightly approximated with a series of linear problems. Our contribution provides the first efficient and constructive set of solutions to Barlow's problem.The minimal redundancy representation (also known as factorial code) has many applications, mainly in the fields of Neural Networks and Deep Learning. The Binary ICA (BICA) is also shown to have applications in several domains including medical diagnosis, multi-cluster assignment, network tomography and internet resource management. In this work we show this formulation further applies to multiple disciplines in source coding such as predictive coding, distributed source coding and coding of large alphabet sources.