# Deep Switch Networks for Generating Discrete Data and Language

## TL;DR

This paper introduces multilayer switch networks that generate high-dimensional discrete data, offering an interpretable, stable, and efficient alternative to existing generative models like deconvolution networks and adversarial training.

## Contribution

The paper presents a novel multilayer switch network architecture with a statistical training framework based on maximum likelihood, avoiding complex backpropagation and adversarial methods.

## Key findings

- Effective in generating synthetic high-dimensional binary and natural language data.
- Stable training process using maximum likelihood without adversarial components.
- Demonstrates good generalization and interpretability in experiments.

## Abstract

Multilayer switch networks are proposed as artificial generators of high-dimensional discrete data (e.g., binary vectors, categorical data, natural language, network log files, and discrete-valued time series). Unlike deconvolution networks which generate continuous-valued data and which consist of upsampling filters and reverse pooling layers, multilayer switch networks are composed of adaptive switches which model conditional distributions of discrete random variables. An interpretable, statistical framework is introduced for training these nonlinear networks based on a maximum-likelihood objective function. To learn network parameters, stochastic gradient descent is applied to the objective. This direct optimization is stable until convergence, and does not involve back-propagation over separate encoder and decoder networks, or adversarial training of dueling networks. While training remains tractable for moderately sized networks, Markov-chain Monte Carlo (MCMC) approximations of gradients are derived for deep networks which contain latent variables. The statistical framework is evaluated on synthetic data, high-dimensional binary data of handwritten digits, and web-crawled natural language data. Aspects of the model's framework such as interpretability, computational complexity, and generalization ability are discussed.