Exploring the encoding of linguistic representations in the Fully-Connected Layer of generative CNNs for Speech

TL;DR

This study investigates how the fully connected layer in CNNs for speech synthesis encodes linguistic information, revealing shared sublexical representations and proposing techniques for interpretability.

Contribution

It is the first to analyze the encoding of linguistic information in the FC layer of CNNs for speech, introducing novel exploration methods and manipulation techniques.

Findings

FC layer encodes lexically invariant sublexical representations

Manipulating FC layer affects speech output in predictable ways

Shared linguistic codes are systematically represented in FC weights

Abstract

Interpretability work on the convolutional layers of CNNs has primarily focused on computer vision, but some studies also explore correspondences between the latent space and the output in the audio domain. However, it has not been thoroughly examined how acoustic and linguistic information is represented in the fully connected (FC) layer that bridges the latent space and convolutional layers. The current study presents the first exploration of how the FC layer of CNNs for speech synthesis encodes linguistically relevant information. We propose two techniques for exploration of the fully connected layer. In Experiment 1, we use weight matrices as inputs into convolutional layers. In Experiment 2, we manipulate the FC layer to explore how symbolic-like representations are encoded in CNNs. We leverage the fact that the FC layer outputs a feature map and that variable-specific weight matrices are temporally structured to (1) demonstrate how the distribution of learned weights varies between latent variables in systematic ways and (2) demonstrate how manipulating the FC layer while holding constant subsequent model parameters affects the output. We ultimately present an FC manipulation that can output a single segment. Using this technique, we show that lexically specific latent codes in generative CNNs (ciwGAN) have shared lexically invariant sublexical representations in the FC-layer weights, showing that ciwGAN encodes lexical information in a linguistically principled manner.