# Applying Visual Domain Style Transfer and Texture Synthesis Techniques to Audio - Insights and Challenges

## TL;DR

This paper explores adapting visual style transfer techniques to audio spectrograms, highlighting challenges, domain differences, and proposing 1D CNN architectures for improved audio texture synthesis.

## Contribution

It critically analyzes the limitations of vision-based CNNs for audio and introduces alternative 1D CNN approaches tailored for audio texture synthesis.

## Key findings

- Gram matrix style correlates with timbral signatures
- Layer activity captures pitch and rhythm structures
- Proposed architectures improve audio texture alignment

## Abstract

Style transfer is a technique for combining two images based on the activations and feature statistics in a deep learning neural network architecture. This paper studies the analogous task in the audio domain and takes a critical look at the problems that arise when adapting the original vision-based framework to handle spectrogram representations. We conclude that CNN architectures with features based on 2D representations and convolutions are better suited for visual images than for time-frequency representations of audio. Despite the awkward fit, experiments show that the Gram matrix determined "style" for audio is more closely aligned with timbral signatures without temporal structure whereas network layer activity determining audio "content" seems to capture more of the pitch and rhythmic structures. We shed insight on several reasons for the domain differences with illustrative examples. We motivate the use of several types of one-dimensional CNNs that generate results that are better aligned with intuitive notions of audio texture than those based on existing architectures built for images. These ideas also prompt an exploration of audio texture synthesis with architectural variants for extensions to infinite textures, multi-textures, parametric control of receptive fields and the constant-Q transform as an alternative frequency scaling for the spectrogram.