J-Net: Randomly weighted U-Net for audio source separation

TL;DR

This paper investigates the effectiveness of randomly weighted neural networks for audio source separation, finding that they can perform comparably to trained models and facilitate architecture experimentation.

Contribution

It demonstrates the positive correlation between random and trained network performance in audio tasks and reveals that fixing the decoder to random weights improves results.

Findings

Randomly weighted networks perform well in audio source separation.

Fixing the decoder to random weights outperforms a trained encoder.

Positive correlation exists between random and trained network performance.

Abstract

Several results in the computer vision literature have shown the potential of randomly weighted neural networks. While they perform fairly well as feature extractors for discriminative tasks, a positive correlation exists between their performance and their fully trained counterparts. According to these discoveries, we pose two questions: what is the value of randomly weighted networks in difficult generative audio tasks such as audio source separation and does such positive correlation still exist when it comes to large random networks and their trained counterparts? In this paper, we demonstrate that the positive correlation still exists. Based on this discovery, we can try out different architecture designs or tricks without training the whole model. Meanwhile, we find a surprising result that in comparison to the non-trained encoder (down-sample path) in Wave-U-Net, fixing the decoder (up-sample path) to random weights results in better performance, almost comparable to the fully trained model.