iSTFTNet2: Faster and More Lightweight iSTFT-Based Neural Vocoder Using 1D-2D CNN

TL;DR

iSTFTNet2 introduces a 1D-2D CNN architecture for neural vocoding, enhancing speed and lightweight design while maintaining high speech quality, by effectively modeling high-dimensional spectrograms.

Contribution

It proposes a novel 1D-2D CNN structure for neural vocoders, improving speed and model efficiency without sacrificing speech quality.

Findings

iSTFTNet2 is faster than previous models.

It is more lightweight with comparable speech quality.

The new architecture effectively models high-dimensional spectrograms.

Abstract

The inverse short-time Fourier transform network (iSTFTNet) has garnered attention owing to its fast, lightweight, and high-fidelity speech synthesis. It obtains these characteristics using a fast and lightweight 1D CNN as the backbone and replacing some neural processes with iSTFT. Owing to the difficulty of a 1D CNN to model high-dimensional spectrograms, the frequency dimension is reduced via temporal upsampling. However, this strategy compromises the potential to enhance the speed. Therefore, we propose iSTFTNet2, an improved variant of iSTFTNet with a 1D-2D CNN that employs 1D and 2D CNNs to model temporal and spectrogram structures, respectively. We designed a 2D CNN that performs frequency upsampling after conversion in a few-frequency space. This design facilitates the modeling of high-dimensional spectrograms without compromising the speed. The results demonstrated that iSTFTNet2 made iSTFTNet faster and more lightweight with comparable speech quality. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/istftnet2/.