Conformer LLMs -- Convolution Augmented Large Language Models

TL;DR

This paper introduces Conformer LLMs, combining convolutional layers and Transformers in a causal setup to improve large language models, especially for speech and multi-modal data, achieving significant performance gains.

Contribution

It adapts non-causal conformer architectures for causal large language models, integrating convolutional and Transformer blocks for enhanced performance.

Findings

Significant performance improvements over baseline models.

Effective modeling of local and global dependencies.

Robust architecture applicable beyond speech tasks.

Abstract

This work builds together two popular blocks of neural architecture, namely convolutional layers and Transformers, for large language models (LLMs). Non-causal conformers are used ubiquitously in automatic speech recognition. This work aims to adapt these architectures in a causal setup for training LLMs. Transformers decoders effectively capture long-range dependencies over several modalities and form a core backbone of modern advancements in machine learning. Convolutional architectures have been popular in extracting features in domains such as raw 1-D signals, speech, and images, to name a few. In this paper, by combining local and global dependencies over latent representations using causal convolutional filters and Transformer, we achieve significant gains in performance. This work showcases a robust speech architecture that can be integrated and adapted in a causal setup beyond speech applications for large-scale language modeling.