Partially Randomizing Transformer Weights for Dialogue Response Diversity

TL;DR

This paper introduces PaRaFormer, a simple transformer extension that improves dialogue response diversity by partially randomizing weights, achieving comparable performance without added training complexity or model size.

Contribution

Proposes PaRaFormer, a novel method that enhances response diversity by freezing selected transformer layer weights after random initialization.

Findings

PaRaFormer achieves response diversity comparable to more complex methods.

The approach does not increase training difficulty or model size.

Experimental results validate the effectiveness of partial weight randomization.

Abstract

Despite recent progress in generative open-domain dialogue, the issue of low response diversity persists. Prior works have addressed this issue via either novel objective functions, alternative learning approaches such as variational frameworks, or architectural extensions such as the Randomized Link (RL) Transformer. However, these approaches typically entail either additional difficulties during training/inference, or a significant increase in model size and complexity. Hence, we propose the \underline{Pa}rtially \underline{Ra}ndomized trans\underline{Former} (PaRaFormer), a simple extension of the transformer which involves freezing the weights of selected layers after random initialization. Experimental results reveal that the performance of the PaRaformer is comparable to that of the aforementioned approaches, despite not entailing any additional training difficulty or increase in model complexity.