On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learning

TL;DR

This paper empirically examines how pre-training on different modalities like language and vision impacts the fine-tuning of Transformer models in offline reinforcement learning, revealing modality-specific effects on representations and performance.

Contribution

It provides new insights into how modality-specific pre-training influences Transformer representations and their effectiveness in offline reinforcement learning tasks.

Findings

Pre-trained Transformers develop different internal representations from randomly initialized models.

Pre-trained models change less during fine-tuning, with large gradients affecting performance.

Language pre-training enables the model to learn efficiently even without context, suggesting it captures context-like information.

Abstract

We empirically investigate how pre-training on data of different modalities, such as language and vision, affects fine-tuning of Transformer-based models to Mujoco offline reinforcement learning tasks. Analysis of the internal representation reveals that the pre-trained Transformers acquire largely different representations before and after pre-training, but acquire less information of data in fine-tuning than the randomly initialized one. A closer look at the parameter changes of the pre-trained Transformers reveals that their parameters do not change that much and that the bad performance of the model pre-trained with image data could partially come from large gradients and gradient clipping. To study what information the Transformer pre-trained with language data utilizes, we fine-tune this model with no context provided, finding that the model learns efficiently even without context information. Subsequent follow-up analysis supports the hypothesis that pre-training with language data is likely to make the Transformer get context-like information and utilize it to solve the downstream task.