PixelBytes: Catching Unified Representation for Multimodal Generation

TL;DR

PixelBytes introduces a unified multimodal representation learning approach that integrates text, audio, actions, and images, demonstrating improved autoregressive modeling and diffusion techniques for complex data generation tasks.

Contribution

The paper proposes PixelBytes, a novel framework for unified multimodal representation learning, combining various data modalities and exploring diverse model architectures and diffusion methods.

Findings

Autoregressive models outperform predictive models in multimodal tasks.

Diffusion models can be effectively applied to control problems.

Parallelized generation enhances efficiency in multimodal data synthesis.

Abstract

This report presents PixelBytes, an approach for unified multimodal representation learning. Drawing inspiration from sequence models like Image Transformers, PixelCNN, and Mamba-Bytes, we explore integrating text, audio, action-state, and pixelated images (sprites) into a cohesive representation. We conducted experiments on a PixelBytes Pokemon dataset and an Optimal-Control dataset. Our investigation covered various model architectures, including Recurrent Neural Networks (RNNs), State Space Models (SSMs), and Attention-based models, with a focus on bidirectional processing and our PxBy embedding technique. We evaluated models based on data reduction strategies and autoregressive learning, specifically examining Long Short-Term Memory (LSTM) networks in predictive and autoregressive modes. Our results indicate that autoregressive models perform better than predictive models in this context. Additionally, we found that diffusion models can be applied to control problems and parallelized generation. PixelBytes aims to contribute to the development of foundation models for multimodal data processing and generation. The project's code, models, and datasets are available online.