Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation

TL;DR

Pisces is a novel auto-regressive multimodal foundation model that uses decoupled visual encoding and tailored training to excel in both image understanding and generation tasks, achieving competitive results across multiple benchmarks.

Contribution

Introduces Pisces, a unified multimodal model with separate visual encoders and specialized training, improving performance in both understanding and generation tasks.

Findings

Strong performance on 20+ image understanding benchmarks

Robust capabilities demonstrated on GenEval for image generation

Decoupled visual encoding enhances task-specific performance

Abstract

Recent advances in large language models (LLMs) have enabled multimodal foundation models to tackle both image understanding and generation within a unified framework. Despite these gains, unified models often underperform compared to specialized models in either task. A key challenge in developing unified models lies in the inherent differences between the visual features needed for image understanding versus generation, as well as the distinct training processes required for each modality. In this work, we introduce Pisces, an auto-regressive multimodal foundation model that addresses this challenge through a novel decoupled visual encoding architecture and tailored training techniques optimized for multimodal generation. Combined with meticulous data curation, pretraining, and finetuning, Pisces achieves competitive performance in both image understanding and image generation. We evaluate Pisces on over 20 public benchmarks for image understanding, where it demonstrates strong performance across a wide range of tasks. Additionally, on GenEval, a widely adopted benchmark for image generation, Pisces exhibits robust generative capabilities. Our extensive analysis reveals the synergistic relationship between image understanding and generation, and the benefits of using separate visual encoders, advancing the field of unified multimodal models.