JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence

TL;DR

JanusCoder introduces a multimodal code dataset and models that generate code from visual and textual inputs, advancing visual-programmatic interfaces for code intelligence.

Contribution

The paper presents a large-scale multimodal code corpus and unified models that handle diverse visual and textual programming tasks, surpassing existing specialized approaches.

Findings

JanusCode-800K is the largest multimodal code dataset to date.

JanusCoder models achieve performance comparable to commercial models.

Extensive analysis reveals insights into harmonizing logic and visual expression.

Abstract

The scope of neural code intelligence is rapidly expanding beyond text-based source code to encompass the rich visual outputs that programs generate. This visual dimension is critical for advanced applications like flexible content generation and precise, program-driven editing of visualizations. However, progress has been impeded by the scarcity of high-quality multimodal code data, a bottleneck stemming from challenges in synthesis and quality assessment. To address these challenges, we make contributions from both a data and modeling perspective. We first introduce a complete synthesis toolkit that leverages reciprocal synergies between data modalities to efficiently produce a large-scale, high-quality corpus spanning from standard charts to complex interactive web UIs and code-driven animations. Leveraging this toolkit, we construct JanusCode-800K, the largest multimodal code corpus to date. This powers the training of our models, JanusCoder and JanusCoderV, which establish a visual-programmatic interface for generating code from textual instructions, visual inputs, or a combination of both. Our unified model is a departure from existing approaches that build specialized models for isolated tasks. Extensive experiments on both text-centric and vision-centric coding tasks demonstrate the superior performance of the JanusCoder series, with our 7B to 14B scale models approaching or even exceeding the performance of commercial models. Furthermore, extensive analysis provides key insights into harmonizing programmatic logic with its visual expression. Our code and checkpoints are available at https://github.com/InternLM/JanusCoder.