Droplet3D: Commonsense Priors from Videos Facilitate 3D Generation

TL;DR

Droplet3D leverages commonsense priors from videos to improve 3D generation, addressing data scarcity by using multi-view and semantic information from videos, resulting in more consistent and plausible 3D content.

Contribution

Introduces Droplet3D-4M dataset and a novel generative model that incorporates video-based commonsense priors for enhanced 3D asset creation.

Findings

Effective spatial consistency in 3D generation

Improved semantic fidelity to text prompts

Potential for scene-level applications

Abstract

Scaling laws have validated the success and promise of large-data-trained models in creative generation across text, image, and video domains. However, this paradigm faces data scarcity in the 3D domain, as there is far less of it available on the internet compared to the aforementioned modalities. Fortunately, there exist adequate videos that inherently contain commonsense priors, offering an alternative supervisory signal to mitigate the generalization bottleneck caused by limited native 3D data. On the one hand, videos capturing multiple views of an object or scene provide a spatial consistency prior for 3D generation. On the other hand, the rich semantic information contained within the videos enables the generated content to be more faithful to the text prompts and semantically plausible. This paper explores how to apply the video modality in 3D asset generation, spanning datasets to models. We introduce Droplet3D-4M, the first large-scale video dataset with multi-view level annotations, and train Droplet3D, a generative model supporting both image and dense text input. Extensive experiments validate the effectiveness of our approach, demonstrating its ability to produce spatially consistent and semantically plausible content. Moreover, in contrast to the prevailing 3D solutions, our approach exhibits the potential for extension to scene-level applications. This indicates that the commonsense priors from the videos significantly facilitate 3D creation. We have open-sourced all resources including the dataset, code, technical framework, and model weights: https://dropletx.github.io/.

Tables

Table 1. Table 1: Comparison of Droplet3D-4M and other 3D-language datasets. Droplet3D is a large-scale dataset of 4 million 3D objects, distinguished by its provision of dense 85-view renderings and precisely aligned, multi-view level dense text annotations for each object, which is novel among existing 3D datasets. Note that MVI stands for Multi-view Images.

	Year	Objects	Format	Images	Views	Annotations	Anno-Grain
GSO[11]	2024	1K	Mesh	$-$	$-$	$-$	$-$
OmniObject3D[63]	2024	6K	Mesh	$-$	$-$	$-$	$-$
Thingi10K[78]	2016	10K	Mesh	$-$	$-$	$-$	$-$
3D-FUTURE[14]	2021	10K	Mesh	$-$	$-$	$-$	$-$
Objaverse-OA[39]	2025	14.8K	Mesh	$-$	$-$	$-$	$-$
ShapeNet[3]	2015	51K	Mesh	$-$	$-$	$-$	$-$
Anymate[9]	2025	230K	Mesh	$-$	$-$	$-$	$-$
Objaverse[8]	2022	800K	Mesh	$-$	$-$	$-$	$-$
TexVerse[75]	2025	858K	Mesh	$-$	$-$	3 sentence	Object Level
Objaverse-XL[7]	2023	10.2M	Mesh	$-$	$-$	$-$	$-$
CO3D[49]	2021	19K	MVI	1.5M	79	$-$	$-$
MV-Video[25]	2024	53K	MVI	88M	48	$-$	$-$
uCO3D[38]	2025	170K	MVI	34M	200	$-$	Object Level
G-Objaverse[79]	2024	200K	MVI	30M	38	$-$	$-$
MVImgNet[73]	2023	220K	MVI	6.5M	30	$-$	$-$
MVImgNet2.0[17]	2024	300K	MVI	9M	30	$-$	$-$
Objaverse++[35]	2025	500K	MVI	20M	40	$-$	$-$
Cap3D[40]	2023	791K	MVI	6M	8	15.0 words	Object Level
Droplet3D-4M	2025	4M	MVI	900M	85	260.0 words	Multi-view Level

Table 2. Table 2: Quantitative comparison of Droplet3D with other 3D generation methods that simultaneously support both text and image inputs. Droplet3D outperforms the other two methods.

	PSNR ($\uparrow$)	SSIM ($\uparrow$)	LPIPS ($\downarrow$)	MSE ($\downarrow$)	CLIP-S ($\uparrow$)
LGM[56]	21.38	0.84	0.137	0.011	0.737
MVControl[33]	22.31	0.88	0.31	0.009	0.61
Droplet3D (Ours)	28.36	0.76	0.03	0.0017	0.866

Table 3. Table 3: Comparison of Droplet3D and the backbone it inherits from, DropletVideo , in terms of generation performance. Droplet3D performs better.

	PSNR ($\uparrow$)	SSIM ($\uparrow$)	LPIPS ($\downarrow$)	MSE ($\downarrow$)	CLIP-S ($\uparrow$)
Droplet3D-5B	28.36	0.76	0.03	0.0017	0.866
DropletVideo-5B[74]	20.51	0.87	0.12	0.02	0.76

Table 4. Table 4: Comparison of different video generation backbone models . We compared video models including DropletVideo , in terms of generation performance.

	Params	PSNR ($\uparrow$)	SSIM ($\uparrow$)	LPIPS ($\downarrow$)	MSE ($\downarrow$)	CLIP-S ($\uparrow$)
DropletVideo[74]	5B	20.51	0.87	0.12	0.02	0.76
Cogvideox-Fun[1]	5B	15.16	0.50	0.21	0.09	0.68
Wan2.1-I2V	14B	12.97	0.89	0.19	0.067	0.67
Step-Video-TI2V	30B	15.32	0.79	0.30	0.08	0.64

Equations

$$P=\frac{1}{N}\sum _{i=1}^N\text{sign}(i)\cdot \text{sim}(p_i,g_i)$$

$$R=\frac{1}{M}\sum _{i=1}^M\text{sign}(i)\cdot \text{sim}(p_i,g_i)$$

$$F1=\frac{2\cdot P\cdot R}{P+R}$$

$$M=\min \left(1.0,\frac{\sum \mathrm{matches}(\mathrm{pattern}:M{V}_{\mathrm{pattern}})}{5}\right)$$

Full text

1 Introduction

Thanks to the increasing availability of massive online data, large-scale pre-trained models have demonstrated such generative potential in multi-modalities, such as text[43, 46], image[50, 32], and video[44, 74]. However, the scaling of 3D generative models faces a significant bottleneck due to data scarcity, as the collection of 3D data is considerably more challenging than for other, more readily available modalities. For example, the largest open-source 3D dataset, Objaverse-XL [8, 6, 35], contains merely 10 million samples, orders of magnitude smaller than the image–text paired datasets[51] which usually consists of billions of samples. This presents two main challenges. First, the limited coverage of existing 3D datasets hinders the comprehensive capture of the real world, leading to a scarcity of learnable spatial consistency information that is constrained by data distribution biases. Second, compared to modalities such as text and images, 3D generative models learn less semantic knowledge, restricting their capacity to generate diverse content. Consequently, while the scaling law has been extensively validated in other modalities, the progress of this paradigm in the 3D domain remains comparatively slow.

Fortunately, the abundance of online video data offers a promising solution to mitigate 3D data scarcity[59, 27, 62]. First, vast video resources inherently contain multi-view spatial consistency information, as illustrated by the “toy” and “dancing child” examples in Fig. 1 where objects are captured from multiple views. From this viewpoint, such videos can be regarded as a specific representation of 3D data. Introducing these videos significantly enhances the foundational capability of 3D generative models to maintain cross-view consistency. Second, the scale advantage of video datasets enables broader semantic knowledge than 3D data. As demonstrated by the “QR code” case in Fig. 1, by learning from video samples, a 3D model can comprehend semantic concepts not present in its 3D training corpus and successfully generate objects that conform to textual descriptions. This indicates that the diverse semantic knowledge encapsulated in videos provides 3D models with critical generalizable commonsense priors. Motivated by this, we hope to study a video-driven paradigm for 3D generation, which creates 3D content based on an image and detailed text prompts. A promising approach involves fine-tuning pretrained video generation models with 3D data, enabling the adapted 3D generation models to inherit beneficial commonsense priors from the video domain.

A promising paradigm involves fine-tuning a pre-trained video generation model using 3D data, enabling the resulting 3D model to inherit the rich commonsense priors from its video counterpart, thereby enhancing its generative capabilities. Although existing work[4, 58] recognized this potential, their specific fine-tuning datasets, base models, and strategies exhibited limitations. Specifically, these approaches fail to adequately leverage text supervision and lack explicit mechanisms to preserve the capabilities transferred from the video backbone.

In light of this, our research on this issue encompasses from dataset construction to model development. We proposed a large-scale 3D dataset, Droplet3D-4M. It comprises approximately 4 million 3D models, each accompanied by a 360° orbital rendering video and a detailed multi-view level text annotation. Each video consists of 85 frames captured along a circular camera path around the object, which covers a complete 360-degree view. On the text side, each video is paired with a dense natural language caption averaging about 260 words in length. This caption meticulously characterizes the object’s overall appearance and elaborates on the unique details of different views, such as front, side, or back views, from sequential viewpoints as the camera orbits. This dense multi-view-level text annotation provides rich supervisory signals for the model to understand and generate each viewpoint, which benefits 3D generative models developing.

Furthermore, we introduce a novel 3D generative model, Droplet3D, which supports both image and dense text input, allowing our model to generate content that meets user needs with greater granularity. It is fine-tuned from DropletVideo[74], a video diffusion model capable of modeling integral spatio-temporal consistency. Droplet3D is capable of generating a sequence of 85 dense multi-view images capturing a full 360-degree orbit of the object. In addition, to adapt to the input requirements of Droplet3D, we introduce a text rewriting module and an initial image canonical view alignment module to preprocess the user’s prompt, thereby aligning it with the distribution of Droplet3D-4M. Finally, based on the generated orbital views, we employed downstream applications using both 3D Gaussian splatting and textured mesh reconstruction algorithms, surpassing the fixed output format commonly used in current prevailing 3D generation solutions[77, 31].

Comprehensive qualitative and quantitative experiments demonstrate the superiority of our proposed approach for 3D content generation tasks based on images and text. Extensive visualization experiments show various interesting capabilities of Droplet3D, demonstrating the potential of video-driven 3D generation methods for creative generation. Notably, in contrast to the prevailing object-level 3D solutions[23, 64], our model demonstrates considerable potential for scene-level generation.

The contributions of this work are as follows:

• •

We observe that commonsense priors learned from videos can facilitate 3D generation, and we provide an initial exploration and validation of this video-driven 3D generative solution.

• •

We release Droplet3D-4M, a large-scale 3D dataset paired with circumferential descriptions, containing 4 million 3D models, dense multi-view images, and detailed orbital text captions.

• •

We introduce Droplet3D, a 3D multi-view generation model. It inputs both image and dense text and generates multi-view images, which supports downstream 3D Gaussian splatting and textured meshes reconstructions.

• •

We have open-sourced the dataset, code, the model weights, and the complete technical solution. We hope this initiative fosters algorithmic innovation in the public domain.

2 Related Work

2.1 3D Datasets

The 3D dataset aims to construct a collection of 3D content encompassing various data formats to support the research and evaluation of intelligent algorithms such as reconstruction and understanding. Existing 3D datasets are primarily constructed based on meshes or multi-view images.

Mesh-based 3D datasets, including Objaverse[8], Objaverse-XL[7], ShapeNet[3], Thingi10K[78], 3D-FUTURE[14], and OmniObject3D[63], provide mesh models defined by vertices, edges, and faces. This format makes them highly suitable for tasks such as 3D rendering and geometric analysis. For example, the widely used GSO[11] dataset falls into this category, which constructs 3D mesh structures based on scans of everyday objects. Additionally, the currently largest dataset, Objaverse-XL[7], also belongs to this category, containing over 10.2 million 3D objects sourced from a variety of origins, ranging from artificially designed models to photogrammetric scans, significantly exceeding its base version which composes of 800 thousand models[8]. However, despite its large scale, the quality of its samples varies considerably. Many samples contain only limited geometric patches and lack sufficient visual textures and text annotation information. This limitation restricts the application of such datasets in highly controllable 3D generation tasks requiring precise perspective-aware descriptions.

On contrast, multi-view image-based 3D datasets, such as MVImgNet[73], MVImgNet2.0[17], and Co3D[49], provide 2D images or video frames capturing objects or scenes from multiple angles, supporting tasks such as Structure-from-Motion and 3D reconstruction. Among them, MVImgNet[73] includes 6.5 million frames from 219,188 videos across 238 categories, along with annotations such as masks and camera parameters. The further expanded MVImgNet2.0[17] covers approximately 300K objects with complete 360-degree views. Although these datasets provide rich multi-view data, their scale still lags significantly behind that of Objaverse-XL. Moreover, they also lack detailed text captions describing the appearance from specific perspectives. For example, although Cap3D[40] provides multi-view images, it provides short captions that describe the object’s overall appearance, thereby overlooking the variations that should inherently exist in descriptions across different viewpoints. This hinders tasks requiring the integration of visual-text understanding, such as text-guided 3D or video generation.

In light of this, we propose the large-scale 3D dataset Droplet3D-4M, designed to support the training of image-and-text-based 3D generation tasks. This dataset includes high-quality mesh models strictly selected from Objaverse-XL, as well as densely distributed multi-view rendered images surrounding the objects. More importantly, Droplet3D-4M provides detailed text annotations at the surrounding multi-view level with an average length of 260 words, offering critical multi-modal supervision annotations for 3D generation tasks.

2.2 3D Generation Models

In recent years, the task of generating 3D content from text or a single image has advanced rapidly, underscoring the remarkable potential of AIGC to expand our repository of 3D digital assets. Existing research falls broadly into two paradigms.

The first paradigm, termed native 3D generation methods [42, 26, 77], can directly generate editable 3D assets represented by meshes or voxels from image or text prompts. These methods typically require training on large-scale datasets and employ feedforward inference mechanisms to achieve efficient generation. Although highly efficient and multi-view consistent, their performance heavily depends on the scale and quality of the training data, which often face accessibility and acquisition limitations.

Harnessing the remarkable generative power of large-scale pre-trained models in images [50, 32], and videos [44, 74], the second paradigm unifies 2D generative models to distill or reconstruct multi-view observations into 3D representations such as NeRF and 3DGS, yielding an efficient 3D generation framework. In the line of combining image generation models, pioneering work DreamFusion [45] and subsequent studies [57, 71, 34, 5, 72, 60] employ techniques like Score Distillation Sampling (SDS) [45] or its improved variants to distill 3D information from 2D diffusion priors. To enhance multi-view consistency, more recent studies [55, 37, 67] introduce conditional controls, such as camera embeddings [54, 37] and epipolar constraints [22], to fine-tune pre-trained diffusion models. Meanwhile, other works [19, 56, 65] propose to learn 3D via a two-stage pipeline: first generating intermediate multi-view images and then utilizing these images to assist in feedforward 3D reconstruction. However, the generated intermediate images in such methods are typically sparse, and even with the limited range, leading to the spatial consistency required for high-quality 3D reconstruction remains elusive.

More recently, video generation models [2, 18] have effectively maintained inter-frame consistency by integrating temporal continuity, making them particularly suitable for generating coherent multi-view outputs. Inspired by this, several studies (e.g., V3D[4], SV3D[58], VideoMV[80], etc. [4, 58, 41, 80, 68, 16]) suggest adopting fine-tuned video diffusion models to generate multi-view outputs for subsequent 3D generation. Although impressive generation results, these approaches fail to adequately leverage text supervision and lack explicit mechanisms to preserve the capabilities transferred from the video backbone. Additionally, some methods [21, 33, 66, 41, 36, 69] consider jointly modeling both text and image prompts to produce 3D assets with better appearance and geometry control. For example, IM-3D [41] generates multi-view images from text and image pairs with the aid of a video diffusion model, followed by 3D reconstruction to generate high-quality 3D assets. Nevertheless, videos and video generation models are still underexplored in 3D generation due to simple text or image prompts. Therefore, in this work, we introduce a novel 3D generative model, Droplet3D, which supports both image and dense text input, to further explore the potential of video modality in advancing 3D generation. Our model allows generate content that meets user needs with greater granularity, and supports downstream 3D Gaussian splatting and textured meshes reconstructions.

3 The Droplet3D-4M Dataset

To take advantage of the commonsense priors acquired by video generation models trained in large-scale video data and thereby enhance the generalization capability of 3D generative models, we construct a novel dataset Droplet3D-4M to bridge the gap between the video and 3D domains. Specifically, a qualifying 3D sample must satisfy two criteria. First, it should support dense, coherent multi-view sequences. This ensures compatibility with the output interface of video generation models, facilitating the generation of spatially consistent 3D content. Second, it should be accompanied by dense text annotations. This provides stronger supervision, enabling the 3D generator to maximally retain the video model’s semantic understanding of text. As the 3D generator inherits broader semantic knowledge from video data, this dense text supervision allows the generative model to effectively exploit this inherited capability.

Accordingly, our proposed dataset, Droplet3D-4M, comprises dense multi-view rendered videos and fine-grained, multi-view-level text annotations, as illustrated in Fig. 2. Each rendered video consists of an 85-frame image sequence captured from uniformly distributed 360° orbital viewpoints. The angular difference between adjacent frames is strictly controlled to be within 5°, ensuring video coherence, which is a critical factor In training video-driven generative models. In terms of text annotations, we provide dense descriptions with an average length of 260 words, far exceeding those in existing 3D datasets. More importantly, the annotations not only cover holistic appearances such as shape and style but also specifically describe appearance variations induced by changes in viewpoint. For example in Fig. 2, the second paragraph of the text annotation details the side and rear features of the figurine, describing that the yellow backpack becomes partially visible from a side view and is only fully revealed from the back. This fine-grained, view-aware annotation paradigm provides unprecedented supervisory signals for the 3D generative model, effectively guiding and preserving the backbone network’s capacity for complex semantic understanding. In contrast to current prevailing 3D datasets, which typically lack text annotations or provide only brief descriptions at object-level, Droplet3D-4M achieves a significant enhancement in information richness, which has been illustrated in Tab. 1.

For the construction of Droplet3D-4M, we propose a data pre-processing pipeline that balances quality and efficiency, and construct a new dataset, Droplet3D-4M. This pipeline employs adaptive sampling techniques instead of resource-intensive rendering during the initial screening phase, followed by targeted high-fidelity rendering only for validated assets. Compared to conventional workflows, this approach reduces computational overhead by a factor of $4$ to $7\times$ while generating metadata-rich outputs of superior quality for multimodal learning tasks. The pipeline consists of three key parts: multi-view video rendering, image evaluation metric filtering, and multi-view-level caption generation, as shown in Fig. 3. Note that the raw 3D models we collected are from Objaverse-XL[7], comprising 6.3 million models sourced from GitHub and Sketchfab.

3.1 Multi-View video Rendering

We designed both coarse and fine rendering strategies for multi-view videos to mitigate the explosive computational consumption caused by rendering all data directly using the Blender engine. For the coarse rendering, we initially rendered the 6.3 million 3D models sourced from Objaverse-XL at low quality using the pyrender library. For the fine rendering, we employed BlenderProc to render the filtered 4 million models that met the quality and aesthetic score criteria.

The fundamental rendering hyperparameters were kept consistent across both rendering processes. The perspective camera configuration was set to a 50-degree vertical field of view and maintained a fixed aspect ratio of 1.0. Prior to rendering, the 3D meshes underwent geometric normalization, including aligning the centroid to the world origin, uniformly scaling to a unit maximum extent, and optional axial rotation. The scene lighting utilized a multi-light setup: a directional ambient light with an intensity of 2.0 provided base illumination, supplemented by three positional point lights, each with an intensity of 8.0, strategically placed on the left, right, and rear of the object at a distance of 2 units.

The coarse rendering process leveraged PyRender’s off-screen rendering capabilities and EGL backend acceleration, while the fine rendering process utilized Blender’s EEVEE renderer to enhance quality. For the coarse rendering process, only 8 views were rendered for metric calculation and filtering. For the high-quality rendering, 85 camera positions were uniformly distributed along a circular trajectory with a fixed 0° elevation angle, with radial distances randomly varying between 1.6 and 2.0 units. All cameras were focused on the scene origin, with the $+Y$ axis serving as the upward reference direction.

3.2 Image-Assessment Metric Filtering

We filtered the dataset by selecting high-quality multi-view rendering videos based on aesthetics and image quality. We utilize the publicly available LAION aesthetics model[51] to compute aesthetic scores and the DOVER-Technical model[61] to evaluate image quality. Only clips surpassing predefined thresholds are retained. Notably, nearly 77% of clips achieve an aesthetic score above 4.0, while approximately 81% exceed a score of 4.0 in image quality, underscoring the dataset’s high visual fidelity.

3.3 Multi-View consistent captions Generation

3.3.1 Supervised-finetuning VLM captioner

The proposed framework employs an AI-driven approach to convert video content into detailed textual descriptions, significantly reducing the need for human annotation. Traditional video captioning methods often produce brief outputs that fail to capture nuanced visual elements across different perspectives. To address this, our methodology incorporates a dual-phase enhancement mechanism. First, we construct a curated video dataset featuring comprehensive annotations that precisely track object states, camera movements, and – most importantly – variations in detail levels across viewpoint transitions. These preliminary descriptions are then linguistically refined using GPT-4 to ensure grammatical precision and stylistic consistency, resulting in high-quality training material.

For the technical implementation, we fine-tuned multiple cutting-edge multimodal models within the 7-9 billion parameter spectrum, considering both performance and efficiency. The generated annotations exhibit variations in detail levels across viewpoint transitions – including precise camera operations, object-surface changes, illumination changes, and environmental characteristics. Such meticulous documentation offers vital training cues for keeping the multi-view correspondence, especially concerning appearance changes. Quality assurance measurements indicate that the automated captions match or surpass human-generated annotations in terms of multi-view correpondence.

3.3.2 Enhancing MLLMs for Multi-View Captioning via GRPO

Recent studies have demonstrated that test-time scaling can significantly enhance the reasoning abilities of large-scale language models. This technique has been successfully implemented in state-of-the-art systems such as OpenAI’s o1[24] and Deepseek-R1[15], both of which achieve strong performance on challenging tasks involving mathematical reasoning and programming. However, vision-language models (VLMs) fine-tuned using conventional approaches frequently exhibit problematic behaviors, including incorrect object type predictions, misleading functional descriptions, and hallucinations of non-existent visual details. To mitigate these issues, the GRPO algorithm[52] has proven particularly effective when applied to supervised fine-tuning for specialized vision tasks. Empirical results confirm its advantages across multiple domains, including visual question answering[76], spatial relationship understanding[53], and temporal sequence analysis in video content[13].

Therefore, we designed the VLM model enhancements throught the second stage GRPO Reinforcement learning. We designed five dimensions to ensure the output descriptions meet our requirements for multi-perspective image captioning. They are Subject(Judging the type of the object), material(what the object is made of), Functional(what the object is used for), Details(changes due to the camera motion), OCR(whether the visual text during the view change). Also, similar to Deepseek-R1, we designed ‘‘<think>dimensions</think><answer>caption</answer>’’ as the output format. The dimensions contain the descriptions of the object for the five dimensions above. The caption part is the fluent mutli-view caption for this sample, combining the dimension contents above with the view changes. Fig. 4 shows an example.

For the Reward funtions, we mainly focus on the content metric and the multiview-match format metric. The content metric formula is as:

[TABLE]

[TABLE]

[TABLE]

The multiview-match format metric is as:

[TABLE]

The Multi-view pattern can be described as the more key words, like front, circle-around, side, back, self-rotation, 360-degree, appear, the more pattern award will be obtained. The final reward is determined as: $Reward=F1+M$. Ultimately, captions that are insufficiently detailed are compelled to incorporate additional information. As illustrated in Fig. 5 with samples from Droplet3D-4M, it is evident that the captions contain substantial details regarding the appearance and perspectives.

3.4 Data Overall and Statistics

In conclusion, the constructed Droplet3D-4M dataset comprises a meticulously curated collection of 4 million fully-covered multi-view video sequences, encompassing approximately 900 million individual frames with a total runtime exceeding 8,000 hours. Its unprocessed raw 3D models were collected from Objaverse-XL[7], comprising 6.3 million models sourced from GitHub and Sketchfab. This comprehensive visual dataset has been publicly released with the aim of advancing research in spherical vision systems and multi-view analysis. It should be noted that due to the origin of some source models being sampled from the internet, the dataset is published under the CC BY-NC-SA 4.0 license. Researchers are advised that although the material is freely available for academic research, all content within it remains subject to the original distribution permissions, restricting its use to non-commercial purposes only.

In addition, we computed the aesthetic and quality scores of the rendered videos in the dataset, as shown in Fig. 6. As previously mentioned, Droplet3D-4M ultimately retained only those samples with both scores above 4.0.

4 The Droplet3D Model

To inherit the spatial consistency and semantic knowledge priors that are readily accessible to video models into 3D generation algorithms, we designed and trained Droplet3D. To this end, we specifically employ multi-view rendered videos from Droplet3D-4M to interface with the video backbone model. Furthermore, we use multi-view level annotations to provide fine-grained supervision for model fine-tuning, thereby preserving the model’s cross-modal semantic understanding capability. The architecture of Droplet3D is illustrated in Fig. 7.

The technical details of the 3D generation process during Droplet3D is illustrated in Fig. 8. For any given text or image prompt, Droplet3D first aligns it with the model. Initially, we expand the user’s input requirements based on a lightweight large language model, rewriting them into dense textual descriptions that follow the same distribution as the multi-view level captions in the Droplet3D-4M dataset. Simultaneously, to accommodate inputs from arbitrary viewpoints, we designed a anonical viewpoint alignment module to adjust the perspective of the input images. For the backbone network, we introduce a 3D causal VAE to achieve implicit space encoding and decoding of the video, and employ a multi-modal diffusion transformer to facilitate the fusion of text and video modality features while constraining their independence.

Droplet3D demonstrates creativity in the process of generating surrounding multi-view videos based on images and text prompts. As an application, we adapted and integrated various converters to reconstruct different 3D modalities, such as textured mesh and Gaussian splatting, serving as applications of the content generated by Droplet3D.

4.1 Multi-View Image Generation

The generation of multi-view images, or surround-view videos, is a core step in Droplet3D, and it is crucial for creative 3D generation based on images and text. From our perspective, this step can be viewed as a bridge that transfers the general capabilities of video models to 3D generation.

3D Causal VAE

Within the backbone network of Droplet3D, videos are initially processed by a 3D causal Variational Autoencoder (VAE) to be transformed into latent space features. We extend the 3D Causal VAE using a VAE framework coupled with a 3D structural architecture, designed for encoding and decoding video frames. Unlike traditional autoencoders, the outputs of the VAE’s encoder and decoder are modeled as parameterized probability density distributions. Yang et al. [70] applied 3D convolutions to video reconstruction, demonstrating that a 3D structure can effectively mitigate the flickering artifacts often present in reconstructed videos. Inspired by this, we employ a 3D causal VAE to enhance the computational efficiency of Droplet3D, while ensuring the generated videos possess temporal continuity and stability. Simultaneously, on the long-text side, we extract linguistic features based on the encoder structure of the T5 model[47], feeding these features concurrently into the subsequent 3D Modality-expert Transformers.

3D Modality-Expert Transformer

Within the Transformer backbone network, we employ a 3D multi-modal attention mechanism to simultaneously process both text and video inputs. Specifically, for the video input, given that 3D full attention is a technique that has evolved alongside the widespread adoption of Transformers in computer vision, we incorporate 3D positional embeddings within the Transformer architecture. Concurrently, the text input is encoded through a conventional Transformer encoder to facilitate the smooth fusion of multi-modal information. Compared to previous decoupled approaches, this integrated methodology enables a more effective capture of dynamic changes within the video and enhances the generated content’s performance in terms of semantic consistency and diversity.

Further Training Based on DropletVideo

Given that the fundamental rationale for introducing the video generation step is to incorporate the general capabilities of video generation models into 3D asset generation, we proceed with further training based on an appropriate video generation backbone model. DropletVideo[74] is a video generation model that considers integral spatio-temporal consistency, supporting the generation process given an initial frame image and dense text input. This aligns formally with the multi-view image generation pipeline of Droplet3D. Additionally, because its training data includes large-scale video segments constrained by spatial consistency, such as certain street scene or character surround shots, it exhibits strong potential for 3D consistency. Therefore, we conduct secondary training based on its weights to fit the latent space feature distributions for 3D surround-view generation.

4.2 User Input Alignments

4.2.1 Multi-View-Level Text Alignment

To effectively align the variations in linguistic style and length of user-provided prompts and to provide detailed guidance for the surround-view video generation of the target object, we implement a dense prompt generation preprocessing step. Specifically, considering the superior performance of large language models in tasks such as text reasoning and image summarization, we employ the LoRA [20] technique to fine-tune an open-source model via instruction tuning. Experimental results indicate that approximately 500 in-domain samples are sufficient to achieve the desired fine-tuning level. This module is designed to rewrite user prompts while preserving the original semantics. It transforms the prompts into a standardized information structure similar to the captions used during training. Similar to Droplet3D-4M, the rewritten descriptions typically consist of two denser paragraphs. The first paragraph describes the overall style of the object and as many details as possible, while the second paragraph elaborates on multi-view level details based on the user’s prompt, as illustrated in Fig. 10. Users can review or modify the generated text for proofreading before inputting it into Droplet3D. Besides, this module supports multiple languages.

4.2.2 Image Perspective Alignment

3D generation methods typically achieve optimal performance only when provided with a canonical view, demonstrating poor adaptability to arbitrary input perspectives. This imposes strict requirements on user input. For instance, when a user aims to perform 3D generation from an image captured at a rotated angle, the performance of existing generation algorithms significantly degrades. Motivated by this limitation, we introduce a view alignment module in Droplet3D, enabling it to be seamlessly invoked with images from any viewpoint. This module fine-tunes the input view to one of the canonical viewpoints: front, left, right, or back.

We selected some samples with high image-quality from the Droplet3D-4M dataset and collected some AIGC generated samples from Internet, making a small dataset containing 200 examples. Then, we constructed the canonical viewpoints generation dataset from the small dataset. For each example, we manually selected four standard orthogonal views as ground truth, and constructed image pairs using initial images rendered by rotating $[15,\ 40]$ degrees clockwise or counterclockwise from the chosen ground truth viewpoint. Finally, we frame this task as an image editing problem, with the fixed editing prompt: “Convert this image to the closest canonical view, such as front, left, right or back.”

For the model selection and fine-tuning, we selected the FLUX.1-Kontext-dev [30] as the base model, training it with Low-Rank Adaptation (LoRA)[20]. As shown in Fig. 11, the model architecture employs a hybrid design combining dual-stream and single-stream blocks. In the dual-stream configuration, separate weight matrices process image and text tokens independently, with cross-modal fusion achieved through attention mechanisms applied to concatenated token sequences. Following dual-stream processing, the integrated token sequence undergoes joint encoding via successive single-stream blocks before final text token pruning and image token decoding.

5 Implementation and Experiments

5.1 Implementation Details

We employ the parameters of the pre-trained DropletVideo-5B[74] model as the weight initialization for Droplet3D. Similar to DropletVideo, we employ t5-v1_1-xxl[47] as the text encoder, and set the maximum token length to 400 instead of 226, to accommodate longer captions. The model architecture is based on the MMDiT series[12], consisting of 42 layers, with 48 attention heads per layer, and each head has a dimension of 64. The time step embedding dimension is set to 512. For optimization, we use Adam[29] with a weight decay of $3\times 10^{-2}$ and an eps value of $1\times 10^{-10}$. The learning rate is set to $2\times 10^{-5}$. The number of sampled frames ((N)) is fixed at 85. The model is trained using the bfloat16 mixed precision method, similar to the DeepSpeed framework[48].

During inference, the classifier-free guidance scale is set to 6.5 to enhance the motion smoothness of the generated surround-view videos. When trained on the Droplet3D-4M dataset, our model supports an image resolution of 512.

For the Canonical View Alignment training, The LoRA rank (network dimension) was set to 128. The learning rate is 1e-4 and the optimizer is AdamW8bit[10].

5.2 Evaluation

5.2.1 TI-to-3D Quantitative Evaluation

We conducted a quantitative comparative analysis based on the GSO dataset[11]. To validate the accuracy of the generated content by Droplet3D given a single image and text prompt, we compared it with other TI-to-3D methods, including LGM[56], MVControl[33] as shown in Tab. 2.

Given the bias in the coverage of the GSO dataset, we performed down-sampling and selected a subset of 200 samples for the evaluation. The sampled instances can cover all categories in the original dataset while ensuring, to the greatest extent possible, that their distribution is confirmed uniform.

As shown in Tab. 2, our model significantly outperforms the other two methods in terms of generation performance. On the one hand, metrics such as PSNR and SSIM confirm that the content generated by Droplet3D is of higher quality and closer to the ground truth. On the other hand, it demonstrates stronger semantic understanding capabilities, as evidenced by the CLIP scores in the last column. We attribute this to our use of T5 as the text encoder and extensive pre-training on video data.

Fig. 12 presents a qualitative comparison between Droplet3D and two other methods, LGM and MVControl, both of which also support simultaneous input of image and text. The results demonstrate that our method outperforms the others in terms of aesthetic quality, precision, and consistency between the generated content and the input image. This further validates the superiority of our Droplet3D in the TI-2-3D task.

5.2.2 Ablation Study

Considering that the video generation backbone network is a core module of Droplet3D, we conduct quantitative comparisons using different backbone networks. These include DropletVideo[74], which serves as the genetic origin of Droplet3D, and Cogvideox-Fun[1], a model of comparable scale. Additionally, we compare against Wan2.1-I2V-14B and Step-Video-TI2V-30B, which are currently among the most state-of-the-art open-source video generation models.

Tab. 3 demonstrates the effectiveness of the proposed Droplet3D-4M dataset, showing that models further trained with it exhibit improved generation consistency. It can be observed that while the original DropletVideo, which was not trained on Droplet3D-4M, also demonstrates commendable performance across various metrics, it still falls short of the Droplet3D proposed in this work.

Furthermore, as illustrated in Fig. 13, DropletVideo demonstrates a notable capability in generating camera-rotating surround-view videos, with the synthesized content exhibiting high fidelity and geometric accuracy. This proficiency suggests that DropletVideo is well-suited to serve as a robust initialization for the weights of a backbone network. Nevertheless, the model exhibits certain “shortcomings” for 3D. Specifically, it tends to generate inconsistencies in dynamic objects, such as a panda depicted with its eyes closed in a static pose. However, this inherent limitation underscores the critical necessity for subsequent fine-tuning with the proposed Droplet3D-4M to enhance the model’s performance in 3D generations.

In Fig. 13, it can be observed that, in the provided examples, Droplet3D is capable of generating superior 3D content. A plausible explanation for this is that the generated surround-view images by Droplet3D are both plausible and rich in detail, while maintaining high spatial consistency. This endows our model with commendable foundational capabilities for 3D generation.

Additionally, Quantitative results shown in Tab. 4 indicate that DropletVideo is indeed suitable for further training in 3D generation than other video generators. This may be related to its consideration of integral spatio-temporal consistency, as the large number of samples incorporating spatial consistency endows it with inherent 3D generation capabilities. It significantly outperforms other models of comparable scale, such as Cogvideox-Fun, and achieves performance comparable to or exceeding that of the other two larger and more advanced models, even without fine-tuning. Tab. 4 also demonstrates the video generation capabilities of DropletVideo-5B alongside Step-Video-TI2V-30B and Wan2.1-I2V-14B, currently the most popular open-source video generation models, despite their larger parameter counts. The results indicate that DropletVideo indeed possesses a higher capacity for maintaining 3D consistency compared to models of similar scale. While Step and WanX, due to their larger weights, also exhibit considerable ability, they may exhibit more pronounced object movements, such as the bear’s head turning, the gold coin onion blinking or jumping, possibly due to the training samples being more inclined towards narrative changes. In contrast, CogVideoX-Fun-v1.5, with a similar parameter count to DropletVideo-5B, demonstrates poorer performance. Visual examples are shown in Fig. 14. DropletVideo demonstrates a strong capability to generate multi-angle views of objects from input images, consistently achieving high-quality results for most cases. Note that considering the trade-offs between spatial consistency, foundational generation capabilities, and model size, we selected DropletVideo as the backbone for subsequent training in the 3D domain.

5.3 Potential and Applications in 3D Generation

5.3.1 Feasible Controllable-Creativity Driven by Language Prompts

Current mainstream 3D generation models typically only support image or text input[77, 64]. However, Droplet3D conditions on both an initial image and dense text, enabling it to support more targeted creative design, which is quite interesting. This is reflected in the variations of the generated assets when we employ AIGC images along with different textual descriptions.

Fig. 15–18 present four sets of examples. In each set, Droplet3D generates different samples by utilizing the same AIGC-generated image as input, accompanied by distinct text guidance. As shown in Fig. 15, when different text descriptions are provided for the background of the panda astronaut, the generated effects are both distinct and plausible. Droplet3D can generate his space backpack, orange backpack, and a vibrant energy ball that match the panda’s physique and appearance based on its body shape and visual characteristics. Notably, in the example of generating energy ball, Droplet3D can produce objects that are either dazzling or semi-transparent. Compared to generating textured meshes, images or videos can more intuitively produce content with richer colors. This is also one of the factors we believe contributes to the feasibility of video-based generative models in 3D generation. Similarly, as illustrated in Fig. 17, Droplet3D can produce varying details behind the axe depending on whether the text description specifies gems or runes. Besides, as shown in Fig. 18, when different decorations such as a wind-up key, crystal orb, or even a QR code are input, the model is able to understand the semantic meaning of the text. Additionally, its color generation is also relatively accurate. This capability for controllable 3D content generation stems from two main sources. On the one hand, our Droplet3D inherently supports both image and text interfaces at the data level. On the other hand, it benefits from a richer prior learned from a vast volume of video media. First, the greater amount of spatial consistency data extracted from these videos helps to enhance the 3D generator’s ability to produce accurate content. Second, the more general semantic knowledge integrated from the video backbone model enables our model to better comprehend textual requirements. A prime example is the “QR code” generation task. When prompted to generate a “QR code”, an object scarcely found in 3D model datasets, Droplet3D can still accurately identify what it is and generate it correctly, drawing upon its knowledge derived from videos.

These experimental results demonstrate the potential of our proposed method in the field of 3D asset generation. The ability to create based on both images and expressions is one of the attributes we find particularly compelling, and it also validates the feasibility of leveraging video to facilitate 3D creation. Unlike generation algorithms that support only a single modality, either image or text, simultaneously referencing image priors and supplementing with textual descriptions also holds practical utility for 3D content generation.

5.3.2 3D Lifting from Stylized Inputs

Droplet3D exhibits a relatively robust capability in lifting 2D content into 3D. Although the training data for Droplet3D-4M is entirely based on renderings from Objaverse-XL, which consists solely of object-level 3D files, Droplet3D still demonstrates a certain degree of robustness, particularly when the input images differ significantly from the training data distribution, such as stylized AIGC images like comics or sketches. We believe this capability may stem from its prior video training, which has endowed it with extensive general knowledge, making its 3D generation more versatile. This observation is particularly interesting, as it validates, to some extent, our hypothesis that video can facilitate 3D generation. Figs. 19 and 20 demonstrate the lifting capacity for sketches and comic-style images, respectively.

On the left of each figure are the stylized image inputs, while the right shows the surrounding perspective segments generated by our model based on it. For example, as shown in the first row of Fig. 20, our model can identify the body shape and posture of the comic girl and smoothly transform it into a 3D form without altering its style. We believe that this capability to support stylized images contributes to the generation of 3D assets. For instance, as shown in Fig. 19, it holds significant potential to form a well-developed 3D object based on just a few simple strokes from a designer or illustrator, whether it be a character (1st and 2nd rows) or an architecture (3rd row).

5.3.3 Textured Mesh Synthesis

We tested the potential of Droplet3D to create practical and usable 3D assets. Fig. 21 illustrates the results of generating textured meshes using Droplet3D. We employed Hunyuan3D-2[77] as the synthesis tool. Specifically, we utilized Hunyuan3D-2.0 DiT model for the geometric mesh synthetic and employed Hunyuan3D-2.1 painting model to generate textured meshes. We applied this algorithm to convert the multi-view images from Droplet3D into an industrially applicable representation. Compared to other mesh generation models that directly utilize single images, our pipeline enables the creation of a more diverse range of appearances based on long or short text descriptions.

5.3.4 3D Gaussian Splatting Points Synthesis

In addition, we also tested the potential to create 3D Gaussian splatting modality contents. Fig. 22 presents examples of 3D Gaussian splatting generated using Droplet3D. We used the native optimization-based approach[28] for this implementation. As can be observed, the re-rendered images maintain high quality, demonstrating the effectiveness of our proposed method. The generated surrounding viewpoints exhibit strong spatial consistency. Considering that we only employ the most basic reconstruction algorithm without any post-processing steps such as denoising, completion or camera pose estimation, we argue that a plausible reason is that our model can produce dense viewpoints, with the surrounding motion poses being relatively uniform and accurate.

5.3.5 Scene-level 3D Content Generation

Additionally, what excites us most is that, as illustrated in Fig. 23, our model can perform lifting on images with scene-level styles, which validates that our technical approach is fundamentally distinct from other currently popular native 3D generation methods, which are typically object-level. In contrast, Droplet3D has the potential to convert scene-level content into 3D. Fig. 23 sequentially demonstrates scene-level 3D generation for a manor, an island with lightning, a tranquil riverside at night, and a scene deep within a space station. Similar to a cinematic freeze-frame effect, our model can automatically lift these scenes to 3D, thereby reducing labor costs. From this perspective, it can also be regarded as another form of 3D asset creation. However, what is even more intriguing is that the training set, Droplet3D-4M, contains no scene-level samples. Therefore, this capability can be considered entirely inherited from its ancestral source, the DropletVideo video generation model.

Fig. 23 demonstrates the reconstruction of scene-level results using Gaussian splatting. It is interesting that we can further develop more applications based on these generated contents. For example, users can use these scene contents and combine it with the object-level point set. In terms of implementation, we can embed objects into the scene by controlling the scale, displacement, and rotation of both point sets. This intriguing application is intrinsically linked to Droplet3D’s potential for generating scene-level content.

6 Conclusion

In this paper, we propose a technical approach for 3D generation based on video backbone models, aiming to leverage the vast repository of video media materials to enhance the general knowledge capabilities of 3D generation, thereby facilitating the creation of 3D assets. On one hand, we introduce a large-scale 3D training dataset, termed Droplet3D-4M, comprising 4 million 3D models, accompanied by high-quality rendered surround-view videos and viewpoint-level textual descriptions. On the other hand, we present the technical framework of Droplet3D and achieve the training of 3D model weights using the DropletVideo, a video generation backbone network with spatio-temporal consistency. Concurrently, we design a text rewriting module and an image canonical viewpoint alignment module to refine the pipeline. Finally, we substantiate the value of our proposed approach through extensive visualization experiments. Furthermore, comprehensive quantitative experiments demonstrate its foundational capabilities and effectiveness. We hope that the ideas presented in this paper will contribute to the advancement of the 3D generation algorithm community, and the relevant dataset, code, and model weights associated with this work have been made open-source.