Animating the Past: Reconstruct Trilobite via Video Generation

TL;DR

This paper presents a novel automatic text-to-video method for reconstructing trilobite behavior from fossils, enhancing visual realism and consistency to aid scientific research and education.

Contribution

It introduces a prompt learning approach using large language models and reward-based fine-tuning to improve trilobite video generation from fossil images.

Findings

Generated videos show higher realism than baselines

Method effectively captures trilobite visual details

Enhances scientific and educational visualization

Abstract

Paleontology, the study of past life, fundamentally relies on fossils to reconstruct ancient ecosystems and understand evolutionary dynamics. Trilobites, as an important group of extinct marine arthropods, offer valuable insights into Paleozoic environments through their well-preserved fossil records. Reconstructing trilobite behaviour from static fossils will set new standards for dynamic reconstructions in scientific research and education. Despite the potential, current computational methods for this purpose like text-to-video (T2V) face significant challenges, such as maintaining visual realism and consistency, which hinder their application in science contexts. To overcome these obstacles, we introduce an automatic T2V prompt learning method. Within this framework, prompts for a fine-tuned video generation model are generated by a large language model, which is trained using rewards that quantify the visual realism and smoothness of the generated video. The fine-tuning of the video generation model, along with the reward calculations make use of a collected dataset of 9,088 Eoredlichia intermedia fossil images, which provides a common representative of visual details of all class of trilobites. Qualitative and quantitative experiments show that our method can generate trilobite videos with significantly higher visual realism compared to powerful baselines, promising to boost both scientific understanding and public engagement.