CLIPTime: Time-Aware Multimodal Representation Learning from Images and Text

TL;DR

CLIPTime is a novel multimodal framework based on CLIP that predicts both developmental stages and timestamps of fungal growth from images and text, enabling time-aware biological analysis without explicit temporal data.

Contribution

This work introduces CLIPTime, a multitask model that jointly predicts growth stages and timestamps, and provides a synthetic dataset for training and evaluation in biological time-series analysis.

Findings

Effectively models biological progression.

Produces interpretable, temporally grounded outputs.

Outperforms baseline methods in time-aware prediction.

Abstract

Understanding the temporal dynamics of biological growth is critical across diverse fields such as microbiology, agriculture, and biodegradation research. Although vision-language models like Contrastive Language Image Pretraining (CLIP) have shown strong capabilities in joint visual-textual reasoning, their effectiveness in capturing temporal progression remains limited. To address this, we propose CLIPTime, a multimodal, multitask framework designed to predict both the developmental stage and the corresponding timestamp of fungal growth from image and text inputs. Built upon the CLIP architecture, our model learns joint visual-textual embeddings and enables time-aware inference without requiring explicit temporal input during testing. To facilitate training and evaluation, we introduce a synthetic fungal growth dataset annotated with aligned timestamps and categorical stage labels. CLIPTime jointly performs classification and regression, predicting discrete growth stages alongside continuous timestamps. We also propose custom evaluation metrics, including temporal accuracy and regression error, to assess the precision of time-aware predictions. Experimental results demonstrate that CLIPTime effectively models biological progression and produces interpretable, temporally grounded outputs, highlighting the potential of vision-language models in real-world biological monitoring applications.