HoneyBee: A Scalable Modular Framework for Creating Multimodal Oncology Datasets with Foundational Embedding Models

TL;DR

HoneyBee is an open-source framework that integrates diverse biomedical data modalities to generate unified patient embeddings, enabling improved oncology predictions and analyses across multiple cancer types.

Contribution

It introduces a scalable, modular framework that combines multimodal biomedical data with foundation models for comprehensive oncology datasets.

Findings

Clinical embeddings achieved 98.5% accuracy in cancer classification.

Multimodal fusion improved survival prediction accuracy.

General-purpose language models outperformed specialized medical models on clinical text.

Abstract

HONeYBEE (Harmonized ONcologY Biomedical Embedding Encoder) is an open-source framework that integrates multimodal biomedical data for oncology applications. It processes clinical data (structured and unstructured), whole-slide images, radiology scans, and molecular profiles to generate unified patient-level embeddings using domain-specific foundation models and fusion strategies. These embeddings enable survival prediction, cancer-type classification, patient similarity retrieval, and cohort clustering. Evaluated on 11,400+ patients across 33 cancer types from The Cancer Genome Atlas (TCGA), clinical embeddings showed the strongest single-modality performance with 98.5% classification accuracy and 96.4% precision@10 in patient retrieval. They also achieved the highest survival prediction concordance indices across most cancer types. Multimodal fusion provided complementary benefits for specific cancers, improving overall survival prediction beyond clinical features alone. Comparative evaluation of four large language models revealed that general-purpose models like Qwen3 outperformed specialized medical models for clinical text representation, though task-specific fine-tuning improved performance on heterogeneous data such as pathology reports.