OracleSage: Towards Unified Visual-Linguistic Understanding of Oracle Bone Scripts through Cross-Modal Knowledge Fusion

TL;DR

OracleSage introduces a cross-modal framework combining hierarchical visual understanding and graph-based semantic reasoning to improve recognition of ancient Oracle bone scripts, aiding archaeological and linguistic research.

Contribution

The paper presents a novel unified model integrating visual and semantic analysis specifically tailored for complex ancient scripts, with a new dataset and significant performance improvements.

Findings

Outperforms existing vision-language models on OBS recognition

Provides a new dataset with comprehensive annotations for ancient scripts

Establishes a new paradigm for ancient text interpretation

Abstract

Oracle bone script (OBS), as China's earliest mature writing system, present significant challenges in automatic recognition due to their complex pictographic structures and divergence from modern Chinese characters. We introduce OracleSage, a novel cross-modal framework that integrates hierarchical visual understanding with graph-based semantic reasoning. Specifically, we propose (1) a Hierarchical Visual-Semantic Understanding module that enables multi-granularity feature extraction through progressive fine-tuning of LLaVA's visual backbone, (2) a Graph-based Semantic Reasoning Framework that captures relationships between visual components and semantic concepts through dynamic message passing, and (3) OracleSem, a semantically enriched OBS dataset with comprehensive pictographic and semantic annotations. Experimental results demonstrate that OracleSage significantly outperforms state-of-the-art vision-language models. This research establishes a new paradigm for ancient text interpretation while providing valuable technical support for archaeological studies.