Evaluation of Embedding-Based and Generative Methods for LLM-Driven Document Classification: Opportunities and Challenges

TL;DR

This paper compares embedding-based and generative models for classifying geoscience documents, showing generative VLMs with CoT prompting outperform embedding models in zero-shot accuracy.

Contribution

It provides a comprehensive benchmark analysis highlighting the advantages of generative VLMs over embedding models in document classification tasks.

Findings

Generative VLMs like Qwen2.5-VL achieve 82% zero-shot accuracy.

Embedding models like QQMM achieve 63% accuracy.

Fine-tuning improves VLM performance but is sensitive to data imbalance.

Abstract

This work presents a comparative analysis of embedding-based and generative models for classifying geoscience technical documents. Using a multi-disciplinary benchmark dataset, we evaluated the trade-offs between model accuracy, stability, and computational cost. We find that generative Vision-Language Models (VLMs) like Qwen2.5-VL, enhanced with Chain-of-Thought (CoT) prompting, achieve superior zero-shot accuracy (82%) compared to state-of-the-art multimodal embedding models like QQMM (63%). We also demonstrate that while supervised fine-tuning (SFT) can improve VLM performance, it is sensitive to training data imbalance.