Vision-Language Model Based Handwriting Verification

TL;DR

This paper investigates using Vision-Language Models like GPT-4o and PaliGemma for handwriting verification to improve interpretability and reduce training data needs, showing promising results but still lagging behind specialized models.

Contribution

It introduces the application of VLMs with visual question answering and 0-shot reasoning to handwriting verification, emphasizing interpretability and adaptability.

Findings

VLMs improve interpretability and require less training data.

CNN-based ResNet-18 outperforms VLMs in accuracy.

Achieved 84% accuracy on CEDAR AND dataset.

Abstract

Handwriting Verification is a critical in document forensics. Deep learning based approaches often face skepticism from forensic document examiners due to their lack of explainability and reliance on extensive training data and handcrafted features. This paper explores using Vision Language Models (VLMs), such as OpenAI's GPT-4o and Google's PaliGemma, to address these challenges. By leveraging their Visual Question Answering capabilities and 0-shot Chain-of-Thought (CoT) reasoning, our goal is to provide clear, human-understandable explanations for model decisions. Our experiments on the CEDAR handwriting dataset demonstrate that VLMs offer enhanced interpretability, reduce the need for large training datasets, and adapt better to diverse handwriting styles. However, results show that the CNN-based ResNet-18 architecture outperforms the 0-shot CoT prompt engineering approach with GPT-4o (Accuracy: 70%) and supervised fine-tuned PaliGemma (Accuracy: 71%), achieving an accuracy of 84% on the CEDAR AND dataset. These findings highlight the potential of VLMs in generating human-interpretable decisions while underscoring the need for further advancements to match the performance of specialized deep learning models.