Hierarchical Process Reward Models are Symbolic Vision Learners

TL;DR

This paper introduces a novel self-supervised symbolic auto-encoder for diagram understanding that encodes geometric primitives and their relationships, combining hierarchical reinforcement learning with neuro-symbolic reasoning for improved interpretability and performance.

Contribution

It proposes a hierarchical process reward model and stabilization techniques for reinforcement learning in symbolic vision, enhancing diagram reconstruction and reasoning capabilities.

Findings

Achieved 98.2% reduction in MSE for diagram reconstruction.

Surpassed GPT-4o by 0.6% on chart reconstruction with a 7B model.

Improved perception and reasoning benchmarks by +13% and +3%, respectively.

Abstract

Symbolic computer vision represents diagrams through explicit logical rules and structured representations, enabling interpretable understanding in machine vision. This requires fundamentally different learning paradigms from pixel-based visual models. Symbolic visual learners parse diagrams into geometric primitives-points, lines, and shapes-whereas pixel-based learners operate on textures and colors. We propose a novel self-supervised symbolic auto-encoder that encodes diagrams into structured primitives and their interrelationships within the latent space, and decodes them through our executable engine to reconstruct the input diagrams. Central to this architecture is Symbolic Hierarchical Process Reward Modeling, which applies hierarchical step-level parsing rewards to enforce point-on-line, line-on-shape, and shape-on-relation consistency. Since vanilla reinforcement learning exhibits poor exploration in the policy space during diagram reconstruction; we thus introduce stabilization mechanisms to balance exploration and exploitation. We fine-tune our symbolic encoder on downstream tasks, developing a neuro-symbolic system that integrates the reasoning capabilities of neural networks with the interpretability of symbolic models through reasoning-grounded visual rewards. Evaluations across reconstruction, perception, and reasoning tasks demonstrate the effectiveness of our approach: achieving a 98.2% reduction in MSE for geometric diagram reconstruction, surpassing GPT-4o by 0.6% with a 7B model on chart reconstruction, and improving by +13% on the MathGlance perception benchmark, and by +3% on MathVerse and GeoQA reasoning benchmarks.