DecoDINO: 3D Human-Scene Contact Prediction with Semantic Classification

TL;DR

DecoDINO is a novel 3D human-scene contact prediction model that improves accuracy and semantic understanding over previous methods, using dual ViT encoders and semantic classification.

Contribution

It introduces a three-branch network with dual ViT encoders, semantic classification, and improved local reasoning, surpassing prior models like DECO in accuracy and semantic labeling.

Findings

Raises binary-contact F1 score by 7%

Halves geodesic error

Outperforms DAMON Challenge baseline

Abstract

Accurate vertex-level contact prediction between humans and surrounding objects is a prerequisite for high fidelity human object interaction models used in robotics, AR/VR, and behavioral simulation. DECO was the first in the wild estimator for this task but is limited to binary contact maps and struggles with soft surfaces, occlusions, children, and false-positive foot contacts. We address these issues and introduce DecoDINO, a three-branch network based on DECO's framework. It uses two DINOv2 ViT-g/14 encoders, class-balanced loss weighting to reduce bias, and patch-level cross-attention for improved local reasoning. Vertex features are finally passed through a lightweight MLP with a softmax to assign semantic contact labels. We also tested a vision-language model (VLM) to integrate text features, but the simpler architecture performed better and was used instead. On the DAMON benchmark, DecoDINO (i) raises the binary-contact F1 score by 7$\%$, (ii) halves the geodesic error, and (iii) augments predictions with object-level semantic labels. Ablation studies show that LoRA fine-tuning and the dual encoders are key to these improvements. DecoDINO outperformed the challenge baseline in both tasks of the DAMON Challenge. Our code is available at https://github.com/DavidePasero/deco/tree/main.