LATFormer: Locality-Aware Point-View Fusion Transformer for 3D Shape Recognition

TL;DR

LATFormer is a novel transformer-based model that enhances 3D shape recognition by locally fusing point cloud and multi-view image features based on co-occurrence, improving discriminative power.

Contribution

The paper introduces LATFormer, a locality-aware fusion transformer that models local feature co-occurrence for better 3D shape understanding, unlike prior global fusion methods.

Findings

Outperforms existing methods on 3D shape benchmarks

Effectively fuses multi-scale local features from point clouds and images

Reduces redundancy by filtering low co-occurrence scores

Abstract

Recently, 3D shape understanding has achieved significant progress due to the advances of deep learning models on various data formats like images, voxels, and point clouds. Among them, point clouds and multi-view images are two complementary modalities of 3D objects and learning representations by fusing both of them has been proven to be fairly effective. While prior works typically focus on exploiting global features of the two modalities, herein we argue that more discriminative features can be derived by modeling ``where to fuse''. To investigate this, we propose a novel Locality-Aware Point-View Fusion Transformer (LATFormer) for 3D shape retrieval and classification. The core component of LATFormer is a module named Locality-Aware Fusion (LAF) which integrates the local features of correlated regions across the two modalities based on the co-occurrence scores. We further propose to filter out scores with low values to obtain salient local co-occurring regions, which reduces redundancy for the fusion process. In our LATFormer, we utilize the LAF module to fuse the multi-scale features of the two modalities both bidirectionally and hierarchically to obtain more informative features. Comprehensive experiments on four popular 3D shape benchmarks covering 3D object retrieval and classification validate its effectiveness.