MeshFIM: Local Low-Poly Mesh Editing via Fill-in-the-Middle Autoregressive Generation

TL;DR

MeshFIM introduces a fill-in-the-middle autoregressive framework for local low-poly mesh editing, enabling targeted regeneration conditioned on surrounding context, improving efficiency and control.

Contribution

The paper presents MeshFIM, a novel method for local mesh editing that enforces boundary constraints and preserves topology, with detailed ablations and practical applications.

Findings

MeshFIM outperforms baselines in mesh repair and refinement.

It enables interactive editing and automatic defect repair.

The method effectively enforces boundary conditions and suppresses overflow.

Abstract

Autoregressive (AR) models can generate high-quality low-poly meshes from point clouds, but they still operate in an all-or-nothing manner: when a local region is unsatisfactory, the entire mesh must be regenerated, wasting computation and destroying satisfactory mesh structure elsewhere. We introduce MeshFIM, a Fill-in-the-Middle (FIM) framework that regenerates a target region of a low-poly mesh conditioned on the surrounding context. MeshFIM addresses three mesh-specific challenges: enforcing exact attachment along the exposed boundary, preserving topological order in the context, and suppressing overflow beyond the intended region. It does so with five complementary design choices: boundary vertex markers, context positional embeddings, expanded context width, context augmentation, and a low-poly geometry encoder whose gated subtraction mechanism focuses generation on the missing region by leveraging the difference between the reference surface and the existing mesh. Detailed ablation studies are presented to show the effectiveness of every introduced component. Based on MeshFIM, we demonstrate two applications: interactive brush-based editing and automatic defect repair on low-poly mesh (see Figure 1). Last but not least, experiments show that MeshFIM outperforms a range of baselines in mesh refinement, mesh repair and whole mesh generation plus stitch-back scheme.