# Scene Recomposition by Learning-based ICP

## TL;DR

This paper introduces a novel deep reinforcement learning-based ICP method for aligning CAD models to 3D scans, enabling accurate scene reconstruction without real scene annotations.

## Contribution

The paper presents Learning-based ICP, a new alignment approach trained solely on synthetic data that outperforms existing methods in real scene scenarios.

## Key findings

- Outperforms prior ICP methods in accuracy
- Effective on real-world cluttered scenes
- Does not require real scene annotations

## Abstract

By moving a depth sensor around a room, we compute a 3D CAD model of the environment, capturing the room shape and contents such as chairs, desks, sofas, and tables. Rather than reconstructing geometry, we match, place, and align each object in the scene to thousands of CAD models of objects. In addition to the fully automatic system, the key technical contribution is a novel approach for aligning CAD models to 3D scans, based on deep reinforcement learning. This approach, which we call Learning-based ICP, outperforms prior ICP methods in the literature, by learning the best points to match and conditioning on object viewpoint. LICP learns to align using only synthetic data and does not require ground truth annotation of object pose or keypoint pair matching in real scene scans. While LICP is trained on synthetic data and without 3D real scene annotations, it outperforms both learned local deep feature matching and geometric based alignment methods in real scenes. The proposed method is evaluated on real scenes datasets of SceneNN and ScanNet as well as synthetic scenes of SUNCG. High quality results are demonstrated on a range of real world scenes, with robustness to clutter, viewpoint, and occlusion.