From a Bird's Eye View to See: Joint Camera and Subject Registration without the Camera Calibration

TL;DR

This paper introduces an end-to-end framework for multi-view camera and subject registration in bird's eye view without requiring camera calibration, using only RGB images from first-person views.

Contribution

It presents a novel approach combining view-transform detection, geometric transformation for camera registration, and information aggregation, along with a new synthetic dataset.

Findings

Effective in localizing and orienting subjects in BEV

Accurately estimates camera positions and directions

Outperforms baseline methods on synthetic data

Abstract

We tackle a new problem of multi-view camera and subject registration in the bird's eye view (BEV) without pre-given camera calibration. This is a very challenging problem since its only input is several RGB images from different first-person views (FPVs) for a multi-person scene, without the BEV image and the calibration of the FPVs, while the output is a unified plane with the localization and orientation of both the subjects and cameras in a BEV. We propose an end-to-end framework solving this problem, whose main idea can be divided into following parts: i) creating a view-transform subject detection module to transform the FPV to a virtual BEV including localization and orientation of each pedestrian, ii) deriving a geometric transformation based method to estimate camera localization and view direction, i.e., the camera registration in a unified BEV, iii) making use of spatial and appearance information to aggregate the subjects into the unified BEV. We collect a new large-scale synthetic dataset with rich annotations for evaluation. The experimental results show the remarkable effectiveness of our proposed method.