nvTorchCam: An Open-source Library for Camera-Agnostic Differentiable Geometric Vision

TL;DR

nvTorchCam is an open-source, camera-agnostic library built on PyTorch that simplifies applying deep learning algorithms across diverse camera models by abstracting key operations and supporting differentiability and GPU acceleration.

Contribution

It introduces a flexible, differentiable library that enables camera model-independent deep learning applications, facilitating transferability across various camera types.

Findings

Supports multiple camera models including pinhole, fisheye, and 360 panoramas

Enables transfer of trained models across different camera types

Provides GPU-accelerated, differentiable operations for deep learning

Abstract

We introduce nvTorchCam, an open-source library under the Apache 2.0 license, designed to make deep learning algorithms camera model-independent. nvTorchCam abstracts critical camera operations such as projection and unprojection, allowing developers to implement algorithms once and apply them across diverse camera models--including pinhole, fisheye, and 360 equirectangular panoramas, which are commonly used in automotive and real estate capture applications. Built on PyTorch, nvTorchCam is fully differentiable and supports GPU acceleration and batching for efficient computation. Furthermore, deep learning models trained for one camera type can be directly transferred to other camera types without requiring additional modification. In this paper, we provide an overview of nvTorchCam, its functionality, and present various code examples and diagrams to demonstrate its usage. Source code and installation instructions can be found on the nvTorchCam GitHub page at https://github.com/NVlabs/nvTorchCam.