Convolutional Vision Transformer for Cosmology Parameter Inference

TL;DR

This paper introduces a hybrid Convolutional Vision Transformer (CvT) that combines CNNs and ViTs to improve cosmological parameter inference from simulated data, outperforming pure ViT and CNN models in accuracy and efficiency.

Contribution

The study presents a novel hybrid CvT model that enhances cosmological parameter inference, demonstrating advantages over existing ViT and CNN approaches in accuracy and training efficiency.

Findings

CvT outperforms ViT and CNN in parameter constraints.

Pretraining on dark matter fields benefits halo field inference.

CvT achieves similar inference times to ViT despite more parameters.

Abstract

Parameter inference is a crucial task in modern cosmology that requires accurate and fast computational methods to handle the high precision and volume of observational datasets. In this study, we explore a hybrid vision transformer, the Convolution vision Transformer (CvT), which combines the benefits of vision transformers (ViTs) and convolutional neural networks (CNNs). We use this approach to infer the $\Omega_m$ and $\sigma_8$ cosmological parameters from simulated dark matter and halo fields. Our experiments indicate that the constraints on $\Omega_m$ and $\sigma_8$ obtained using CvT are better than ViT and CNN, using either dark matter or halo fields. For CvT, pretraining on dark matter fields proves advantageous for improving constraints using halo fields compared to training a model from the beginning. However, ViT and CNN do not show these benefits. The CvT is more efficient than ViT since, despite having more parameters, it requires a training time similar to that of ViT and has similar inference times. The code is available at \url{https://github.com/Yash-10/cvt-cosmo-inference/}.