Towards virtual painting recolouring using Vision Transformer on X-Ray Fluorescence datacubes

TL;DR

This paper presents a novel pipeline for virtual painting recolouring using XRF data and Vision Transformers, employing synthetic data and deep embedding techniques to improve generalisation and visual quality.

Contribution

It introduces a new pipeline combining synthetic data generation, deep variational embedding, and Vision Transformers for virtual painting recolouring from XRF data.

Findings

Effective virtual recolouring demonstrated with high visual quality

Synthetic data and deep embedding improve model generalisation

Pipeline achieves promising results in visual quality metrics

Abstract

In this contribution, we define (and test) a pipeline to perform virtual painting recolouring using raw data of X-Ray Fluorescence (XRF) analysis on pictorial artworks. To circumvent the small dataset size, we generate a synthetic dataset, starting from a database of XRF spectra; furthermore, to ensure a better generalisation capacity (and to tackle the issue of in-memory size and inference time), we define a Deep Variational Embedding network to embed the XRF spectra into a lower dimensional, K-Means friendly, metric space. We thus train a set of models to assign coloured images to embedded XRF images. We report here the devised pipeline performances in terms of visual quality metrics, and we close on a discussion on the results.