Towards Highly Expressive Machine Learning Models of Non-Melanoma Skin Cancer

TL;DR

This paper explores using discrete generative models and transformers to create high-resolution histological images of non-melanoma skin cancer and generate pathologist-like descriptions, enhancing interpretability and scientific insight.

Contribution

It introduces a novel approach combining VQ-GAN and transformers for detailed image reconstruction and natural language description in skin cancer pathology.

Findings

High-quality image reconstruction of IEC histology

Generation of natural language descriptions using pathologist terminology

Enhanced interpretability through interactive concept vectors

Abstract

Pathologists have a rich vocabulary with which they can describe all the nuances of cellular morphology. In their world, there is a natural pairing of images and words. Recent advances demonstrate that machine learning models can now be trained to learn high-quality image features and represent them as discrete units of information. This enables natural language, which is also discrete, to be jointly modelled alongside the imaging, resulting in a description of the contents of the imaging. Here we present experiments in applying discrete modelling techniques to the problem domain of non-melanoma skin cancer, specifically, histological images of Intraepidermal Carcinoma (IEC). Implementing a VQ-GAN model to reconstruct high-resolution (256x256) images of IEC images, we trained a sequence-to-sequence transformer to generate natural language descriptions using pathologist terminology. Combined with the idea of interactive concept vectors available by using continuous generative methods, we demonstrate an additional angle of interpretability. The result is a promising means of working towards highly expressive machine learning systems which are not only useful as predictive/classification tools, but also means to further our scientific understanding of disease.