# Convpaint—Interactive pixel classification using pretrained neural networks

## TL;DR

Convpaint is a tool that uses pretrained neural networks for fast and accurate pixel classification in images, making it easier to analyze complex data.

## Contribution

Convpaint introduces a modular framework combining pretrained models with fast classifiers for interactive segmentation across diverse imaging tasks.

## Key findings

- Convpaint enables accurate segmentation with minimal annotations and fast training times.
- The tool supports multichannel, 3D, and time-series data using vision transformers and CNNs.
- Convpaint integrates into napari for interactive bioimage analysis workflows.

## Abstract

We present Convpaint, a universal computational framework for interactive pixel classification. Convpaint uses pretrained convolutional neural networks (CNNs), vision transformers (ViTs), or classical filter banks for feature extraction in combination with fast-to-train machine learning (ML) classifiers to enable easy segmentation across a wide variety of tasks. By integrating ViT-based features, Convpaint extends traditional pixel classification to image domains that require rich semantic understanding. Convpaint’s modular design allows users to rapidly switch between feature extractors, balancing speed, spatial accuracy, and semantic depth based on the specific dataset. Available within the Python-based napari software ecosystem, Convpaint integrates seamlessly with other plugins into image processing pipelines, which we demonstrate with example workflows across different data modalities, from subcellular to cellular to animal scale.

•Convpaint enables rapid pixel classification using pretrained neural networks•Vision transformers extend segmentation to tasks requiring semantic understanding•Supports multichannel, 3D, and time-series data•Integrates into napari for interactive, code-free bioimage analysis workflows

Convpaint enables rapid pixel classification using pretrained neural networks

Vision transformers extend segmentation to tasks requiring semantic understanding

Supports multichannel, 3D, and time-series data

Integrates into napari for interactive, code-free bioimage analysis workflows

We needed to perform real-time segmentation during feedback-control microscopy experiments, which requires rapid model training on unseen data before starting the experiment. We also required seamless integration into our Python-based microscope control pipeline. Existing solutions fell short: classical interactive machine learning approaches train quickly but lack semantic understanding for complex tasks, while deep learning methods require lengthy training on large datasets acquired previously. This motivated us to develop Convpaint, bridging both approaches to enable accurate segmentation with minimal annotations and fast training times. The method proved so effective that we began applying it to other image analysis tasks beyond feedback microscopy, which prompted us to develop a user-friendly interface and share it with the broader scientific community.

Hinderling et al. present Convpaint, a napari plugin that repurposes pretrained deep learning models for interactive pixel classification. By combining convolutional neural networks and vision transformers with fast machine learning classifiers, Convpaint enables accurate segmentation across diverse imaging modalities with minimal annotations and rapid training times.