# Multi-Modal Convolutional Parameterisation Network for Guided Image Inverse Problems

## TL;DR

This paper introduces a new neural network method for solving image restoration tasks by combining information from multiple image types.

## Contribution

The novel Multi-Modal Convolutional Parameterisation Network (MCPN) improves image inverse tasks by integrating shared and modality-specific networks.

## Key findings

- MCPN outperforms single-mode networks in image inpainting and super-resolution tasks.
- The proposed method effectively combines information from multiple image modalities.

## Abstract

There are several image inverse tasks, such as inpainting or super-resolution, which can be solved using deep internal learning, a paradigm that involves employing deep neural networks to find a solution by learning from the sample itself rather than a dataset. For example, Deep Image Prior is a technique based on fitting a convolutional neural network to output the known parts of the image (such as non-inpainted regions or a low-resolution version of the image). However, this approach is not well adjusted for samples composed of multiple modalities. In some domains, such as satellite image processing, accommodating multi-modal representations could be beneficial or even essential. In this work, Multi-Modal Convolutional Parameterisation Network (MCPN) is proposed, where a convolutional neural network approximates shared information between multiple modes by combining a core shared network with modality-specific head networks. The results demonstrate that these approaches can significantly outperform the single-mode adoption of a convolutional parameterisation network on guided image inverse problems of inpainting and super-resolution.