Selective Style Transfer for Text

TL;DR

This paper introduces methods for applying style transfer to text images, enabling selective style transfer to specific pixels, and demonstrates its effectiveness in data augmentation for scene text detection.

Contribution

It proposes two architectures for selective style transfer in text images and evaluates their use as a data augmentation technique for improving scene text detection.

Findings

Selective style transfer is feasible across various text domains.

Proposed architectures enable transfer to desired image pixels.

Data augmentation with style transfer boosts text detector performance.

Abstract

This paper explores the possibilities of image style transfer applied to text maintaining the original transcriptions. Results on different text domains (scene text, machine printed text and handwritten text) and cross modal results demonstrate that this is feasible, and open different research lines. Furthermore, two architectures for selective style transfer, which means transferring style to only desired image pixels, are proposed. Finally, scene text selective style transfer is evaluated as a data augmentation technique to expand scene text detection datasets, resulting in a boost of text detectors performance. Our implementation of the described models is publicly available.

Tables

Table 1. TABLE I : Results (F-score) of the EAST text detector with different training data, evaluated on ICDAR 2015 Challenge 4 and ICDAR 2013 Challenge 2. *Result reported on the original EAST github.

Training Dataset	Testing dataset	Augmentations	F
ICDAR 13	ICDAR 13	-	70.97
ICDAR 13	ICDAR 13	1 styles per image	74.55
ICDAR 13	ICDAR 13	4 styles per image	75.29
ICDAR 13+15	ICDAR 15	-	80.83*
ICDAR 15	ICDAR 15	-	78.74
ICDAR 15	ICDAR 15	1 style per image	80.60
ICDAR 15	ICDAR 15	4 styles per image	81.83
COCO-Text	ICDAR 15	-	68.05
COCO-Text	ICDAR 15	1 style per image	69.66
COCO-Text	ICDAR 15	4 styles per image	70.71

Equations

$$\begin{array}{rl}& p=\varphi (c),\\ & F_c={\gamma }^m(c),F_s={\gamma }^n(s),\\ & F_{pm}={\gamma }^m(p),F_{pn}={\gamma }^n(p),\\ & L_c=\sum _k(F_{ck}-F_{pmk}{)}^2,\\ & G_s=F_s\cdot F_s^T,G_p=F_{pn}\cdot F_{pn}^T\\ & L_s=\sum _k(G_{sk}-G_{pnk}{)}^2,\\ & L_{total}={\lambda }_1{L}_c+{\lambda }_2{L}_s.\end{array}$$

$$\begin{array}{rl}& P_t=\delta (c),\\ & p_{text}=P_t\odot p+(1-P_t)\odot c\end{array}$$

$$\begin{array}{rl}& p=\varphi (c),\\ & {\hat{p}}_{text}=\eta (c),\\ & {\theta }_s=p\odot M,\\ & {\theta }_c=c\odot (1-M).\end{array}$$

$$\begin{gathered} L={\lambda }_1\sum ({\hat{p}}_{text}\odot M-{\theta }_s{)}^2+ \\ {\lambda }_2\sum ({\hat{p}}_{text}\odot (1-M)-{\theta }_c{)}^2. \end{gathered}$$

Full text

00footnotetext: *∗*Both authors contributed equally to this work.

I Introduction

Style transfer is the task of combining the style of one image with the content of another image. Although the content of an image can be defined by the objects and the general scenery, the style of an image is not well defined. The style can be understood as the brush stroke of a painting, the color distribution, certain dominant forms and shapes or even a combination of all the above[7]. Previous style transfer works have focused on transferring paintings styles, where the features to be transferred encode the brush strokes, the cubist patterns or the color palette achieving fascinating results[7, 13, 6].

However, text characters are very particular objects for which the common understanding of content and style cannot be adopted. Instead we define the style of the text as the shape, color and background of the characters and the content as the transcription of the text. In this work, we devise two architectures that learn the features that encode a certain text style and are able to transfer them to other text instances while preserving their content. We thus recast the ideas of style transfer, previously applied mostly to paintings, to the text domain.

Specifically, our method is able to automatically change the style of text regions in natural scene images, generating realistic images with the same textual content but with different text styles. In machine printed text images, we are able to train models that stimulate a change of the text font. In handwritten text, we are able to transfer the writing style of a particular writer to another.

Possible applications of text style transfer include text stylization for augmented reality systems, or text font normalization in order to recast the visual appearance of any text instance to a canonical style in order to ease the different steps of a text recognition pipeline.

In particular, we will demonstrate that such an approach is useful as a data augmentation technique (see Figure 1), in order to deal with the problem of annotated data scarcity. One can transfer the styles found in a particular dataset to much bigger available annotated datasets, in order to train better text detectors or recognizers.

The contributions of this work are as follows:

• •

We demonstrate that existing style transfer pipeline can be used for text style transfer.

• •

We extend the prior works to selective style transfer for text, devising two architectures: two-stage and end-to-end, that allow to stylize only the text regions in an image.

• •

We use text style transfer as a data augmentation technique for the scene text detection task, and provide results that show a boost in the detection performance.

The rest of the paper is organized as follows: in the related work section, we overview the style transfer and the text stylizing literature. In the methodology section, we first define the style transfer task and then present the proposed two-stage and end-to-end architectures for selective text style transfer. In the results section, we explain the models we have trained in different text domains, show qualitative results and discuss their performance. Finally, in the data augmentation section, we show how text style transfer can be used to boost a text detector performance.

II Related Work

Gatys et al. [7] propose a method using a pretrained VGG network [22] for style transfer by extrapolating a randomly generated image to a stylized image. The stylized image is obtained by computing the backward propagation on the resulting pixel values which is computationally demanding. Johnson et al. [13] recast the problem as an image transformation task, where a single, fixed learnt painting style is applied to an arbitrary image. A CNN is trained to alter a corpus of content images to match the style of a painting, eventually allowing to stylize images in real time. Simultaneously, Ulyanov at al. [23] introduced the idea of Instance Normalization which is a modified version of Batch Normalization to have computationally less demanding models. A drawback of these works[13, 23] is that an independent model has to be trained for each source style. Dumolin et al. [6] overcomes it by proposing a single CNN that can learn to transfer different source styles (up to 32 in their experiments), allowing to generate images with combined styles.

An appealing direction in the style transfer literature is to apply the style according to semantic segmentation. Li et al.[17] use a Markov Random Field over given semantic maps to decide which patch of the image to stylize. Luan et al.[20] extend this idea to doodles with manual annotation. A final extension comes from Zhao et al.[26], in which they use two models: one for generating soft masks and the other for style transfer. All these works, however, apply the style transfer to the whole image. Conceptually, the closest to our work is that of Gatys et al.[8] where they propose a method to control perceptual factors such as color, luminance and spatial location and scale. In particular, spatial control refers to applying the style to only masked areas which can be sky areas, balls, houses, trees, etc. The proposed method is guided Gram matrices in which they use masking for the calculation of Gram matrices over CNN features at specific image regions.

Although style transfer has not been applied to text, other works have targeted the task of changing text style with different approaches. Liu et al. [19] proposed a pipeline to transform scene text into machine printed text within a scene text recognition model. Abe et al. [1] proposed a Generative Adversarial Network model to create new machine printed text fonts. Aksan et al. [2] propose a generative model to disentangle content and style of handwritten text represented as temporally ordered strokes, and apply it to handwriting synthesis and style transfer. More related to our work, Ankan et al. [16] focus on font to font translation in images of printed documents using a GAN architecture. Also, Azadi et al. [3] propose a conditional GAN to style machine printed text to more complex scene text fonts, learning each character style independently.

Our work, to the best of our knowledge, is the first one exploring the performance of style transfer models on the text style transfer task as well as having an end-to-end model that can perform spatial control without any external information.

III Methodology

We will first explain the details of the model on which we base our implementation. Afterwards, we will explain the details of our architectures, a two-stage and an end-to-end model that apply style transformations to only text areas in the image.

III-A Style Transfer

We use the model proposed by Dumoulin et al.[6] as our baseline. The style transfer task is usually defined as finding an image $p$ which is produced by an encoder-decoder image transformation network, whose content is similar to a source content image $c$ but whose style is similar to a source style image $s$. A key point of style transfer is the definition of both content and style. In [6], two images are considered to have a similar content if their high-level features extracted by a trained classifier are close in Euclidean distance. On the other hand, two images are similar in style if their gram matrices of low-level features as extracted by a trained classifier are close under the Frobenius norm.

More formally, let $\phi$ be the transformer network and $\gamma^{l}$ the output of the $l^{th}$ layer of a CNN pretrained on ImageNet[12]. In our case $\gamma$ is the VGG-16[22]. The training process is as follows: we initially forward the content image $c$ through the transformer network $\phi$ to obtain the stylized image $p$. All images $c$, $s$, $p$ are then forwarded through $\gamma$, and features for them are extracted: correspondingly, $F_{c}$ for the content image from the $m^{th}$ layer, $F_{s}$ for the style image from $n^{th}$ layer and, $F_{pm}$, $F_{pn}$ for the stylized image from both $m^{th},n^{th}$ layers where $m\geq n$. The content loss $L_{c}$ is defined as the mean squared error between $F_{c}$ and $F_{p}$. The style loss $L_{s}$ is computed as the mean squared error between corresponding Gram matrices $G_{s}$ and $G_{p}$ of the features $F_{s}$ and $F_{p}$. The final loss $L_{total}$ that directs the model training is a weighted average of the content and the style losses. In summary:

[TABLE]

III-B Selective Style Transfer for Text

Selective style transfer refers to automatically detecting the relevant areas in the image (in our case, areas where text is present) and restricting the application of style to the detected areas only, leaving the rest of the image unchanged. Accordingly, we design and describe two models that can perform selective style transfer for text.

III-B1 Two-stage architecture

To stylize only textual areas of the images, we exploit TextFCN 222https://github.com/gombru/TextFCN [4, 5] which is a text detector that infers the probability of each pixel belonging to a textual area. To transfer a text style to an input image, we first stylize the whole image. Then, we compute the pixel-level heatmap of the original image with TextFCN. To generate the final image where the style is transferred only to textual areas, we do a blending of the original image and the stylized image weighted with the TextFCN heatmap (see Figure 2). This procedure allows to obtain realistic images, ensuring that non-textual areas are kept unchanged.

More formally, given a stylized image $p$ and a pre-trained TextFCN $\delta$, we process the content image $c$ with $\delta$ to obtain the per-pixel text probability map $P_{t}$. Then we get only textual areas of the stylized image $p$ by taking its Hadamard product with $P_{t}$, and do the same with $c$ and $1-P_{t}$ to get the content of non-textual areas. We sum up the results to get the final image $p_{text}$:

[TABLE]

III-B2 End-to-end Architecture

In the style transfer literature, pre-computed masks of image regions have been extensively used to apply different styles in different regions[8, 20, 17, 26]. However, masks are used to constrain the image transformation network output as we do with the two-stage architecture, or for gradient masking, and none of the existing works is capable of learning the masks along with the style information. To this end, combined with the purpose of reducing the computational complexity, we create a novel end-to-end architecture that is capable of performing selective style transfer on text without needing any text detector. Our model is inspired by the distillation strategy from [11]. The basic idea of distillation is to pass the learnt information of various networks, which are able to solve different tasks, into a single model. In our case, we combine the image style transformation network with the text detector. We take the pretrained image style transformation network and the ground truth annotations for the text to train a randomly initialized image transformation network with mean squared error loss (see Figure 3).

More formally, let $\phi$ be the pretrained image style transformation network, $M$ the masks for the text regions where $M_{ij}\in\{0,1\}$ and, $\eta$ the same network as $\phi$ but randomly initialized. We first obtain the stylized image $p$ forwarding the content image $c$ though $\phi$. We then obtain $\hat{p}_{text}$ as the output of $\eta$ after feeding it with the content image $c$. To get the ground truth for content and for style, $\theta_{c}$ and $\theta_{s}$ respectively, we apply the Hadamard product:

[TABLE]

We use mean squared error as our loss to train the selective style transfer net $\eta$. There are two key points in our loss calculation. First of all, we need to apply the mask $M$ for text regions, and $1-M$ for content regions to $\hat{p}_{text}$ to make sure our model learns to differentiate between text and background. Secondly, since text regions in the image are significantly smaller compared to background, we weight loss contributions of text and background pixels with two parameters $\lambda_{1}$, $\lambda_{2}$, with $\lambda_{1}>\lambda_{2}$. The loss is defined as:

[TABLE]

IV Experiments

To explore the capabilities of text style transfer in various text domains, we train 3 models, namely, a model to transfer scene text styles, a model to transfer machine printed text fonts styles, and a model to transfer handwritten styles. Figure 4 shows some of the source styles used. In this section, we explain the implementation and training details for each model, and give qualitative results.

IV-A Scene Text

We trained our baseline style transfer model for two-stage architecture with 34 scene text styles from COCO-Text dataset [24] using cropped word images as source styles and ImageNet [12] as the training dataset. Then, we trained our end-to-end model for selective text style transfer using the COCO-Text [24] dataset (only the images containing legible text). To train the end-to-end model in the distillation fashion, we set parameters $\lambda_{1}=100,\lambda_{2}=1$ and trained for $77$ epochs. After that, we got images where the style was correctly transferred to text, but were quite blurry and dark. Then we trained for $50$ epochs with equal weight for textual and not textual areas to balance the style and content loss.

Figure 5 shows results of the scene text model applied to COCO-Text scene images, using both the two-stage architecture and the end-to-end architecture to get the selective text style transfer results. The performance is appealing, transferring the source styles with high fidelity in both character shapes and colors to a wide diversity of scene texts. The text content is preserved quite well in most images, and only in some cases where the task is very complex due to the original text size or tangled style the result is illegible. Both the weight blending with TextFCN and the end-to-end architecture allow getting realistic images that are very useful as data augmentation (see Section V). This realistic results have also a huge potential in artistic applications, such as graphic design. Graphic designers could quickly visualize how a given text style fits in a existing scene text. Moreover, the model can perform style transfer averaging several styles, which allows to create new styles.

IV-B Machine Printed Text

We train our baseline model with 8 machine printed source text fonts: Arial, Times, Lubster, Corsiva, Caveat, Pacifico, Consolas and Syncopate and Imagenet [12] training images. Figure 6 shows results of the model applied to machine text. It transfers successfully the main features of the source font style, such as line width, text orientation, and main font character style. However, it fails transferring the specific styles of some characters, and the final output is influenced by the initial image.

IV-C Handwritten text

The baseline model is trained with 8 styles from different writers, using images from the IAM dataset [21] as source styles and the ImageNet [12] dataset as training images. Figure 6 shows results of the model applied to IAM dataset images. The model transfers correctly the main features of the text, as the tight characters and the thick stroke of the style in the first column, and the elongated and italic style of the writer in the second column. However, it fails on transferring more fine-grained characteristics of the source writer style, and some words of the resulting text are blurry.

IV-D Cross Domain

In this section, we go one step further and test the capability of our text style transfer models to transfer style to images of other text domains.

IV-D1 Machine Text to Scene Text

Transferring scene text styles to machine printed text has a huge potential in augmented reality scenarios and as a data augmentation technique, generating synthetic images with a given text style but different text content. Figure 7 shows results of styling machine printed text with the scene text model. The model successfully transfers scene text style to machine printed text with high fidelity.

Figure 9 shows some content augmentation results, where Arial text has been stylized with the destination scene text font, and the stylized text has been inserted in the image manually.

IV-D2 Handwritten to Scene Text

Figure 7 shows results of styling handwritten text with the scene text model. This task is quite complex, since the scene text tends to be thick and detached while handwritten text is tangled formed by tangled thin strokes. However, the scene text style transfer model successfully transfers some style features to it keeping it legible, and results are a combination of the transferred scene text style and the original handwritten style.

IV-D3 Scene Text to Machine Text

A model capable to convert any scene text to machine printed text would be a nice tool to improve scene text understanding pipelines. The machine printed text model allows us to do so, as shown in Figure 10. It correctly transfers the machine text source style if the scene text is simple, but it fails when scene text has a complex font, is too small or rotated. Note that the artifacts in those images are due to noisy high responses of the text detector.

IV-D4 Handwritten to Machine Text

Converting handwritten text to machine printed text could be very useful in a handwritten text understanding pipeline. Our machine text model transfers style features from machine fonts to handwritten text, but it breaks the content, resulting in illegible images, as shown in Figure 10.

IV-D5 Machine Text to Handwritten

Converting machine text to handwritten text can be a great tool to generate synthetic data to train handwritten text understanding models. However, our handwritten text model fails transferring the font styles to machine text, as shown in Figure 8. It only achieves to copy some general handwritten style features to some machine text fonts closer to handwritten styles, like Caveat.

V Data Augmentation

In this section we include experiments that demonstrate the usefulness of the proposed selective scene text style transfer as a data augmentation tool to improve text detectors’ performance. The reason for using style transfer as a data augmentation technique is twofold. According to the recent seminal work by Geirhos et. al. [9], CNNs are strongly biased towards recognising textures rather than shapes. To overcome the biases, they have made use of neural style transfer as an augmentation technique which resulted not only in overcoming the biases that a CNN has but also improving results on ImageNet classification[12]. Thus, data augmentation using selective style transfer is expected to help overcome similar biases of the text recognizer. A second rationale is that some datasets do not offer the necessary amount of images to efficiently train large neural networks. Style transfer in such cases can artificially increase the size of the datasets by combining different styles and contents. This is especially true for ICDAR 2013 and ICDAR 2015.

Moreover, the proposed data augmentation technique has a clear benefit compared with other methods to generate synthetic data [10, 25]. The generated images contain a text with a different visual style, but the text appears in the same place as in the original images, which makes the text position in the image realistic while preserving the content of the text, which makes the transcription match the scene semantics.

In the experiments, we use the consolidated and widely used EAST [27] text detector333https://github.com/argman/EAST. We consider the following datasets:

• •

ICDAR 2013 [15]: the dataset contains $229$ training images and $229$ test images that capture focused text on sign boards, posters, etc.

• •

ICDAR 2015 [14]: the dataset contains $1000$ training images and $500$ testing images with incidental scene text, which means text that appears in the scene without the user focusing on it.

• •

COCO-Text [24]: the dataset contains $63k$ images from the COCO [18] dataset with text regions annotated.

For these experiments, we trained an additional scene text style transfer model using $96$ different styles from the ICDAR 2015 training dataset, and used the two-stage architecture to perform selective text style transfer. We augment ICDAR 2013, ICDAR 2015 and COCO-Text datasets using the two-stage model with $1$ or $4$ random additional styles per image. Figure 1 shows some of the ICDAR 2015 resulting augmentations. We train the EAST text detector on the augmented and regular datasets. Stylizing COCO-Text with ICDAR 2015 training styles, allows to get a COCO-Text dataset closer to the target testing data, which is ICDAR 2015 testing set. To evaluate the trained models, we use the Robust Reading Competition framework (ICDAR 2015 Challenge 4: Incidental Scene Text Localization task and ICDAR 2013 Challenge 2: Focused Scene Text). Results in Table I show that text style transfer is a useful data augmentation technique, achieving an improvement in F-Score performance between $2$-$4$% in all the setups just using $1$ or $4$ augmentations per image.

The boost in performance confirms that neural style transfer offers an advantageous and practical data augmentation technique that works both within the datasets as well as cross dataset, domain adaption scenarios.

VI Conclusions

We have shown that a style transfer model is able to learn text styles as the characters shapes, line style, and colors, and to transfer it to an input text preserving the original characters. We have explored the performance of text style transfer in 3 text modalities: scene text, machine printed text and handwritten text and in cross-modal scenarios, proving the usefulness of text style transfer as a data augmentation technique to train scene text detectors. Cross-modal experiments show the potential of this pipeline in virtual reality scenarios to style an arbitrary text with a given scene style, and the realism of the generated images suggest that the pipeline could be useful in artistic applications, such as graphic design. We open the field for further research in different directions, such as data augmentation for scene text detection or recognition or handwritten writer identification.

The proposed selective style transfer two-stage and end-to-end architectures allow to automatically get realistic images where only text has been styled. Furthermore, the end-to-end selective style transfer pipeline can be applied in other style transfer tasks besides text. We provide PyTorch style transfer code, based on Google’s TensorFlow Magenta implementation, including the end-to-end selective style transfer implementation, and text style transfer trained models for both frameworks.

Acknowledgments

This work was supported by projects TIN2017-89779-P, Marie-Curie (712949 TECNIOspring PLUS), aBSINTHE (Fundacion BBVA 2017), Doctorats Industrials (AGAUR), the CERCA Programme / Generalitat de Catalunya, NVIDIA Corporation and a UAB PhD scholarship.