Hangul Fonts Dataset: a Hierarchical and Compositional Dataset for Investigating Learned Representations

TL;DR

This paper introduces the Hangul Fonts Dataset, a benchmark with known hierarchical and compositional structure, to evaluate and improve deep networks' ability to learn such properties in complex, naturalistic data.

Contribution

The paper presents a novel dataset with explicit hierarchical and compositional structure, enabling precise analysis of deep networks' learned representations.

Findings

Supervised deep networks capture some geometric hierarchy.

Unsupervised methods show limited hierarchy and compositionality.

The dataset reveals shortcomings of current models in learning structure.

Abstract

Hierarchy and compositionality are common latent properties in many natural and scientific datasets. Determining when a deep network's hidden activations represent hierarchy and compositionality is important both for understanding deep representation learning and for applying deep networks in domains where interpretability is crucial. However, current benchmark machine learning datasets either have little hierarchical or compositional structure, or the structure is not known. This gap impedes precise analysis of a network's representations and thus hinders development of new methods that can learn such properties. To address this gap, we developed a new benchmark dataset with known hierarchical and compositional structure. The Hangul Fonts Dataset (HFD) is comprised of 35 fonts from the Korean writing system (Hangul), each with 11,172 blocks (syllables) composed from the product of initial consonant, medial vowel, and final consonant glyphs. All blocks can be grouped into a few geometric types which induces a hierarchy across blocks. In addition, each block is composed of individual glyphs with rotations, translations, scalings, and naturalistic style variation across fonts. We find that both shallow and deep unsupervised methods only show modest evidence of hierarchy and compositionality in their representations of the HFD compared to supervised deep networks. Supervised deep network representations contain structure related to the geometrical hierarchy of the characters, but the compositional structure of the data is not evident. Thus, HFD enables the identification of shortcomings in existing methods, a critical first step toward developing new machine learning algorithms to extract hierarchical and compositional structure in the context of naturalistic variability.

Figures

Tables

Table 1. Table 1: Hyperparameters for dense networks

Name	Type	Range/Options
Init. momentum	Float	.5
Learning rate reduction on plateau	float	.5
Epochs of patience for learning rate reduction	int	10
Epochs of patience for early stopping	int	10
Dense layer size	int	$\min ({dim}_i-1,{dim}_o\times 10)$ : $2\times {dim}_i$
${\log }_{10}$ learning rate	float	-6 : 1
${\log }_{10}$(1-momentum)	float	-2 : -.00436 (momentum=.01)
${\log }_{10}{L}_2$ weight decay	float	-6 : 1
Batch size	int	32 : 512
Input dropout rate	float	.1 : .99
Input dropout rescale	float	.1 : 10
Hidden dropout rate	float	.1 : .99
Hidden dropout rescale	float	.1 : 10

Equations

$$\text{Norm. acc.}=\frac{a-c}{1-c}$$

Full text

1 Introduction

Advances in machine learning, and representation learning in particular, have long been accompanied by the creation and detailed curation of benchmark datasets [1, 2, 3, 4, 5]. Often, such datasets are created with particular structure believed to be representative of the types of structures encountered in the world. For example, many image datasets have varying degrees of hierarchy and compositionality, as exemplified by parts-based decompositions, learning compositional programs, and multi-scale representations [6, 7, 8]. In contrast, synthetic image datasets often have known, (at least partial) factorial latent structure [9, 10, 11]. Having a detailed understanding of the structure of a dataset is critical to interpret the representations that are learned by any machine learning algorithm, whether linear (e.g., independent components analysis) or non-linear (e.g., deep networks). Learned representations can be used to understand the underlying structure of a dataset. Indeed, one of the desired uses of machine learning in scientific applications is to learn latent structure from complex datasets that provide insight into the data generation process [12, 13, 14]. Understanding how learned representations relate to the structure of the training data is an area of active research [15, 16, 17, 18].

Early benchmark image datasets such as MNIST (Fig 1A) and CIFAR10/100 [2, 19] enabled research into early convolutional architectures. Large image datasets like ImageNet (Fig 1B) and COCO [3, 20] have fueled the development of networks that can solve complex tasks like pixel-level segmentation and image captioning. Although these datasets occasionally have known semantic hierarchy (ImageNet classes are derived from the WordNet hierarchy [3, 21]) or labeled attributes which may be part of a compositional structure (attributes like “glasses” or “mustache” in the CelebA dataset [22]), the overall complexity of these images prevents a quantitative understanding of how the hierarchy or compositionality is reflected in the data or deep network representations of the data. On the other hand, synthetic benchmark datasets such as dsprites (Fig 1C), and many similar variations [9, 23, 10, 11], have known factorial latent structure [24]. However, these datasets typically do not have (known) hierarchy or compositionality. Thus, benchmark datasets, which have known hierarchical and compositional structure with naturalistic variability, are lacking.

A number of methods have been proposed to uncover “disentangeled” latent structure from images [25, 26, 27, 28, 29, 24, 30, 31, 32, 33, 34, 35, 36, 6] and understand hierarchical structures in data and how they are learned in deep networks [37, 15]. For datasets where the form of the generative model is not known, deep representation learning methods often look for factorial or disentagled representations [25, 26, 27, 38, 39]. While factorial representations are useful for certain tasks like sampling [24], they do not generally capture hierarchical or compositional structures. Deep networks can learn feature hierarchies, wherein features from higher levels of the hierarchy are formed by the composition of lower level features. The hierarchical multiscale RNN captures the latent hierarchical structure by encoding the temporal dependencies with different timescales on for character-level language modelling and handwriting sequence generation tasks [40]. Deep networks have been shown to learn acoustic, articulatory, and visual hierarchies when trained on speech acoustics, neural data recorded during spoken speech syllables, and natural images, respectively [41, 42, 43, 44]. Developing methods to probe representations for hierarchical or compositional structures is important to develop in parallel to benchmark machine learning datasets.

In this work, we present the new Hangul Fonts Dataset (HFD) (Fig 1D) designed for investigating hierarchy and compositionality in representation learning methods. The HFD contains a large number of data samples (391,020 across 35 fonts), annotated hierarchical and compositional structure, and naturalistic variation. Together these properties address a gap in benchmark datasets for deep learning, and representation learning research more broadly. To give examples of the potential use of the HFD, we explore whether typical deep learning methods can be used to uncover the underlying generative model of the HFD. We find that deep unsupervised networks to do recover the hierarchical or compositional latent structure, and supervised deep networks are able to partially recover the hierarchy latent structure. Thus, the Hangul Fonts Dataset will be useful for future investigations of representation learning methods.

2 The Hangul Fonts Dataset

The Korean writing system (Hangul) was created in the year 1444 to promote literacy [45]. Since the Hangul writing system was partially motivated by simplicity and regularity, the rules for combining phoneme characters (and their respective glyphs) into syllables (blocks) are regular and well specified. The Hangul alphabet consists of jamo (characters) broken into 19 initial consonants, 21 medial vowels, and 27+1 final consonants (including no final consonant) which generate $19\times 21\times 28=11,172$ possible character blocks each of which corresponds to a syllable. Not all combinations are used in the Korean language, however all possible blocks were generated for use in this dataset. The Hangul Fonts Dataset (HFD) uses this prescribed structure as annotations for each image. The dataset consists of images of all blocks drawn in 35 different open-source fonts from [46, 47, 48, 49] for a total of 391,020 annotated images.

Each Hangul block can be annotated most simply as having initial, medial, and final (IMF) independent generative variables. However, there are also variables corresponding to a geometrical hierarchy and variables corresponding to compositions of glyphs. The hierarchical variables are induced by the geometrical layout of the blocks. There are common atomic glyphs used across the initial, medial, and final characters (after a set of possible translations, rotations, and scalings) [50]. The compositional variables indicate which atomic glyphs are used for each block (in a “bag-of-atoms” representation). Together, these different descriptions of the data facilitate investigation into what aspects of this structure representation learning methods will learn when trained on the HFD.

2.1 The structure of a block: hierarchy and compositionality

There are geometrical rules for creating a block from atomic character glyphs. The initial consonant is located on the left or top of the block as either single or double characters (ㄱ and ㄲ in Fig 2A) There are 5 possible medial vowel character types: below, right-single, right-double, below-right-single, and below-right-double (ㅗ,ㅏ,ㅔ, ㅘ, and ㅞ in Fig 2A). The final consonant is at the bottom of the block as single, double, or absent characters (ㄱ and ㄳ in Fig 2A). The 30 geometrical possibilities together induce a 2-level hierarchy in the individual IMF labels and blocks that share geometric structure. The geometric variables describe the coarse layout (high level) of a block which is shared by many IMF combinations (low level) (Fig 2B and C, bottom and middle levels). Additionally, the 30 geometrical categories can be split into their intial, medial and final geometries (Fig 2B and C, bottom and middle levels). The geometric context of a character can change the style of the glyph within a block for a specific font, which is relevant for the representation analysis in Section 3. The medial character geometry can have a large impact on how an initial glyph is translated and scaled in the block. Similarly, the final geometry can impact the scaling of the initial and medial glyphs. These contextual dependencies can be searched for in learned representations of the data. For example, a supervised deep network trained to predict the initial character may use information about the medial geometry early in the network but then eventually discard that information when predicting the initial class.

Embedded within the IMF and geometrical structure is a set of glyph compositions. There are a base set of atomic glyphs from which all IMF glyphs are drawn (Fig 3A, Atom row). Then, one initial, medial, and final glyph is composed into a block (Fig 3A, IMF and Block rows). In this view, each block is built from a composition of potential rotations applied to a base set of atoms which are then structured by the geometrical rules and composed into a block. The underlines in the Atom and IMF rows of Fig 3A correspond to inclusion in the final colored blocks in the bottom row. In this paper, for comparisons with learned representations, the composition structure is encoded in 2 ways (although the full structure is available in the dataset). The first is a “bag-of-atoms” binary feature set where each block is given a binary feature vector which contains a 1 if the block contains at least one atom from the top row of Fig 2A (16 features). The second is a “bag-of-atoms” binary feature set where the rotations have not been taken into account (24 features). These two feature sets do not encode the complete compositional structure, but they are amenable to common representation comparison methods.

The size and shape of a glyph can change within a font depending on the context. Some of these changes are consistent across fonts and stem from the changing geometry of a block with different initial, medial, or final contexts (Fig 2). Different types of variations such as rotation, translation, and more naturalistic style variations arise in the dataset (Fig 3B). Glyphs can incorporate different rotations, scalings, and translation during composition into a block (Fig 3B, left 3 sets). There are variations across fonts due to the nature of the design or style of the characters. These include the style of characters which can vary from clean, computer font-like fonts to highly stylized fonts which are meant to resemble hand-written characters (Fig 3, rightmost set). Line thickness and the degree to which individual glyphs overlap or connect also vary. This variation is specific to a font and is based on the decision the font designer made, analogous to hand-written digits (i.e., MNIST).These types of variation are the main source of naturalistic variation in the dataset since they cannot be exactly described, but could potentially be modeled [7, 36].

2.2 Generating the dataset

We created a text file for the 11,172 blocks using the Unicode values from [51]. We then converted the text files to an image file using the convert utility [52] and font files. The image sizes were different across blocks within a font, so the images were resized to the max image size across blocks. The image sizes of blocks were also different across fonts, so the blocks were resized to the median size across fonts. Individual images for the initial, medial, and final characters are included, when availble. The exact scripts used to generate the dataset, a Dockerfile which can be used to recreate or extend the HFD, and curated open fonts are provided (see Appendix A). Further summary statistics for the dataset can be found in Appendix B.

3 Searching for hierarchy and compositionality in learned representations

Both shallow and deep learning models create representations (or transformations) of the input data. Methods like Principal Components Analysis (PCA) produce linear representations and Nonnegative Matrix Factorization (NMF) produces a shallow nonlinear representation inference in a linear generative model, and deep networks produce an increasingly nonlinear set of representations for each layer. Here, we compare the learned representation in unsupervised shallow methods, deep variational autoencoders, and deep feedforward classifiers. We consider whether the learned representations are organized around any of the categorical labels and hierarchy variables with an unsupervised $k$-nearest neighbor analysis. Then, we investigate whether the hierarchy or compositionality variables can be decoded with high accuracy from few features in the representations.

It is desirable that deep network representations can be used to recover the generative variables of a dataset. However, it is currently not known whether deep network representations are typically organized around generative variables. In order to understand this, we test whether the latent hierarchical structure of the Hangul blocks is a major component of the learned representations using unsupervised clustering of the representations. We compare the hierarchy geometry classes from Fig 2A to $k$-nearest neighbor clusterings of the test set representations (where $k$ is set to the number of class in consideration, for more details, see Appendix C). For the shallow and deep unsupervised methods (Fig 4A and B), we find that the medial label and geometry, final label, and all_geometry variables are all marginally present ($0<\text{normalized accuracy}\leq 0.25$, see Section 4 for definition) in the representations. The other variables are not recovered by the unsupervised methods ($\text{normalized accuracy}\approx 0$). This shows that while VAE variants may be able to disentangle factorial structure in data, they are not well suited to extracting geometric hierarchy from the HFD with high fidelity.

In contrast (and unsurprisingly), supervised deep networks cleanly extract and recover the label they are trained on (Fig 4C-E, first 3 colums) with increasing accuracy across layers ($\text{Norm. acc.}>0.25$). When trained on the initial label, the initial, medial, and all_geometry variables can all be marginally recovered, highlighting the contextual dependence of the initial glyph on the medial geometry. The initial geometry is not present in the first 2 layers, but becomes marginally present in the final layers. When trained on the medial labels, the medial geometry is present with high accuracy and the all geometries labels are marginally present. When trained on the final labels, the final geometry becomes present by the last 2 layers. There is a small amount of interaction with the medial geometry, but it is not as large as the initial-medial interaction. These results indicate that supervised deep networks do learn representations that mirror aspects of the hierarchical structure of the dataset that are most relevant for the task, and generally do not extract non-relevant hierarchy information.

Understanding whether deep network representations tend to be more distributed or local is an open area of research [17, 53, 54]. We investigated whether deep networks learn a local representation by training sparse logistic regression models to predict the latent hierarchy and compositionality variables from the representations (Fig 5). If the representation of a hierarchy or compositionality variable is present and simple (linear), we would expect the normalized accuracy to be high (near 1 on the y-axis of the plots in Fig 5). If a representation of a variable is “local”, we would expect the varible to be decoded using approximately the same number of features as it has dimensions (near $10^{1}$ on the x-axis of Fig 5) and “distributed” representation to have a much higher ratio. To test this, we compare these two measures across models and target variables and also across layers for the supervised deep networks.

We find that unsupervised ($\beta$-)VAEs (Fig 5A) learn consistently distributed representations of the latent variables (typically 30-60x more features than the variable dimension are selected). In terms of the prediction accuracy, the cross validated $\beta$-VAEs tend to have higher accuracy across variables than the VAE and the $\beta$-VAE selected for traversals, although there is a fair amount of heterogeneity. For supervised deep networks (Fig 5B-D), the supervision variable (initial, medial, final, respectively), has high accuracy across layers, and moves from a more distributed to a more local representation at deep layers. For the initial and medial labels, the medial geometry can also be read out with high accuracy and an increase in localization across layers. The initial geometry is not read out with high accuracy in the initial and medial label networks, and the final geometry variable can only be predicted well for the final label network. The all_geometry variable can be predicted at marginal accuracy for all networks. The compositional Bag-of-Atoms (BoA) features cannot be predicted well (often at or below chance) for any network and the BoA mod rotations can only be read out with marginal accuracy for the initial label network. These results suggest that standard, fully-connected deep networks do not typically learn local representations for variables except for those they are trained on (and correlated variables).

4 Methods

4.1 Representation learning methods

Principal Component Analysis (PCA), Independent Component Analysis (ICA), and Non-negative Matrix Factorization (NMF) from Scikit-Learn [55] were used to learn representations from the data. These methods were all trained with 100 components which is at least 3-times larger than any of the latent generative variables under consideration. The models were trained on the training and validation sets and the representation analysis was on the test set.

Variational autoencoders (VAEs) learn a latent probabalistic model of the data they are trained on. The $\beta$-VAE is a variant of a VAE which aims to learn disentangled latent factors [26, 27] by trading off the reconstruction and KL-divergence terms with a factor different than 1. We implement the $\beta$-VAE from Burgess et al. [27], which encourages the latent codes to have a specific capacity. We experiment with both $\beta>1$ from [27] as well as $\beta<1$ from [56, 57]. $\beta$-VAE networks with convolutional and dense layers were trained on the dataset. 100 sets of hyperparameters were used for training the $\beta$-VAEs. The hyperparameters and their ranges are listed in Appendix D. In order to cross-validate the networks, we checked if the same blocks across fonts are nearest neighbors in the latent space. For each block in each font, the nearest neighbor is found. If the neighbor has the same label as the block, we assign an accuracy of 1, otherwise 0. This is averaged across all blocks and pairs of fonts in the validation set. The model with the best cross-validation accuracy for each label was chosen and the downstream analysis was done on the test set latent encodings. We also cherry-picked networks which had interpretable latent traversals (Fig 6).

Fully-connected networks with 3 hidden layers were trained on one of the initial, medial, or final glyph variables. For each task, 100 sets of hyperparameters were used for training. The hyperparameters and their ranges are listen in Appendix D. The model with the best validation accuracy was chosen and the downstream analysis was done on the test set representations (test accuracies reported in Appendix B). Code for training the networks and reproducing the figures will be posted publicly. Deep networks representation analysis was partially completed on the NERSC supercomputer. All deep learning models were trained using PyTorch [58] on Nvidia GTX 1080s or Titan Xs.

To compare accuracies (and chance accuracies) across models with differing numbers of classes (between 2 and 30), we 0-1 normalize the accuracies across models to make comparisons more clear. Specifically, for a model with given $\text{accuracy}=a$ and $\text{chance}=c$, we report

[TABLE]

which is 0 when $a=c$ 1 if $a=1$, independent of the number or distribution of classes.

4.2 Generative structure recovery from representation of the data

The 35 fonts were used in a 7-fold cross validation loop for the machine learning methods. The fonts were randomly permuted and then 5 fonts were used for each of the non-overlapping validation and test sets. The analysis of representations was done on the test set representations. For the supervised deep networks, the following methods were applied to the activations of every layer both before and after the ReLU nonlinearities. For the unsupervised VAEs, they were applied to samples from the latent layer. The logistic regression analysis was not applied to the linear representations.

Clustering a representation produces a reduced representation for every datapoint in an unsupervised way. If one chooses the number of clusters to be equal to the dimensionality or number of classes the generative variables has, then they can be directly compared (up to a permutation). We cluster the representations with KMeans and then find the optimal alignment of the real and clustered labels (see Appendix C for more details). We then report the normalized accuracy of this labeling across training variables, layers, and hierarchy variables.

Sparse logistic regression attempts to localize the information about a predicted label into a potentially small set of features. To do this, we used logistic regression models fit using the Union of Intersection (UoI) method [59, 60]. The UoI method has been shown to be able to fit highly sparse models without a loss in predictive performance [61]. We report the normalized accuracy and mean number of features selected divided by the number of features or classes across training variables, layers, and hierarchy variables. For this analysis, 2 new training and testing sub-splits were created from the representations on the original test set that was held out during deep network training.

5 Discussion

The Hangul Fonts Dataset (HFD) presented here has hierarchical and compositional latent structure that allows each image (block) to have ground-truth annotations, making the HFD well suited for deep representation research. Using a set of unsupervised and supervised methods, we are able to extract a subset of the variables from the representations of deep networks. Several VAE variants have relatively poor variable recovery from their latent layers, while supervised deep networks have clear representation of the variables they are trained on and interacting variables. Understanding how to better recover such structure from deep network representations will broaden the application of deep learning in science.

In many scientific domains like cosmology, neuroscience, and climate science, deep learning is being used to make high accuracy predictions given growing dataset sizes [62, 42, 63, 64]. However, deep learning is not commonly used to directly test hypotheses about dataset structure. This is partially because the nonlinear, compositional structure of deep networks, which is conducive to high accuracy prediction from complex data, is not ideal for interrogating hypotheses about data. In particular, it is not generally known how the structure of a dataset influences the learned data representations or whether the structure of the dataset can be “read-out” of the learned representations. Understanding which dataset structures can be extracted from learned deep representations is important for the expanded use of deep learning in scientific applications.

The HFD is based on a set of fonts which provide some naturalistic variation. However, the amount of variation is likely much smaller than what would be found in a handwritten dataset of Hangul characters. One benefit to using fonts is that the dataset can be easily extended as new fonts are created. To this end, we release the entire dataset creation pipeline to aid in future expansion of the HFD or the creation of similar font-based datasets. Another potential limitation and area of future work is determining how to encode variables like hierarchy and compositionality. In this dataset, there is a natural class-based encoding for the shallow geometry hierarchy. The Bag-of-Atoms composition encoding ignores structure that is potentially relevant for recovering compositionality (much like Bag-of-Words features discard potentially useful structure in natural language processing).

In this work, relatively small fully-connected and convolutional networks were considered. However, these techniques can be applied to larger feedforward networks, recurrent networks, or networks with residual layers to understand the impact on learned representations. Understanding how proposed methods for learning factorial or disentangled representations [24, 25, 26, 32] impact the structure of learned representations is important for using deep network representations for hypothesis testing in scientific domains. Compared to disentangling [38], relatively little work addresses how to define and evaluate hierarchy and compositionality in learned representations. Furthermore, unsupervised or semi-supervised cross-validation metrics that can be used for model selection across a range of structure recovery tasks (e.g., disentangling, hierarchy recovery, compositionality recovery) are lacking.

Acknowledgments and Disclosure of Funding

JAL, AH, and KEB were supported by the Deep Learning for Science Laboratory Directed Research and Development. We are grateful for the feedback on the project from the Neural Systems and Data Science Lab.

Appendix A Generating the Hangul Fonts Dataset

To increase the reproducibility and potential for future development of the Hangul Fonts Dataset (HFD), we are releasing the code used to generate and normalize the HFD. The code is currently undergoing our institutional review process for licensing, but will be released under a Lawrence Berkeley National Labs BSD variant license by the time of publication along with a DOI from a corresponding Zenodo archive. The code and instructions for reproducing the HFD, including a docker image and Dockerfile which specifies the environment, can be found at https://github.com/BouchardLab/HangulFontsDatasetGenerator. The 35 open font files, their licences, and a number of additional fonts have been curated here.

This code (with default settings) generates the following

• •

pngs/*: folders per font and fontsize (default 24) containing png files of each block

• •

pdfs/*: folder per font containing pdfs which show all possible blocks

• •

h5s/*: folder per font containing HDF5 files which contain the following (key: variable) pairs

• –

images: block images at max size across blocks within the font (not used directly)

• –

images_median_shape: block images at median size across fonts (used for training/analysis)

• –

labels: concatenated IMF class labels

• –

{initial, medial, final}_geometry: IMF geometry hierarchy class labels

• –

all_geometry: class labels based on the product of the IMF geometry class labels

• –

atom_bof: IMF atoms features

• –

atom_mod_rotations_bof: IMF atoms mod rotations features

Appendix B The Hangul fonts dataset: summary statistics and visualization

B.1 Font families

For certain fonts, bold and light versions of the same font are included, a “natural” but fairly regular source of variation across fonts (Fig 7).

We also performed correlations across fonts to analyze how fonts differ from one another. Fig 8 shows the dendrogram that results from hierarchical clustering of the correlation matrix using Ward’s method. GothicA1-Bold, GothicA1-SemiBold, GothicA1-Regular, GothicA1-Light, GothicA1-Thin, GothicA1-Black, GothicA1-Medium, and GothicA1-ExtraBold all fall under the same GothicA1 category.

B.2 Summary statistics of dataset

Various statistics were calculated on fontsize 24 images within a single and across all fonts. The mean, median, and standard deviation of the images were taken in Fig 9. This was done for all blocks in all fonts, all blocks in a single font, and a single block in all fonts. All three statistics for all fonts all blocks and one font all blocks preserve the block structure of the images, whereas ’가’ is clearly shown for all fonts single block across the statistics. Fig 10A shows a histogram of the pixels of all the images within a font for all 35 fonts. The histograms are very similar across the four fonts highlighted in the legend. Fig 10B shows a histogram of pixels within a font across all characters for all 35 fonts. The Frobenius norm is taken for all characters to study how characters differ within a font. NanumMyeongjo and NanumBrush are similar fonts as they have overlapping character norms. GothicA1-Regular, which resembles computer-type fonts, has the thinnest distribution as its characters do not differ greatly.

B.3 Dimensionality Reduction: PCA, ICA, NMF

Linear models were trained on individual fonts and the learned dictionaries are shown in Fig 11.

B.4 Classification accuracy

The supervised deep networks were trained on 3 tasks: initial, medial, and final (IMF) classification. Here we report the test-set accuracy for logistic regression as well as the best deep networks (selected by the validation accuracy). Logistic regression had accuracies of $57.6\pm 2.7\%$, $70.8\pm 2.6\%$, and $73.0\pm 3.1\%$ respectively for the IMF tasks. Deep networks had accuracies of $98.5\pm 2.0\%$, $98.0\pm 2.6\%$, and $97.4\pm 3.3\%$ respectively for the IMF tasks. Chance accuracy is $5.3\%$, $2.6\%$, and $3.1\%$ respectively for the IMF tasks.

B.5 UMAP visualization

Fig 12A-C shows the result of applying UMAP to a single font’s images, GothicA1-Regular, with initial, medial, and final labels. Individual glyphs are plotted with a red kernel density estimate in the background. Fig 12B shows the best clustering with regards to the geometry of the glyphs. The more verticially-oriented glyphs (ㅏ,ㅑ,ㅕ,ㅓ) cluster together in the right side while the more horizontally-oriented glyphs (ㅛ,ㅡ,ㅜ,ㅠ) cluster in the left. In Fig 12C several of the duplet glyphs embed in the same location, suggesting similarity in the duplet structure.

Fig 13A-C shows the result of applying UMAP to the images with initial, medial, and final geometry labels. Actual points are plotted rather than glyphs. Fig 13B shows the best separation among the 5 geometric types as they are each distinct, and hence affect the overall structure of the block character. For example, right-single and right-double medial geometric types are always placed on the left region of the blocks. In contrast, initial and final geometry types which include none, single, or double do not drastically influence the greater structure of the block. Single and double geometric types have very similar embeddings in both the initial and final geometry plots.

Appendix C Methods continued

C.1 KMeans clustering accuracy

For a dataset of sample size $n$ and a representation $h\in\mathbb{R}^{n\times d}$ we want to compare a clustering of $h$ and a categorical generative variable $y\in\{1,\ldots,M\}$. First, $h$ is clustered into $M$ clusters with k-means (with k=$M$) or from cutting the dendrogram to give $M$ clusters. Given this clustering, each sample is assigned a class label $\hat{p}$. We then have to find the best alignment between the $M$ cluster labels and the $M$ generative variable labels. To do this we form a similarity matrix $S\in\mathbb{R}^{M\times M}$. To calculate $S_{ij}$, we first form the set $A$ which contains the samples labeled $\hat{p}_{i}$ and $B$ which contains the samples labeled $y_{j}$. Then the similarity is the cardinality of the intersection of the sets divided by the cardinality of the union of the sets: $S_{ij}=\tfrac{|A\cap B|}{|A\cup B|}$, that is, given all samples labeled with either label, what fraction of them are labelled as both. We then use the Hungarian method [65] to optimally pair the generative labels $y$ with a permutation of the cluster labels $\hat{y}$ using this similarity matrix. If the cluster labelling is an exact permutation, the clustering accuracy will be 100%, and chance for a random relabelling.

Appendix D Hyperparameters

All supervised networks were trained with 3 hidden layers of the same dimensionality and ReLU nonlinearities. Table 2 lists the hyperparameters that were randomly sampled and their ranges. $\dim_{i}$ and $\dim_{0}$ indicate the input and output dimensionality of the data and task.

Unsuperised $\beta-$VAE encoders were trained with 7 $2-$d convolutional layers of the same channel depth and kernel size, ReLU nonlinearities, and one hidden layer of the same dimensionality. The decoders were trained with one hidden layer of the same dimensionality, transposed $2-$d convolutional layers of the same channel depth and kernel size, and ReLU nonlinearities. The $\gamma<1$ networks were trained with the training regime from [27]. The $\gamma>1$ networks were trained with the same training regime as $\gamma<1$ for 125 epochs (using $\gamma<1$), after which the $\gamma$ term was increased exponentially, similar to the training regime proposed by [56].

Appendix E Datasheet for the Hangul Fonts Dataset

We follow the Datasheets for Datasets [66] recommendations for documenting datasets.

E.1 Motivation

For what purpose was the dataset created?

Who created the dataset and on behalf of which entity?

Who funded the creation of the dataset?

The Hangul Fonts Dataset was created as a benchmark machine learning dataset for investigating whether and how hierarchy and compositionality are found representation learning methods with a focus on deep networks. It was created by Jesse Livezey, Ahyeon Hwang, and Kristofer Bouchard at Lawrence Berkeley National Laboratory (LBNL) and was funded through the Laboratory Directed Research and Development Program at LBNL.

E.2 Composition

What do the instances that comprise the dataset represent? Does the dataset contain all possible instances or is it a sample of instances from a larger set? What data does each instance consist of?

How many instances are there in total?

Is there a label or target associated with each instance?

Are there recommended data splits?

Are there any errors, sources of noise, or redundancies in the dataset?

Is the dataset self-contained, or does it link to or otherwise rely on external resources?

Does the dataset contain data that might be considered confidential? Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? Does the dataset relate to people?

The Hangul Fonts Datasets consists of all possible blocks for each font along with a set of labels and features for each block which are common across all fonts. We provide scripts to render each block and font into an image and store them together in an HDF5 file for each font and fontsize specified. In addition, we have curated a set of Hangul fonts which are licensed under the TODO.

Each font has 11,172 possible blocks and we have curated 35 fonts for a total of 391,020 possible images. The spoken consonants and vowel correspond to a set of “initial”, “medial”, and “final” labels for each block with 19, 21, and 28 classes respectively. In addition, each block can also be assigned a hierarchy label based on the block geometry. Finally, the individual glyphs (atoms) are composed across the initial, medial, and final locations along with potential scalings, translations, or rotations. We use these relationship to define a bag-of-atoms set of features.

Rendering the fonts to images transforms them from a vector-based representation to a pixel-based representation which will introduce some smoothing or loss of information depending on the font size used. We find that there is not much gain in classification accuracy beyond font size 24 which is the font size we recommend and generate by default.

The dataset relies on a set of open font files, which we have curated. Otherwise, the dataset is self contained and we provide scripts for users to generate their own copy of the dataset. Using the scripts, users are also able to generate datasets with varying fontsizes or new open or closed fonts.

The dataset does not contain any data that might be considered confidential or harmful. Although by their nature fonts and the Hangul writing system relate to people in their creation and use, the instances in this dataset are not related to data from any people.

E.3 Collection Process

How was the data associated with each instance acquired? What mechanisms or procedures were used to collect the data?

If the dataset is a sample from a larger set, what was the sampling strategy?

Who was involved in the data collection process and how were they compensated?

Over what timeframe was the data collected?

Were any ethical review processes conducted?

The open font files which are used in conjunction with the library to generate the images were curated from lists of Hangul fonts at TODO and TODO. All availble, open, de-duplicated fonts were used in the final dataset. Jesse Livezey and Ahyeon Hwang curated the font files and wrote the library and were paid as part of a postdoc and a research assistant position, respectively, at LBNL. The fonts were curated and the library was written between October 2017 and February 2019. No ethical review process was conducted.

E.4 Preprocessing/cleaning/labeling

Was any preprocessing/cleaning/labeling of the data done?

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data?

Is the software used to preprocess/clean/label the instances available?

The vector font files are converted into a set of images. There are choices in terms of font size and final image standardization made and documented in the library. The initial, medial, and final labels, the geometry labels, and the atom labels are based on the structure of the Hangul writing system. The identification of different atoms across rotations was done by hand. In lieu of releasing the binary image files, we provide the library used to create the images along with a Docker image which can be used to exactly replicate the images.

E.5 Uses

Has the dataset been used for any tasks already?

Is there a repository that links to any or all papers or systems that use the dataset?

What (other) tasks could the dataset be used for?

Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?

Are there tasks for which the dataset should not be used?

The dataset has not been used outside of its original preprint/publication. The dataset could be used for other types of generative and supervised machine learning analysis of Hangul fonts. There were preprocessing choices made which may impact future use, but the library used to generate the images is being released. We cannot think of potential tasks for which the dataset should not be used.

E.6 Distribution

Will the dataset be distributed to third parties outside of the entity on behalf of which the dataset was created?

How will the dataset will be distributed (e.g., tarball on website,API, GitHub)?

When will the dataset be distributed?

Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use(ToU)?

Have any third parties imposed IP-based or other restrictions on the data associated with the instances?

Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?

The library used to generate the images from the font files will be distributed on Github under an open source license. We have also curated a set of open font files. The library will be distributed under a Lawrence Berkeley National Labs BSD variant license license after institutional review is complete. There are no restrictions or controls on the dataset or instances.

E.7 Maintenance

Who is supporting/hosting/maintaining the dataset?

How can the owner/curator/manager of the dataset be contacted?

Is there an erratum?

Will the dataset be updated?

If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances?

Will older versions of the dataset continue to be supported/hosted/maintained?

If others want to extend/augment/build on/contribute to thedataset, is there a mechanism for them to do so?

The dataset will be hosted on Github (with an archive or the publication version on Zenodo) and maintained by authors who can be contacted through Github and/or their institutional email addresses. The dataset will be updated to correct any errors which will be noted in a erratum. Future versions of the dataset will be extended to work with additional open fonts as they become available and contributions/corrections from others. Contributions/corrections can be made through Github. Old versions will be tagged and maintained as applicable. The dataset does not relate to people.