Finite State Machines for Semantic Scene Parsing and Segmentation

TL;DR

This paper presents a novel stochastic inference framework using finite state machines for semantic scene annotation and segmentation, enabling complex operations like reordering and label dependency modeling.

Contribution

It introduces a generative FSM-based approach that encodes annotation lattices and integrates multiple operations for improved scene parsing.

Findings

Effective scene annotation and segmentation

New FSM operations for reordering and label dependencies

Unified framework for multiple inference tasks

Abstract

We introduce in this work a novel stochastic inference process, for scene annotation and object class segmentation, based on finite state machines (FSMs). The design principle of our framework is generative and based on building, for a given scene, finite state machines that encode annotation lattices, and inference consists in finding and scoring the best configurations in these lattices. Different novel operations are defined using our FSM framework including reordering, segmentation, visual transduction, and label dependency modeling. All these operations are combined together in order to achieve annotation as well as object class segmentation.

Figures

Tables

Table 1. TABLE I: This table shows the performance of our model for different settings on the Sunset04 dataset.

Lines	Reordering + Grouping Model	Label Dependency Model	Visual Model	FAR (%)	FRR (%)	EER (%)
1:	Yes/No	flat	flat	25.00	75.00	50.00
2:	No	flat	learned	04.98	28.81	16.90
3:	No	learned	flat	13.47	41.26	27.36
4:	No	learned	learned	03.11	11.34	07.22
5:	Yes + flat	flat	learned	04.98	28.78	16.88
6:	Yes + flat	learned	flat	11.43	36.52	23.98
7:	Yes + flat	learned	learned	02.70	08.64	05.67
8:	Yes + learned	flat	learned	04.98	28.96	16.97
9:	Yes + learned	learned	flat	11.13	33.99	22.56
10:	Yes + learned	learned	learned	02.49	07.80	05.14

Equations

$$P(\mathbb{X},\mathbb{Y},\mathbb{C},\mathbb{k},\pi )=$$

$$P(\pi ).$$

$$P(\mathbb{C},\mathbb{k}|\pi ).$$

$$P(\mathbb{Y}|\mathbb{C},\mathbb{k},\pi ).$$

$$P(\mathbb{X}|\mathbb{Y},\mathbb{C},\mathbb{k},\pi ).$$

$$\begin{array}[]{lll}\displaystyle P(\mathbb{Y}_{{s}})&=&\displaystyle\frac{\sum_{p\in s}{\bf f}_{u}(\mathbb{Y}_{s},p)}{\sum_{\ell\in{\cal C}}\sum_{p\in s}{\bf f}_{u}(\ell,p)}\\ &&\\ \displaystyle P(\mathbb{Y}_{s}|\mathbb{Y}_{s^{\prime}})&=&\displaystyle\frac{\sum_{p\in s}\sum_{p^{\prime}\in s^{\prime}}{\bf f}_{b}({\bf Y}_{s},\mathbb{Y}_{s^{\prime}},p,p^{\prime})}{\sum_{\ell\in{\cal C}}\sum_{p\in s}\sum_{p^{\prime}\in s^{\prime}}{\bf f}_{b}(\ell,\mathbb{Y}_{s^{\prime}},p,p^{\prime})}.\end{array}$$

$$\begin{array}[]{lll}\displaystyle P({\bf X}|{\bf Y},{\bf C},{\bf k},{\bf\pi})&=&\displaystyle\displaystyle\prod_{i=1}^{\mathbb{k}}\prod_{s\in c_{i}}P({\bf X}_{s}|\mathbb{Y}_{c_{i}})\\ &&\\ \displaystyle\textrm{with}\ \ \ \ P({\bf X}_{s}|{\bf Y}_{c_{i}})&\propto&\displaystyle\frac{1}{1+\exp(-f_{\mathbb{Y}_{c_{i}}}({\bf X}_{s}))},\end{array}$$

Full text

Keywords: Finite state machines, scene parsing and segmentation

I Introduction

The general problem of image annotation is usually converted into classification. Many existing state of the art methods (see for instance [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]) treat each keyword (also referred to as label or category) as a class, and then build the corresponding category-specific classifier in order to identify images or image regions belonging to that class, using a variety of machine learning techniques such as Markov models [18, 19, 20, 21], latent Dirichlet allocation [22, 23, 24, 25], probabilistic latent semantic analysis [26, 27, 28], support vector machines [29, 30, 31, 32, 33, 34, 19, 35, 36], deep learning [37, 38, 39, 40, 41, 42, 43, 44, 45, 46], etc. Annotation methods may also be categorized into: region-based object class segmentation (OCS) requiring a preliminary step of image segmentation (e.g., [47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58], etc.), and holistic (e.g., [59, 11, 60, 61, 17, 62, 63, 29, 64], etc.) operating directly on the entire image space. In both cases, training is achieved in order to learn how to attach labels to the corresponding visual features.

In this paper, we introduce an original OCS method based on FSMs. Our approach is Bayesian; it finds superpixel labels that maximize a posterior probability, but its key-novelty resides in the representational power of FSMs in order to build a comprehensive model for scene segmentation and labeling. Indeed, we translate our OCS into searching, via FSMs, the optimum of a discrete energy function mixing i) a unary term that models conditional probability of visual features given their (possible) labels, ii) an interaction potential which provides joint statistics, of co-occurrence, between those labels, and more importantly iii) a novel reordering and grouping term. The latter allows us, via FSMs, to examine (generate, label, score and rank) many partitions of segments of a scene, and to return only the likely ones.

At least two reasons drove us to apply FSMs for OCS:

i) Firstly, as FSMs can model huge (even infinite) languages111In OCS, the alphabet of the language corresponds to all the superpixels of a given scene., with finite memory and time resources, our method does not require explicit generation of all possible segmentations and labelings. Instead, it first models them implicitly by combining (composing) different FSMs and then efficiently finds the shortest path (i.e., the most likely segmentation and labeling) in a global FSM.

ii) Secondly, superpixel reordering allows us to examine the possible segmentations at different orders and, using the chain rule, to maximize the interaction potentials resulting into better labeling. For that purpose, scenes are first described with graphs where nodes correspond to superpixels and edges connect neighboring superpixels. Then, reordering is achieved by generating random (Hamiltonian) walks on these graphs using FSMs. Note that graphs with very low connectivity ($\leq 4$ immediate neighbors for each superpixel) dramatically reduce the complexity of this reordering and also grouping while, at the same time, make it possible to explore larger sets of possible solutions resulting into an effective and also efficient scene labeling machinery as discussed later in this paper.

II Scene Labeling Model

Given $n$ lattice points ${\cal V}=\{1,\dots,n\}$, we define in this section $\mathbb{X}=\{X_{1},\dots,X_{n}\}$ as a set of observed random variables, corresponding to a subdivision of an image $\mathbb{X}$ into smaller units, referred to as superpixels. Let $\mathbb{C}=\{c_{i}:c_{i}\subseteq\mathcal{V}\}_{i=1}^{\mathbb{k}}$ be a random partition of $\mathcal{V}$; an element $\mathbb{X}_{c_{i}}\subseteq\mathbb{X}$ is defined as a collection of conditionally independent random variables (here $\mathbb{X}_{c_{i}}=\{X_{k}\in\mathbb{X}:k\in c_{i}\}$) and $\mathbb{Y}=\{\mathbb{Y}_{c_{1}},\dots,\mathbb{Y}_{c_{\mathbb{k}}}\}$ the underlying (unknown) labels taken from a label set ${\cal C}=\{\ell_{i}\}_{i}$.

For a given observed superpixel set $\mathbb{X}$, our scene labeling defines a joint probability distribution over multiple superpixel reorderings, groupings (segmentations), labelings and finds the best tuple $(\mathbb{Y},\mathbb{C},\mathbb{k},\mathbb{\pi})$ as the max of the following probability distribution

[TABLE]

Here $\mathbb{\pi}\in{\cal G}(\mathcal{V})$ denotes a permutation (reordering) that maps each element $i\in\mathcal{V}$ to $\mathbb{\pi}_{i}\in\mathcal{V}$ and ${\cal G}(\mathcal{V})$ denotes the symmetric group on $\mathcal{V}$ including all the bijections (permutations) from $\mathcal{V}$ to it-self. The model above illustrates a generative scene labeling process which first (i) reorders (in multiple ways) the lattice $\mathcal{V}$ resulting into $\mathbb{\pi}_{1}\dots\mathbb{\pi}_{n}$, (ii) partitions/groups (in multiple ways) $\mathbb{\pi}_{1}\dots\mathbb{\pi}_{n}$ into $\mathbb{k}$ subsets $c_{1}\dots c_{\mathbb{k}}$, then (iii) emits (in multiple ways) label hypotheses $\mathbb{Y}_{c_{1}}\dots\mathbb{Y}_{c_{\mathbb{k}}}$ for the segments $\mathbb{X}_{c_{1}}\dots\mathbb{X}_{c_{\mathbb{k}}}$ and finally (iv) estimates their visual likelihood so only relevant labels will be strengthened. Example in Fig. (1, top) illustrates one realization of this stochastic process.

Note that a naive and brute force generation of all the possible reorderings, groupings, labelings would be out of hand; as detailed in the subsequent sections, our approach does not explicitly generate all these configurations, but instead implicitly specifies them using finite state machines in order to build a scored lattice of possible segmentations and labelings.

II-A Reordering & Grouping Models for Multiple Image Segmentation

Reordering corresponds to permutations that transform an image lattice $\mathcal{V}$ into many words in $\{\mathbb{\pi}\}_{\mathbb{\pi}\in{\cal G}(\mathcal{V})}$, while grouping breaks every word $\mathbb{\pi}=\mathbb{\pi}_{1}\dots\mathbb{\pi}_{n}$ into $\bf k$ subwords. Applying all the permutations $\{\mathbb{\pi}\}_{\mathbb{\pi}\in\mathcal{G}(\mathcal{V})}$, followed by all the possible groupings makes it possible to generate all the possible partitions of $\mathcal{V}$. Subsets in each partition (again denoted $\mathbb{C}=\{c_{i}\}_{i=1}^{\bf k}$) are defined as $c_{i}=\{\pi_{k_{i}},\dots,\pi_{\ell_{i}}\}$, with $1\leq k_{i}\leq\ell_{i}\leq n$. Reordering and grouping models, also shown in Eqs. 2-3, are necessary not only to delimit segment boundaries with a high precision but also to evaluate label dependencies between segments at multiple orders (see section II-B and Fig. 1, bottom).

In this work, we restrict image partitions to include only connected segments with homogeneous superpixels. For that purpose our superpixel grouping follows random walks (RWs): it randomly visits superpixels in $G=(V,E)$ (graph associated to the lattice $\cal V$), and groups (with a high probability) only connected and visually similar ones. Each RW corresponds to a path in $G$ which is not necessarily Hamiltonian. If one restricts RWs to include only permutations, then the resulting paths will be Hamiltonian222Note that planar 4-connected graphs (including 4-connected regular grids) are necessarily Hamiltonian. and correspond to partitions of $\mathcal{V}$ that necessarily include connected subsets.

Considering the lattice $\mathcal{V}$, our reordering model $P(\mathbb{\pi})$ equally weights permutations in $\mathcal{G}(\mathcal{V})$, i.e., $\forall\mathbb{\pi}\in\mathcal{G}(\mathcal{V})$, $P(\mathbb{\pi})=1/{{|{\cal G}(\mathcal{V})|}}$ while our grouping model $P(\mathbb{C},\mathbb{k}|\mathbb{\pi})$ assumes all segments in a given partition conditionally independent given $\mathbb{\pi}$, so one may write $P(\mathbb{C},\mathbb{k}|\mathbb{\pi})=P(\mathbb{k})\prod_{i=1}^{\mathbb{k}}P(c_{i}|\mathbb{\pi})$. All the partition sizes have the same mass, i.e., $P(\mathbb{k})=1/n$, $\forall\ \mathbb{k}\in\{1,2,\dots,n\}$ and $P(c_{i}|\mathbb{\pi})=P(\mathbb{\pi}_{k_{i}})\prod_{j=1}^{\ell_{i}-k_{i}}P(\mathbb{\pi}_{k_{i}+j}|\mathbb{\pi}_{k_{i}+j-1})$. Here $P(\mathbb{\pi}_{k_{i}})$ is taken as uniform, i.e., $1/n$ and $P(\mathbb{\pi}_{k_{i}+j}|\mathbb{\pi}_{k_{i}+j-1})$ is taken as the random walk transition probability from node (superpixel) $\mathbb{\pi}_{k_{i}+j-1}$ to node $\mathbb{\pi}_{k_{i}+j}$, which is positive only if the underlying superpixels share a common boundary and it is set to P(\mathbb{\pi}_{k_{i}+j}|\mathbb{\pi}_{k_{i}+j-1})\ \propto\ \mathds{1}_{\{(\mathbb{\pi}_{k_{i}+j},\mathbb{\pi}_{k_{i}+j-1})\in E\}}\cdot{\bf\kappa}\big{(}\psi(\mathbb{\pi}_{k_{i}+j}),\psi(\mathbb{\pi}_{k_{i}+j-1})\big{)}; here ${\bf\kappa}$ is the histogram intersection kernel and $\psi(\mathbb{\pi}_{k_{i}+j})$ denotes a visual feature extracted at superpixel $\mathbb{\pi}_{k_{i}+j}$. With this model, if the transition between neighboring superpixels is achieved with a high conditional probability, then these superpixels are considered as visually similar and likely to come from the same physical object.

II-B Visual and Label Dependency Models

Once superpixels reordered and grouped in multiple ways, we use a unary visual and label interaction models, described below, in order to score the resulting partitions. As shown in the remainder of this paper, only highly scored partitions are likely to correspond to correct object segmentations.

Label Dependency Model. This model captures scene structure and a priori knowledge about segment/label relationships (either co-occurrence or geometric relationships) in order to consolidate labels which are consistent with already observed scenes. Considering a first order Markov process and using the chain rule, our bi-gram label dependency model is $P(\mathbb{Y}|\mathbb{C},\mathbb{k},\pi)=P(\mathbb{Y}_{c_{1}})\prod_{i=2}^{\mathbb{k}}P(\mathbb{Y}_{c_{i}}|\mathbb{Y}_{c_{i-1}})$.

Let ${\cal I}=\{{\bf I}_{1},\dots,{\bf I}_{N}\}$ be a training set, of fixed size images, labeled at the pixel level (with labels in $\cal C$). Let ${\bf f}_{u}(\ell,p)=\sum_{i=1}^{N}\mathds{1}_{\{{\bf I}_{i}(p)=\ell\}}$ be the frequency of co-occurrence of pixel $p$ and label $\ell$ in $\cal I$. Similarly, we define ${\bf f}_{b}(\ell,\ell^{\prime},p,p^{\prime})$ as $\sum_{i=1}^{N}\mathds{1}_{\{{\bf I}_{i}(p)=\ell\}}\mathds{1}_{\{{\bf I}_{i}(p^{\prime})=\ell^{\prime}\}}$. Given superpixels $s$, $s^{\prime}$ with labels resp. as ${\mathbb{Y}}_{s}$, ${\bf Y}_{s^{\prime}}$, we define

[TABLE]

Visual Model. This defines the likelihood of ${\bf X}=\{{\bf X}_{c_{1}},\dots,{\bf X}_{c_{\bf k}}\}$ given the labels ${\bf Y}=\{{\bf Y}_{c_{1}},\dots,{\bf Y}_{c_{\bf k}}\}$. Assuming conditionally independent superpixels and segments given their labels, and assuming that each ${\bf X}_{c_{i}}$ depends only on ${\bf Y}_{c_{i}}$, we define our visual model as

[TABLE]

here $f_{\mathbb{Y}_{c_{i}}}(.)$ is an SVM classifier (based on histogram intersection kernel) trained, using LIBSVM, to discriminate superpixels belonging to a category $\mathbb{Y}_{c_{i}}$ from ${\cal C}\backslash\mathbb{Y}_{c_{i}}$.

III Finite State Machine Inference

In this section, we implement the models discussed earlier using FSMs. We remind, in [65], the definition of stochastic FSMs333All the definitions and implementations of FSMs are reported in the technical report [65]., in particular finite state acceptors (FSA) and transducers (FST), and we show how we design and combine those machines in order to build a global transducer. The latter encodes in a compact way, the lattice of possible annotations of a given scene and the best annotation corresponds to the best path in that lattice.

Given a scene paved with a collection of non-overlapping superpixels, our labeling model first reorders these superpixels (via a “reordering FSA” $\bf R$) and group them (via a “grouping FST” $\bf G$). These two FSMs when composed together, allow us to generate many reordered partitions of segments444Each segment in these partitions is seen as a “phrase” in a “language” with an alphabet corresponding to all the superpixels of a scene.. Among these partitions, only a few of them are relevant and correspond to meaningful objects in the scene. Therefore, and in order to find these relevant partitions, we combine the $\bf R$ and $\bf G$ FSMs with ${\bf V}\circ{\bf D}$ (resulting from the composition of visual and label dependency FSTs; see example in [65]) that scores segments in all possible partitions, depending on their unary and binary interactions, and returns only the most likely partition and its labels. The most likely solution (partition and its labels) corresponds to the best (shortest) path in the global FSM (${\bf V}\circ{\bf D}\circ{\bf G}\circ{\bf R}$) provided that negative log-likelihood transform is applied to all FSM transition probabilities; this solution also minimizes the energy $-\log P({\bf X},{\bf Y},{\bf C},{\bf k},\pi)$ (see Eq. 1). Figure 2 shows an example of this global FSM.

Note that the global composition process (${\bf F}={\bf V}\circ{\bf D}\circ{\bf G}\circ{\bf R}$) could be time and memory demanding. One may significantly reduce complexity of different transducers (and thereby the global one) at different levels including the visual and the label dependency models by only keeping sparse transitions (related to strictly positive or large statistics). Another simplification consists in reducing the number of possible labels, especially if one is interested in domain specific applications with restricted labels. In practice, with these simplifications (and besides superpixel and feature extraction), FSM inference takes $<$ 2s to annotate an image of $20\times 20$ superpixels using a standard 3Ghz PC.

IV Validation and Discussion

Evaluation Set and Setting. In order to validate the proposed OCS approach, we use the “Sunset04” database. The latter contains 100 sunset scenes with objects belonging to 4 categories (“Sky”, “Sea”, “Sand” and “Sun”); note that this database is challenging and interesting for our model as it has visually confusing categories (such as “Sky vs. Sea”, etc.). Given an image, the goal is to assign each group of pixels (i.e., superpixel) to one of these 4 categories. In practice, a given image is subdivided into a regular grid of 400 ($20\times 20$) superpixels, each one is connected to its 4 immediate neighbors: top, bottom, left, right (see other e.g. [66, 67]) and described using the bag-of-word SIFT representation. Precisely, dense SIFT features are extracted and quantized using a codebook of 200 visual words and a two level spatial pyramid is used to describe each superpixel resulting into a feature vector of 1000 dimensions.

For each image in Sunset04, we turn OCS into an FSM decoding process (as described earlier); half of the Sunset04 database is used in order to train (obtain statistics of) different FSMs (i.e., label dependency and visual models) while the other half is used as a test set for OCS decoding. For each category, we measure the underlying false acceptance rates (FAR) and false rejection rates (FRR) as well as the equal error rates (as average of FAR and FRR) all at the pixel level; we report the mean accuracy as the expectation of these errors through different categories.

Model Evaluation. Table I reports the accuracy of our FSM decoding for different setting of our model. The main goal is to understand the contribution of each step in the FSM decoding/inference process. In this table, “Yes” stands for the use of the grouping+reordering models; the transition probabilities of the random walk grouping model are either uniform (flat) or set (learned) as described in Section II. A “No” stands for no-reordering which means that a given test image is parsed using a unique order (lexicographic in practice). We also consider, in these experiments, flat and learned label dependency and visual models. Flat models mean that all the underlying statistics are uniform while learned models correspond to statistics taken from Eqs. 6-7. In contrast to flat reordering+grouping models, setting flat label dependency (or visual) model is strictly equivalent to the “non-use” of that model as its impact on the decoding process becomes completely neutral.

Impact of Reordering and Grouping Model. Lines 2-4 vs. 5-10, show that the OCS performances are consistently better when using reordering (R) and grouping (G) models as the latter parse and group superpixels with different orders. This results into a better exploration of the space of possible solutions of Eq. 1 (segmentations, labelings and dependencies) and thereby a better OCS performance. Note that learned R and G models (Lines 8-10) always outperform flat ones (Lines 5-7), as the (random walk-based) grouping model gives more preference and better scoring to visually similar superpixels which are more likely to correspond to actual objects.

Impact of Dependency and Visual Models. It is interesting to see that the gain when using the visual model (without dependency model, i.e., lines 2, 5, 8) is more important than the gain obtained when using the dependency model (without visual model, i.e., lines 3, 6, 9), as the former is image/content dependent while the latter acts as a prior/context. However, it is clear that the gain obtained when combining both models is more substantial and always consistent (lines “4 vs. 2”, “7 vs. 5”, “10 vs. 8”). Note that line 1 is strictly equivalent to void dependency and visual models, and the underlying results correspond to a random classifier, that assigns random class labels to superpixels; the same behavior is observed both with and without the reordering and grouping model. This observation shows that the FSM decoding process is able to benefit from reordering and grouping only if visual and dependency models are able to score the resulting segmentation with a decent precision. The converse is also true as the effect of reordering+grouping on the decoding process is clearly important (lines 5-10 vs. 2-4; see also Fig. 3).

V Conclusion

In this paper, we introduced a complete framework for scene parsing and annotation based on finite state machines. The approach allows us to examine and score many configurations in order to achieve segmentation as well as annotation. Instead of generating intractable configurations of segmentations and labelings, the method relies on finite state machines in order to specify a lattice of these configurations and the best one (reordering, segmentation, labeling and scoring) corresponds to the shortest path in that lattice. Experiments show that indeed this method is very effective for object class segmentation.