Interoperability and machine-to-machine translation model with mappings to machine learning tasks

TL;DR

This paper proposes a mathematical framework for semantic translation in cyber-physical systems, enabling interoperability across diverse information models and standards using machine learning techniques.

Contribution

It introduces a translator-based interoperability model with formal definitions and explores mappings to machine learning tasks for automatic semantic translation.

Findings

Mathematical formulation of translator learning tasks

Mappings to machine learning solutions for semantic translation

Discussion on learning translators without common physical context

Abstract

Modern large-scale automation systems integrate thousands to hundreds of thousands of physical sensors and actuators. Demands for more flexible reconfiguration of production systems and optimization across different information models, standards and legacy systems challenge current system interoperability concepts. Automatic semantic translation across information models and standards is an increasingly important problem that needs to be addressed to fulfill these demands in a cost-efficient manner under constraints of human capacity and resources in relation to timing requirements and system complexity. Here we define a translator-based operational interoperability model for interacting cyber-physical systems in mathematical terms, which includes system identification and ontology-based translation as special cases. We present alternative mathematical definitions of the translator learning task and mappings to similar machine learning tasks and solutions based on recent developments in machine learning. Possibilities to learn translators between artefacts without a common physical context, for example in simulations of digital twins and across layers of the automation pyramid are briefly discussed.

Tables

Table 1. TABLE I : Examples of loss/utility functions.

Type	Function	Example
Causation	$J(y^A,u^B)$, $J(u^A,y^B)$	Step responses
Correlation	$J(y^A,y^B)$	Related measurements
Abstract	$J(x^A,x^B)$	Efficiency optimization

Equations

$$m^A\leftarrow E^A(u^A,y^A,x^A;G^A),$$

$$(x^A,u^A)\leftarrow D^A(m^A;x^A,u^A,y^A;G^A),$$

$${\hat{m}}^B$$

$$T^{AB}$$

Full text

I Introduction

Automation systems in Industry 4.0 [1, 2] and the Internet-of-Things (IoT) [3] are designed as networks of interacting elements, which can include thousands to hundreds of thousands of physical sensors and actuators. Efficient operation and flexible production require that physical and software components are well integrated, and increasingly that such complex automation systems can be swiftly reconfigured and optimized on demand using models, simulations and data analytics [4, 5]. Achieving this goal is a nontrivial task, because it requires interoperability of physical devices, software, simulation tools, data analytics tools and legacy systems from different vendors and across standards [6, 7, 8, 4]. Standardisation of machine-to-machine (M2M) communication, like the OPC Unified Architecture (OPC UA111https://opcfoundation.org/about/opc-technologies/opc-ua/) [9] which offers scalable and secure communication over the automation pyramid, and development of Service Oriented Architectures (SOA), like the Arrowhead Framework [10], are developments supporting the vision of interoperability in Industry 4.0 and the IoT.

However, in addition to data exchange by protocol-level standardisation and translation [8], information models are required to correctly interpret and make use of the data. There are many different information models and standards defining semantics of data and services, which are developed and customized to fit different industry segments, products, components and vendors. This implies that the problem to translate data representations between different domains is increasingly relevant for robust on-demand interoperability in large-scale automation systems. This capacity is referred to as dynamic interoperability [4], and operational interoperability [7] meaning that systems are capable to access services from other systems and use the services to operate effectively together. Thus, focus needs to shift from computing and reasoning in accordance with a representational system to automatic translation and computing over multiple representational systems [11, 4] and engineers should operate at the levels where system-of-systems goals and constraints are defined.

In this paper, we outline a mathematical model of the problem to translate between representational systems in cyber-physical systems (CPS) with integrated physical and software components, and we map some alternative definitions of the translation problem in this model to machine learning tasks and the corresponding state-of-the-art methods. In this model, concepts like symbol grounding, semantics, translation and interpretation are mathematically formulated and possibilities to more automatically create semantic translators with machine learning methods are outlined.

II Interoperability model

When integrating SOA systems and services, which are designed according to different standards and specifications, various interfaces that are also subject to domain-specific assumptions and implementation characteristics need to be interconnected. It is common practice to engineer the connections between such interfaces in the form of software adapters that make different components, data, services and systems semantically interoperable, so that functional and non-functional system requirements can be met. This way, a modular structure is maintained, which makes testing and the eventual replacement of a module and updates of the related adapters tractable in otherwise complex systems, at the cost of a quadratic relationship between the number of adapters and the number of interfaces.

In deterministic protocol translation, where representational and computational completeness allows for the use of an intermediate “pivot” representation of information, the quadratic complexity of the adapter concept can be reduced to linearity, see for example [8] and [12]. However, in the case of semantic translation considered here, it is not clear that such universal intermediate representations exist and constitute a resource-efficient and feasible approach to translation. Furthermore, the research field of dynamic and operational interoperability in SOA lacks a precise mathematical formulation and consensus about the key problem(s). Therefore, we approach the translation problem by formulating it in precise mathematical terms that can be mapped to machine learning tasks.

We define the M2M interoperability problem in terms of translator functions, $T^{AB}$, which map messages, $m^{A}$, from one domain named CPS A to messages in another domain, $m^{B}$, named CPS B, see Figure 1. The translators can be arbitrarily complex functions that are generated as integrated parts of the overall SOA, thereby maintaining a modular architecture as in the case of engineered adapters. In general, the translated messages, $\hat{m}^{B}$, cannot be semantically and otherwise identical to the messages communicated within CPS B, $m^{B}$, but we can optimize the translator functions to make the error small with respect to an operational loss or utility function. In the following, we elaborate on the latter point and introduce the additional symbols and relationships of the model as the basis for defining translator learning tasks, which in principle can be addressed with machine learning methods.

The model is divided in three levels: cyber (white), physical representation (light gray) and the shared physical environment (gray), see Figure 1. At the cyber level, the graphs $G^{A}$ and $G^{B}$ define all discrete symbolic and sub-symbolic metadata that is specific for CPS A and CPS B, respectively. For example, the nodes and edges of these graphs can represent subject, predicate, and object semantic triples defined in the Resource Description Framework (RDF). Each CPS also has discrete internal states, $x^{A}$ and $x^{B}$ respectively, such as the computer program variables of all devices in a CPS, which are not directly readable or writeable in the SOA but may be read and modified indirectly via the messages and services. The environment has inputs, $u$, which can be affected by actuator devices, and outputs, $y$, which can be measured with sensor devices. In CPS A, the outputs of the sensor devices are represented at the cyber level as discrete variables $y^{A}$ and the actuators are controlled by discrete variables $u^{A}$, and similarly for CPS B. From the viewpoint of causality, $u$ influences $y$ and thus changes of elements of $u^{A}$ may influence the values of elements in both $y^{A}$ and $y^{B}$, and vice versa.

Messages are generated by encoder functions on the form

[TABLE]

which typically are implemented in the form of computer programs. Similarly, the internal states are updated by decoder functions

[TABLE]

which are matched to the corresponding encoder functions. However, a decoder $D^{B}$ can in general not be combined with an encoder $E^{A}$, and vice versa.

Although some technical details and challenges are hidden in this abstract model, the model enables us to define concepts and relationships that otherwise are ambiguous and described differently in the literature depending on the context. The task to model dynamic relationships between $u$ and $y$ in terms of $u^{A}$ and $y^{A}$ (or $u^{B}$ and $y^{B}$ etc) is the central problem of system identification [13]. The task to model and control one CPS in terms of the relationships between $u^{A}$, $y^{A}$, $x^{A}$ and sometimes also $G^{A}$ is more complex [14] and typically involves hybrid models with state-dependent dynamic descriptions. This is a central problem in automatic control and CPS engineering.

Symbol grounding [15] refers to the relations between a symbol defined by $G^{A}$ and the related discrete values of $\{x^{A},u^{A},y^{A}\}$ (similarly for $G_{B}$) and the property of the environment $\{u,y\}$ that the symbol represents. A grounding problem appears when a symbol defined in $G^{A}$ have an underfitted relationship to the referenced property of the environment represented via $\{x^{A},u^{A},y^{A}\}$ (similarly for $G_{B}$), such that symbols in $G^{A}$ and $G^{B}$ cannot be conclusively compared for similarity although both systems are defined in the same environment. Therefore, symbol grounding is just as relevant for translator learning as it is for reliable inference in cognitive science and artificial intelligence.

Listing 1 presents two examples of SenML messages that are constructed to illustrate the character of a semantic translation problem, $\hat{m}^{B}=T^{AB}(m^{A})$.

Both messages encode information about the temperature in one office at our university and thus represents related physical properties. A and B can for example refer to the heating and ventilation systems in the office, respectively, and thus the temperatures are not necessarily identical. The message from System A includes the service URI and the time, longitude and latitude of the temperature measurement with unit ‘K’ for Kelvin and numeric value $293$. The message from System B includes the name of the temperature sensor, the unit “Cel” for Celsius, the value $-20.4$ and the time of the temperature measurement.

A translator, $T^{AB}$, could in this scenario for example be used by an indoor climate and energy optimization service that is capable to interpret messages of the second kind in Listing 1, but not of the first kind. By using the translator this service could for example improve the quality of the indoor climate, further reduce the energy used, or be more fault resilient in case of a sensor fault. As outlined above, messages encoded by CPS A in Figure 1 can in general not be correctly interpreted by CPS B, and vice versa. How can a translator that solves this problem be automatically generated?

We approach this problem by defining a computable function, $J$, that determines to what extent the system-of-systems (SoS) formed by CPS A, CPS B etc fulfils particular operational requirements and goals. For example, the function $J$ could be formulated as a loss function in machine learning, or a utility function of a multi-agent system, and the translator learning task is to minimize the loss or maximize the expected utility. Some possible definitions of the function $J$ are listed in Table I.

The key points are that engineering resources are focused on defining $J$ in terms of SoS goals and requirements, and that it is possible to optimize $J$ by defining and updating $T^{AB}$ using machine learning methods on the form

[TABLE]

and similarly for other choices of $J$ and in the case of expected utility maximization.

For example, in the office example introduced above, $J$ could be a causality type loss and $T^{AB}$ could be a recurrent neural network, which is trained until the ventilation system decodes $\hat{m}^{B}$ so that the effects of varying $u^{B}(m^{B};\ldots;\hat{m}^{B})$ on $y^{A}$ are correctly predicted across instrumented offices.

In general, the translator function should depend on symbols in $G^{A}$ and $G^{B}$, and it can depend also on other information sources, like public datasets [16] and historical CPS data used to fit sub-symbolic relationships more accurately. In principle, the translator $T^{AB}$ can be considered to perform three tasks:

Estimate the decoder, $D^{A}$. 2. 2.

Map information from domain A to domain B. 3. 3.

Estimate the encoder, $E^{B}$.

Like in the field of machine translation of natural language we can attempt to explicitly model these individual mappings, or we can model the overall mapping $T^{AB}$. We elaborate on machine learning tasks and methods that may be useful to address the translator learning task outlined above after briefly introducing some related work in the next section.

III Related work on interoperability solutions

To fully exploit the potential of the IoT and Industry4.0, engineering resources should to a larger extent be focused on high-level benefits of interoperability and system integration [4]. Automated approaches to establish and maintain interoperability are needed to enable on-demand service composition and meet the demands for flexible production and high efficiency given the high complexity and diversity of automation systems driven by the rapid technological development [6]. Architectures similar to the model presented here have been independently developed by Maló [12] who describe architectures that allow for maximum interoperability. Our model describes the specific task to translate between services and data formats, but can in general be considered as a special case of the architectures considered in that work. Concepts and methods developed for the semantic web [17] are widely used to integrate human- and machine-readable metadata to support the adapter engineering and system integration processes, such as ontologies, ontology alignment and ontology-based reasoning engines. The semantic web tags websites with ontological metadata, typically encoded in RDF or higher-level ontology languages like the Web Ontology Language (OWL). The Semantic Sensor Network (SSN) [18], an ontology specialized for describing sensors, is one example of a domain-specific ontology. The Open Semantic Framework (OSF) [19] combines many such specific ontologies into an extendable framework, fusing both general and domain specific knowledge. Ontologies form the core of semantic technologies, but not all ontologies can be combined and function together. Ontologies that are based on different standards and definitions can model related physical and cyber entities in different ways, thus leading to contradictions and under-determined relationships between symbols when different technologies are combined.

In addition to semantic interoperability, which focuses on supporting the engineering process with such standardized metadata models, methods and tools for automatic on-demand dynamic interoperability [4] and operational interoperability [7] are developed. Symbolic reasoners can be applied to create Web-like mashups in highly dynamic environments [20], but suffers from state-space explosion when physical states are included. This challenge is recognized also in the domain of symbolic artificial intelligence. Furthermore, automatic reasoning in terms of symbolic metadata is unreliable in complex and uncertain real-world environments because symbolic data does not include all necessary information about the context, environment and system (cf. comments on symbol grounding and underfitted symbol relationships in the former section). Therefore, ontology-based translation is extended with sub-symbolic mapping and reasoning mechanisms. A recent example in this direction is deep alignment of ontologies [16]. Deep alignment enables discovery of sub-symbolic mappings between elements of ontologies by a data-driven optimization method, where textual descriptions are represented by word-vectors learned from an auxiliary data set, similar to techniques used in natural language processing.

The development of more potent interoperability methods and technologies are of central importance for modern SOA, like the aforementioned Arrowhead Framework. For example, ontology-based XML-message translation has been extended with semantic annotations [21], see also former work in [12]. That translator can map elements, perform unit conversion, detect missing data and, in certain cases, find and add the missing data. Another example is the architecture for device management using autonomic computing [22], where a manager monitors and plans execution using ontologies and a reasoning engine.

Data lakes, like the The Big Data Europe platform222https://www.big-data-europe.eu/platform/, is another approach where heterogeneous data annotated with RDF metadata are combined to allow querying, machine learning and inference across different representational domains. The metadata model considered in this context is based on similar concepts, but the problem addressed is different compared to the problem of dynamic and operational interoperability of SOA services in CPS systems.

IV Mapping to Natural Language Processing

Vector embedding of sub-symbolic relations is a powerful concept often applied in natural language processing (NLP). Vector embeddings of words, sentences and contexts enable mappings between words in terms of vector operations in an $n$-dimensional space, where typically $n>100$. Initially, relatively simple vector space models [23] were used and can represent some important word and document relations [24]. Lately, neural network based approaches like Word2Vec [25] have shown great performance, and thereby the use of simpler embeddings like one-hot vectors have mostly disappeared. Word2Vec maps the words (symbols) to a manifold in a vector space, thereby creating model with sub-symbolic representations. Recent advancements have been achieved using attention models [26] to create embeddings that produce different mappings for the same word given different contexts [27]. Sub-symbolic vector embeddings of this type has recently been used for ontology alignment purposes [16].

Work in the field of machine translation is another important source of examples and guidance. Translation based on traditional statistics have recently been outdated by neural machine translation (NMT) as the state of the art. This switch was exemplified by Google, who have been using NMT for their translation service since 2016 [28]. An upgrade to the translation system allowed them to translate between unseen language pairs [29], a process they call as zero-shot translation. The translation system in [28] uses recurrent neural networks with attention. More recent translation systems use pure attention models based on the transformer model [26] to achieve state-of-the-art results [30]. All translation systems referenced above are based on word or sub-word input features. There are also examples of fully character-level convolutional approaches [31].

To achieve good results on NLP tasks, the training protocol is of key importance. A major recent advancement in NLP is the step to semi-supervised pre-training. With pre-training, a language model in the form of a neural network is created using a large dataset, which can subsequently be fine-tuned for other problems with little data and computational resources. One of the latest improvements in semi-supervised pre-training is BERT333http://jalammar.github.io/illustrated-bert/ [27], which can be downloaded in pre-trained form. It is an exciting open problem to adapt these concepts and recent technological advancements to the problem of sub-symbolic ontology alignment and more generally to the M2M translator learning task introduced in Section II.

V Mapping to Graph Neural Networks

Graph neural network (GNN) models are relatively new in the machine learning field. Conventional architectures like recurrent- and convolutional neural networks are based on the assumption that there is repetitive structure in the input. In contrast, GNNs cannot be based on that assumption because graph data is more irregular in nature, see [32] and [33] for an overview of the field. Several concepts from the image recognition and NLP fields have been adapted to GNNs, like graph convolution [34], graph attention [35] and graph embeddings [36]. The resulting methods have been successfully used for example to study molecule structures in chemistry and perform traffic route planning [32].

An interesting development in the field of semantic technologies is RDF2Vec [36], which is an extension of the Word2Vec model to graph embeddings. Much like word embedding, graph embedding is a powerful tool to represent graphs in a metric space, where for example graph clustering and similarity tasks can be addressed. The Relational Graph Convolutional Network (R-GCN) [37] is another interesting recent development for processing of RDF-graphs, which is based on the message-passing network architecture in [34]. By allowing for different convolution operators for different kinds of edges the R-GCN represents RDF-data more effectively than if all edges are treated the same. The R-GCN is validated on entity classification and link prediction tasks.

GNNs are currently actively developed and offer interesting new possibilities to perform graph embeddings and data-driven ontology alignment and mappings between $G^{A}$, $G^{B}$, $G^{C}$ etc needed to address the M2M translator learning task outlined in Section II.

VI Discussion: Translator Learning Strategies

Inspired by the recent developments in NLP it is tempting to adopt an encoder-decoder translation scheme, similar to that in [28] or [31] (see Figure 2(a)). These models are typically trained end-to-end (E2E) with pairs of known translations, using a message reconstruction loss on the form $J(\hat{m}^{B},m^{B})$. This is feasible in NLP where large repositories of such translation pairs have been developed. The M2M translation case is challenging because the repertoire of input representations, “languages” and “dialects” is diverse and more quickly growing, and data sets contains information about production and operations that typically cannot be publicly shared for the collection of such large data sets. Thus, we will likely have access to less data and relatively few known translation pairs, since identifying and tagging these pairs is costly and time consuming, which is challenging for obtaining a scalable on-demand interoperability solution.

Accurate one-to-one word translation is not always possible in NLP. For example, a round-trip translation of the Swedish word “Lagom” to English with Google Translate results in “Moderate”, followed by “Måttlig”, which is semantically related albeit different (sub-symbolic representations are different in most Swedish native speakers). This is expected because there is no one-to-one mapping between that Swedish word and a word in the domain of English language. However, the meaning of the word “Lagom” can essentially be explained to an English native speaking person by a longer description, with one or a few follow-up questions needed to validate and further align the interpretation of the concept. Similarly, some messages in CPS A might require several messages to be accurately represented in CPS B, and vice versa. That is why we define the translators, $T^{AB}$, as an integrated part of the overall SOA of the SoS, such as the aforementioned Arrowhead Framework. This way it is possible, in principle, that the translator requests or provides additional information needed to proceed with a translation. For example, although the messages in Listing 1 refers to the same location, an external information source is needed to identify this relationship, for example as described in [21]. It should be noted that NLP translations of this type are currently challenging to learn.

Instead of learning the translators in an E2E fashion, it is possible to use the messages communicated within each system as a starting point. Even if we cannot expect to have access to large data sets of prealigned A–B message pairs, we do expect high rates of internal messages in each CPS. Thus, we can optimize the embeddings of the messages in each domain separately and make use of the common environmental degrees of freedom to learn relationships between such embeddings. For example, vector space embeddings can be learned in the form of latent representations of autoencoders as illustrated in Figure 2, and in this context methods and concepts that are successfully used for NLP can be reused and further developed. Translations between the latent spaces of the CPS A and CPS B encoders can for example be learned by solving Equation 4 using loss/utility functions of the type listed in Table I. Such an autoencoding scheme does not solve the problem of missing translation pairs. However, with sub-symbolic representations of symbols that are optimized with metadata and data from the physical domain, the problem to learn mappings between symbols is simplified and enables faster convergence and learning with less data. It also enables clustering and classification of messages, which is useful to improve training and testing protocols. The use of auxilliary goals have for example helped when solving NLP tasks [27].

A final remark in this discussion concerns the nature of the environment, which up to this point was considered to be reality, so that the mathematical relationships between $u$ and $v$ can be described in terms of physical models. The translator learning task introduced in Section II is not limited to natural environments because it only requires that the systems have related degrees of freedom in the environment. If the transformations from $\{u,v\}$ to $\{u^{A},v^{A}\}$ and $\{u,v\}$ to $\{u^{B},v^{B}\}$ are orthogonal there is little that can be learned using the approach proposed here. However, in systems of our primary interest correlations between some $y^{A}$ and $y^{B}$ are expected and causal relationships are expected between some $u^{A}$ and $y^{B}$, and vice versa. This is the case also for interconnected simulation models like digital twins and at higher levels and across levels of the (ISA-95) automation pyramid because most systems and services do not function independently of the others.

VII Concluding remarks

Industrial IoT and Industry 4.0 require adaptable solutions to deal with the high heterogeneity of systems and data. In this paper, we have presented a mathemathical interoperability model in which we can describe data-, dynamic- and operational interoperability as machine learning tasks. Unlike previous works, which focus on interoperability as an engineering task, our model allows engineers to define operational goals which can be used for automatic optimization of translators. The model is flexible and can be used with a variety of machine learning tools and methods.

Using the model, we propose learning strategies based on advances in natural language processing and graph neural networks, allowing for grounded translators. Symbol grounding is achieved using sub-symbolic representations learned in a shared environment. In this paper we have mostly assumed that the shared environment is physical, but in principle the shared environment could be any environment suitable for fitting sub-symbolic relationships, for example simulations involving digital twins. Using digital twins, translators can be trained and tested virtually, potentially reducing the time-to-deployment and probability of errors.

While engineered adapters based on ontology alignment and proof engines are explainable, and eventual problems that occur at runtime can be analyzed and solved by the engineers, translators generated with machine learning methods can be more challenging to comprehend. This is something industry often find undesirable. Data availability is also an issue since there are no large public data sets of semantically dissimilar M2M-type messages available as far as we know.

In future work we aim to address these issues and provide proof of concept of the model and translator learning task using a simulated environment.

VIII Acknowledgements

We thank Magnus Sahlgren for helpful advice in the area of natural language processing and Sergio Martin del Campo for helpful comments on an early version of the manuscript.