Industrial DevOps

TL;DR

This paper introduces Industrial DevOps, a methodology for integrating DevOps practices into industrial production to enhance flexibility, observation, and development, demonstrated through energy management and predictive maintenance scenarios.

Contribution

It proposes the Industrial DevOps approach, including a software platform and role model, to facilitate continuous integration and development in industrial environments.

Findings

Successful application in energy management scenario

Effective integration of production systems with DevOps practices

Improved agility and responsiveness in industrial processes

Abstract

The visions and ideas of Industry 4.0 require a profound interconnection of machines, plants, and IT systems in industrial production environments. This significantly increases the importance of software, which is coincidentally one of the main obstacles to the introduction of Industry 4.0. Lack of experience and knowledge, high investment and maintenance costs, as well as uncertainty about future developments cause many small and medium-sized enterprises hesitating to adopt Industry 4.0 solutions. We propose Industrial DevOps as an approach to introduce methods and culture of DevOps into industrial production environments. The fundamental concept of this approach is a continuous process of operation, observation, and development of the entire production environment. This way, all stakeholders, systems, and data can thus be integrated via incremental steps and adjustments can be made quickly. Furthermore, we present the Titan software platform accompanied by a role model for integrating production environments with Industrial DevOps. In two initial industrial application scenarios, we address the challenges of energy management and predictive maintenance with the methods, organizational structures, and tools of Industrial DevOps.

Full text

I Introduction

The digital transformation of the conventional, manufacturing industry enables a new level of automation in production processes. More and more technical machines and production plants become increasingly intelligent and autonomous. Equipped with network capabilities, they are able to consume and supply data to others. This new trend, often referred to as Industrial Internet of Things or Industry 4.0 [1], confronts the production operator with the challenge of connecting and monitoring the individual machines and devices [2]. While software used to be only part of the production process, nowadays software increasingly defines the production process itself [3], so that the resulting software systems inevitably become more complex. In order to ensure a smooth production process these software systems have to be designed with a special focus on reliability, scalability, and adaptability [4].

As is usual with complex systems, however, this complicates the design, development, and maintenance of such a system. In particular for small and medium-sized enterprises (SMEs), this poses enormous challenges as these SMEs often do not have suitable software development departments. Instead, software is often developed by domain experts (e.g., mechanical engineers) with basic programming skills, but no education in software engineering. Alternatives such as establishing a dedicated software development department or delegating software development to specialized external companies are also risky. Business or domain requirements and their technical implementations often diverge, so adaptations are cumbersome, costly, and the time until they are released is long.

Agile and iterative principles, methods, and techniques that are common in other fields of software engineering, such as e-commerce systems [5, 6], can provide solutions to this. Section II of this paper describes how these methods can be transferred to the domain of industrial production environments and highlights necessary changes in processes and culture. Section III presents a software platform with which these methods can be applied in the manufacturing industry. Section IV describes a role model for applying the required organizational and technical structures and Section V outlines our two initial application scenarios. Section VI concludes this paper and points out future work.

II Industrial DevOps

The traditional separation of software operation from its development leads to several issues due to a lack of communication, collaboration, and integration. DevOps [7] is a movement to bridge this gap. We propose Industrial DevOps as an approach for transferring DevOps values, principles, and methods to industrial systems integration. In this way, we expect that development and operation of such an integrating software system will be improved and the discrepancy between production operators and software developers will be reduced, while ensuring high software quality. The following principles are of crucial importance for Industrial DevOps.

II-1 Continuous Adaption and Improvement Process

The core element of Industrial DevOps is a coherent, cyclic, and continuous process as illustrated in Fig. 1. During its operation, the software system as well as the production itself is monitored and analyzed. Based on analysis results, new requirements are identified. The implementation of these requirements will either be done by adapting the software or by reconfiguration in the IT operation or production. Adjustments are tested automatically and after passing all tests, the new software replaces the old one in production, where it is monitored again.

Applying this iterative process aims to enable continuous adaptions and improvements of the software. Thus, complex systems are going to be created by starting from simple ones that are extended in incremental steps. Another advantage can be found in the maintenance of the systems. While IT systems that run in production after a development phase are provided with updates for a certain period of time, followed by a certain period of service, the actual operating time is often considerably longer. As a result, operation becomes increasingly complex and risky, but at the same time migrating to another system becomes more and more difficult. The cyclic process of Industrial DevOps is intended to achieve that development, updates, and service become continuous processes that run permanently parallel to the operation of the system.

II-2 Lean Organizational Structure

When applying external software solutions, this often requires enterprises to adapt their production processes to these solutions. Industrial DevOps comes with an organizational structure to prevent this and instead support the alignment of the software with the production process.

A key factor of Industrial DevOps is that requirements are discussed between all stakeholders and people from different business units are brought together. This practice is referred to as BizDevOps as a more general term [8]. Industrial DevOps extends this by applying BizDevOps to integrating systems and data of industrial production environments. For the implementation of these requirements, domain experts and developers work closely together. Moreover, the development provides appropriate means, which allow domain experts to solve the domain problems themselves to a certain extent. This is accomplished by an extendable software platform, which is owned by the user. Ownership can be direct or indirect if the software is owned by an open source community.

Information from all levels is available for anyone in the value stream through monitoring. Even if the information that is required differs, the source of information is consistent and allows investigation about cause and effect (culture of causality). This should enable lean organizational learning [9].

II-3 Customer-Centric Value Generation

Organizations who apply customer-centric value generation align all departments to contribute to this goal [10]. All activities in the organization which do not directly or indirectly contribute to this goal are abandoned. Some activities which are required for organizational or compliance reasons are organized to stay out of the way of the overall goal. Instead of wasting huge amounts of money in market research, experiments are started in a way that new features, functions, or products are offered to customers and direct measurements are made by utilizing IT to find out the customers response. These measurements are not only done once, but continuously.

III The Titan Platform

Titan is a software platform for integrating and monitoring industrial production environments using Industrial DevOps. In order to integrate tools or infrastructure (e.g., machines, IoT devices, software, or databases), modular software components are created. These components, in Titan called bricks, serve for integrating exactly one such system. This allows on the one hand to exchange the underlying tool or infrastructure easily, but on the other hand also to replace the integrating software implementation. To connect bricks (and thus the underlying systems of the production), Titan applies the principles of flow-based programming [11] and calls a network of connected bricks a flow.

Titan provides a graphical modeling language, which enables the organization to graphically model the integration and, based on this, to configure the integrated system. Hence, there is no programmer required to perform these configurations or (pre-configured) changes. Instead, a person that has received training on the modeling language can work together with a domain expert to do this.

Thus, this platform allows to react efficiently and effectively to changing requirements in production. Furthermore, it enables conducting experiments with little effort and to adjust quickly in case of experiment failures. Titan integrates the required monitoring, which includes resource consumption, system utilization, and business data.

IV The Titan Role Model

Industrial DevOps involves people from different domains, business units, and organizations. The Titan role model (see Fig. 2) defines roles to be assigned to people or groups that use, operate, or develop the Titan platform and describes their responsibilities as well as how they interact.

A Developer is a trained software engineer. This person has learned to develop software in many programming languages. Many different aspects of software development like code quality or version management are covered. In Industrial DevOps, many developers who are spread over many teams in different organizations work together on a single system. The role model introduces the following sub-roles: The Platform Developer works on new features and improvements of the Titan platform itself. The Tool Developer creates software tools or interfaces for hardware that is integrated into the system. The Brick Developer creates bricks and works closely together with the Tool developers.

The IT Operator is trained in installing, configuring, and running IT infrastructure. This person is also responsible for installing, updating, configuring, and running software applications.

The Flow Manager is a person that is closely related to the business, but with a special training to master all aspects of flow modeling. This person works as a connection between business and technology. The Flow Manager implements new flows and flow changes according to business requirements.

The Business Unit Manager can be located on all levels of the hierarchy. This person knows about the business requirements and works together with the Flow Manager, the Domain Expert, and the Process Operator to refine requirements.

The Domain Expert owns deep knowledge about the domain at hand.

The Process Operator benefits from a new flow or a change. This person is involved in utilizing the flow in the actual production environment.

The IT Security Expert is responsible for the IT security domain of the whole organization. The IT Security Expert works together with development and operations through all levels of the system.

As an extension to regular DevOps roles, the Titan role model introduces business roles, domain experts, flow managers, and process operators, which do not need to be developers.

V Initial Industrial Application Scenarios

Enterprises, in particular in the manufacturing industry, face the challenge of reducing and optimizing their energy consumption for economic and ecological reasons. This requires in-depth monitoring and analysis of the individual energy consuming devices, machines, and plants. Doing this, also referred to as energy management, serves as an initial application scenario for Industrial DevOps and Titan. We designed a monitoring infrastructure, which monitors, analyses, and visualizes the electrical power consumption in industrial production environments [12]. Employing the Titan platform, it is able to integrate different kinds of sensors that use different data schemata, formats, and protocols. Referring to the Titan role model, this monitoring infrastructure enables Business Unit Managers, Domain Experts, and Process Operators to gain detailed insights into the energy usage and to identify saving potentials. Thus, it is possible to perform and evaluate optimization measures across departmental boundaries within the company.

Predictive maintenance is another initial application scenario we address with Industrial DevOps and Titan. Titan can serve as a platform, which detects faults in machines and production plants in advance on the basis of data from integrated sensors and systems. An essential aspect here is that not only data from a single machine and production plan is considered for analysis. Instead, measurements and events from of the entire production as well as environmental influences can be taken into account. The predictive maintenance function thus has a much broader amount of data available and can therefore also include non-obvious influences. With the insights gained from the data, maintenance intervals can be optimized. Additionally by returning analysis results to the machines, they can optimize themselves.

VI Conclusions and Future Work

We propose Industrial DevOps as an approach to transfer practices, methods, and culture of DevOps for integrating systems in industrial production environments. It suggests an organizational structure in which domain experts, business managers, developers, and operators work together on solutions. Production environments should be integrated, configured, and operated in a validated learning loop to manage changes and to control costs.

Future work lies in evaluating the proposed principles, methods, and organizational structures of Industrial DevOps in different industrial application scenarios. Therefore, we further develop the Titan platform. To overcome the disadvantages of proprietary integration systems, we will release it as free and open source software. We are already using Clean Code approaches in the development to prevent software quality from degrading over time [13]. Along with including quantitative quality characteristics such as performance in the DevOps cycle [14] as well as rigorous monitoring in operation, this is supposed to contribute in creating a long-living software platform [15].