Observability and Chaos Engineering on System Calls for Containerized Applications in Docker

TL;DR

This paper introduces ChaosOrca, a novel fault injection system for system calls in containerized applications, enabling resilience evaluation under production-like workloads without application instrumentation.

Contribution

ChaosOrca is the first system to perform non-intrusive, comprehensive system call fault injection in containerized environments for resilience testing.

Findings

ChaosOrca effectively detects resilience weaknesses in real-world applications.

System call perturbations reveal vulnerabilities in application fault tolerance.

The approach operates under production-like workloads without modifying applications.

Abstract

In this paper, we present a novel fault injection system called ChaosOrca for system calls in containerized applications. ChaosOrca aims at evaluating a given application's self-protection capability with respect to system call errors. The unique feature of ChaosOrca is that it conducts experiments under production-like workload without instrumenting the application. We exhaustively analyze all kinds of system calls and utilize different levels of monitoring techniques to reason about the behaviour under perturbation. We evaluate ChaosOrca on three real-world applications: a file transfer client, a reverse proxy server and a micro-service oriented web application. Our results show that it is promising to detect weaknesses of resilience mechanisms related to system calls issues.

Tables

Table 1. Table 1: Descriptive Statistics of The Subject Programs

Application	Desc.	Lang.	LoC	#Containers
TTorrent	File downloading tool	Java	14.3K	1
Nginx	Reverse proxy	C	141.1K	1
Bookinfo	Micro-service based web application	Java, Python, Ruby	10.0K	6

Table 2. Table 2: The Statistics of System Calls per Second for Each Application under The Given Workload without Any Perturbations

System Call	TTorrent	Nginx	Bookinfo
accept	1780 (9.92%)		293.45 (3.90%)
accept4		21.21 (1.17%)
bind			173.57 (2.31%)
clone		166.57 (9.17%)	232.77 (3.09%)
close	92 (0.51%)		280.37 (3.73%)
connect			163.03 (2.17%)
ctl		19.24 (1.06%)
dup			175.95 (2.34%)
dup2	64.31 (0.36%)
exit	62.4 (0.35%)		174.36 (2.32%)
fcntl			161.89 (2.15%)
futex	3240 (18.06%)		854.96 (11.36%)
getpid		160.52 (8.84%)	159.6 (2.12%)
getsockname		48.77 (2.69%)	174.84 (2.32%)
getsockopt			167.27 (2.22%)
ioctl			189.22 (2.51%)
list			247.43 (3.29%)
lseek	< 0.01	< 0.01
madvise	79.28 (0.44%)		176.86 (2.35%)
mmap	191.09 (1.07%)		< 0.01
mprotect	677 (3.77%)		15.74 (0.21%)
munmap	0.03 (< 0.01%)		< 0.01
newfstat	< 0.01	140.56 (7.74%)	206.17 (2.74%)
newlstat
newstat		89.96 (4.95%)	1300 (17.27%)
newuname
open	< 0.01	166.15 (9.15%)	222.19 (2.95%)
poll			181.12 (2.41%)
pread64	1410 (7.86%)	99.37 (5.47%)
pwrite64	1520 (8.47%)	147.4 (8.12%)
read	3570 (19.90%)		158.48 (2.11%)
recvfrom		90.37 (4.98%)	130.69 (1.74%)
recvmsg			174.7 (2.32%)
select			388.57 (5.16%)
sendfile64		264.61 (14.57%)
sendto			161.78 (2.15%)
setsockopt		29.41 (1.62%)	287.41 (3.82%)
shutdown			170.71 (2.27%)
sigprocmask	46.78 (0.26%)
socket			263.94 (3.51%)
wait	3040 (16.95%)
write	2110 (11.76%)	87.04 (4.79%)	238.31 (3.17%)
writev		284.63 (15.68%)
yield	55.18 (0.31%)

Table 3. Table 3: Representative Sample of Chaos Experiments on TTorrent

0     System Call	Error Code	Delay	Trend of Syscalls	Network IO	Cpu Usage	Memory Usage
1     open	-	1s	fewer, spike	small dip	increase	normal
2     open	-	5s	increase	dip	dip, increase	normal
3     open	EACESS	-	increase	normal	increase	normal
4     open	EACESS	1s	increase	small dip	increase	normal
5     open	EACESS	5s	increase	dip	dip, increase	normal
6     write	-	1s	tiny	tiny	dip	normal
7     write	-	5s	small	tiny	dip	normal
8     write	EACESS	-	tiny	stops	tiny, tiny	normal
9     write	EACESS	1s	tiny, tiny spike	stops	tiny, tiny	normal
10     write	EACESS	5s	tiny, small spike	stops	tiny, tiny	normal
11     write	EPERM	-	tiny	stops	tiny, tiny	normal
12     write	EPERM	1s	tiny, small spike	stops	tiny, tiny	normal
13     write	EPERM	5s	small, increase	tiny, small	tiny, small	normal
14     write	ENOENT	-	small, spike	stops	tiny, tiny	normal
15     write	ENOENT	1s	spike, tiny	stops	tiny, tiny	normal
16     write	ENOENT	5s	tiny, small spike	stops	tiny, tiny	normal
17     write	EIO	-	small, small spike	stops	tiny, tiny	normal
18     write	EIO	1s	small, small spike	stops	tiny, tiny	normal
19     write	EIO	5s	small, small spike	stops	tiny, tiny	normal
20     read	-	1s	big dip	big dip	big dip	normal
21     read	-	5s	big dip	big dip	big dip	normal
22     read	EACESS	-	tiny	stops	tiny, tiny	normal
23     read	EACESS	1s	tiny	stops	tiny, tiny	normal
24     read	EACESS	5s	small, small	stops	tiny, tiny	normal
25     read	EPERM	-	tiny	stops	tiny, tiny	normal
26     read	EPERM	1s	tiny	stops	tiny, tiny	normal
27     read	EPERM	5s	tiny	stops	tiny, tiny	normal
28     read	ENOENT	-	tiny	stops	tiny, tiny	normal
29     read	ENOENT	1s	tiny	stops	tiny, tiny	normal
30     read	ENOENT	5s	small	stops	tiny, tiny	normal

Table 4. Table 4: Representative Sample of Chaos Experiments on Nginx

0     System Call	Error Code	Delay	HTTP Res.	Latency	Trend of Syscalls	Network IO	Cpu Usage	Memory Usage
1     open	-	1s	0	up after 50ms	fewer during	tiny, spike	tiny, spike	increasing
2     open	-	5s	0	up after 96ms	FEWER, spike	tiny, spike	tiny, spike	increasing
3     open	EACCES	-	403	0	fewer	small	small	increasing
4     open	EACCES	1s	403 (fewer)	increasing 102ms	fewer, spike	tiny, spike	tiny, spike	increasing
5     open	EACCES	5s	403 (FEW)	increasing 140ms	FEWER, spike	tiny, spike	tiny, spike	increasing
6     open	EPERM	-	500	0	fewer	small	smaller	increasing
7     open	EPERM	1s	500 (fewer)	increasing 57ms	normal, spike	tiny, spike	tiny, spike	increasing
8     open	EPERM	5s	500 (FEW)	increasing 175ms	FEWER, spike	tiny, spike	tiny, spike	increasing
9     open	ENOENT	-	404	0	fewer	small	small	increasing
10     open	ENOENT	1s	404 (fewer)	increasing 150ms	normal, spike	tiny, spike	tiny, spike	increasing
11     open	ENOENT	5s	404 (FEW)	increasing 97ms	FEWER, spike	tiny, spike	tiny, spike	increasing
12     open	EIO	-	500	0	fewer	small	smaller	increasing
13     open	EIO	1s	500 (fewer)	increasing 96ms	increase, spike	tiny, spike	tiny, spike	increasing
14     open	EIO	5s	500 (FEW)	increasing 172ms	FEWER, spike	tiny, spike	tiny, spike	increasing
15     open	EINTR	-	500	0	fewer	small	smaller	increasing
16     open	EINTR	1s	500 (fewer)	increasing 96ms	fewer, spike	tiny, spike	tiny, spike	increasing
17     open	EINTR	5s	500 (FEW)	increasing 70ms	FEWER, spike	tiny, spike	tiny, spike	increasing
18     open	ENOSYS	-	500	0	normal	small	smaller	increasing
19     open	ENOSYS	1s	500 (fewer)	increasing 149ms	normal, spike	tiny, spike	tiny, spike	increasing
20     open	ENOSYS	5s	500 (FEW)	increasing 159ms	FEWER, spike	tiny, spike	tiny, spike	increasing
21     write	-	1s	0	up after 75ms	fewer, spike	tiny, spike	tiny, spike	increasing
22     write	-	5s	0	up after 51ms	fewer, spike	tiny, spike	tiny, spike	increasing
23     write	EACCES	-	200	0	normal	normal	increase	increasing
24     write	EACCES	1s	200 (FEW)	up after 56ms	fewer, spike	tiny, spike	tiny, spike	increasing
25     write	EACCES	5s	0	up after 107ms	FEWER, spike	tiny, spike	tiny, spike	increasing
26     read	-	1s	200	0	normal	normal	increase	increasing
27     read	-	5s	200	0	normal	normal	increase	increasing
28     read	EACCES	-	200	0	normal	normal	increase	increasing
29     read	EACCES	1s	200	0	normal	normal	increase	increasing
30     read	EACCES	5s	200	0	normal	normal	increase	increasing
31     sendfile	-	1s	0	up after 35ms	fewer, spike	tiny, spike	tiny, spike	increasing
32     sendfile	-	5s	0	up after 172ms	FEWER, spike	tiny, spike	tiny, spike	increasing
33     sendfile	EACCES	-	0	0	increase	small	normal	increasing
34     sendfile	EACCES	1s	0	up after 59ms	normal, spike	tiny, spike	tiny, spike	increasing
35     sendfile	EPERM	-	0	0	normal	small	normal	increasing
36     sendfile	EPERM	1s	0	up after 64ms	fewer, spike	tiny, spike	tiny, spike	increasing

Table 5. Table 5: Representative Sample of Chaos Experiments on Bookinfo

System Call	Error Code	Delay	HTTP Res.	Latency	Trend of Syscalls	Network IO	Cpu Usage	Memory Usage
open	-	1s	200 / 302	increase 5ms	normal	normal	increase	normal
open	-	5s	200(lower) / 302	up after 2ms	increase	normal	increase	increase
open	EACESS	-	200(lower) / 302	0	fewer, up	smaller	increase	normal
read	-	1s	200(fewer) / 302	increasing 2ms	increase	smaller	increase	increase
read	-	5s	200(FEW) / 302	up after 4ms	increase, spike	small, spike	increase, spike	increase
read	EACESS	-	0	0	normal	small	smaller	normal
read	EACESS	1s	0	up after 1ms	increase, spike	small, spike	smaller, spike	increase
read	EACESS	5s	0	up after 1ms	normal, spike	small, spike	smaller, spike	increase
select	-	1s	200(FEW) / 302	up after 6ms	normal, spike	tiny, spike	smaller, spike	increase
select	-	5s	0	up after 12ms	tiny, spike	tiny, spike	small, spike	increase
select	EACESS	-	crash	-	-	-	-	-
select	EACESS	1s	crash	-	-	-	-	-
select	EACESS	5s	crash	-	-	-	-	-
select	EPERM	-	crash	-	-	-	-	-
select	EPERM	1s	crash	-	-	-	-	-
select	EPERM	5s	crash	-	-	-	-	-

Full text

1 Introduction

Docker containers are increasingly getting adopted as an efficient mean to package and deploy applications [23]: over two million container images are available on DockerHub and popular images are downloaded millions of times. A typical system is composed of a number of Docker containers interacting with each others. Each container is an instance of an image, which describes the software stack that is needed by the application, starting from the OS, for example debian or alpine, all the required dependencies, and so on. The compiled application and configuration files are copied in the image. The images used in production are typically optimized for performances and security, meaning that anything that is not specifically required by the application is stripped-down: in the Docker image there is no ssh access to log into the container, no remote debugger, and so on.

Docker is a key-enabling technology for microservices, an architectural style promoting simple, loosely-coupled, individually updateable services organized around business products [7, 35]. Rather than imposing a generic framework that all microservices need to implement, a key principle of microservices is that different microservices share no code: each microservice is implemented using the technological stack that is best suited. This stack is then specified as a portable Docker image. As a result, different microservices significantly differ one from another: they can be implemented in different programming languages, use different libraries, and so on. As Docker containers rely on the kernel of the host OS, the only thing they all have in common is the kernel. Thus, they all rely on system calls to delegate certain actions to the guest operating system (OS) kernel, such as networking or writing to a file.

Docker and microservices bring many benefits, in particular increasing the rate at which changes can be brought to production: for example, Netflix releases hundreds of changes to production every day. However, while the state of a monolithic system is rather binary, it either works according to the SLA or not, the state of a complex system built around microservices emerges from their interactions. In a monolithic system, a common framework or middleware would typically take care of providing reliable communications between components. This is no more the case with microservices, where each microservice needs to cope with the failures of other microservices and with the failures of the underlying stack. In this context, how to observe, analyze and improve the resilience of systems built over Docker and microservices?

This paper presents ChaosOrca, a tool to conduct chaos engineering [27] experiments to assess the resilience of any Docker-based microservices. Following the principles of chaos engineering [14], ChaosOrca supports the following.

First, ChaosOrca supports formalizing the steady state of the container, by automatically recording system metrics such as CPU and RAM consumption, network I/O and the system calls invoked by the container. In addition, ChaosOrca supports additional metrics provided by developers. Second, ChaosOrca supports triggering events than can impact the steady state of the container, by injecting perturbations into the system calls invoked by the containers. For a given system call invoked by the container, a perturbation can delay the actual execution of the system call or can force returning an error code. Third, ChaosOrca enables developers to specify experiments on the production system. By default, ChaosOrca perturbs the $N$ most invoked system calls for a given container, and monitors it before, during and after the experiment. It then provides a report to the developers summarizing the experiment, for them to verify of falsify the resilience hypothesis for that container. Finally, ChaosOrca is built on isolation, it allows experiments to target specific containers, while the other containers running aside are not be impacted by the injected perturbations.

A key feature of ChaosOrca is that it is oblivious from the internals of the Docker containers involved in chaos experiment. In other words, no matter what software stack runs inside the container, ChaosOrca will always be able to involve that container in a chaos experiment. Our empirical evaluation on TTorent (written in Java), Nginx (written in C) and Bookinfo (written in Java, JavaScript, Python and Ruby) shows that ChaosOrca 1) can observe Docker-based microservices of very different nature, 2) can identify issues related to the resilience of those microservices against perturbation in system calls, 3) can improve the observability of microservices and the relevance of chaos experiments and 4) it can be applied in production with low overhead, though not all kinds of overhead are equal.

To sum up our contributions are:

• •

A novel architecture for achieving observability and system call perturbation for containerized applications in Docker, able to analyze the resilience of Docker applications in a black-box manner.

• •

A prototype implementation based on the state-of-art libraries used in industry, applicable to off-the-shelf Docker microservices. The prototype is called ChaosOrca and is made publicly available at https://github.com/KTH/royal-chaos/tree/master/chaosorca.

• •

An evaluation of the approach on three case studies, representing typical polyglot microservices in Docker, namely TTorent (written in Java), Nginx (written in C) and Bookinfo (written in Java, JavaScript, Python and Ruby)

The remainder of the paper is organized as follows. Section 2 gives a brief introduction about relevant concepts. Section 3 discusses about the design of ChaosOrca like the requirements, components and implementation. Section 4 answers the 4 research questions by discussing about the evaluation experiments. Section 6 compares the related work in fault injection and observability. Section 7 concludes the whole paper.

2 Background

2.1 System Call

A system call is the fundamental interface between an application and the kernel [45]. In the Linux operating system, there are more than 300 unique system calls and more than 100 different error codes that can be returned from system calls [11]. System calls are primitives exposed by the OS kernel to deal with crucial resources like hardware devices, network or files in a systematic and secure manner. When an application requests the use of such a resource, it has to interact with the OS kernel through a system call.

It is very common for applications to invoke system calls, even if developers are not aware of such invocations. Figure 1 is a flame graph [30] that illustrates user events and kernel events when downloading a file with TTorrent. In the figure, the green (light) blocks stand for user events, caused by Java code. The red (dark) ones are events related to system calls inside the kernel. The width of the bars stand for the frequency at which that function is present in the stack traces, or part of a stack trace ancestry.[30] For TTorrent, those system calls mostly happen when it opens a file, uses network resources or writes downloaded blocks into a file.

2.2 Containerization

Containerization is a lightweight virtualization mechanism to support rapid shipment and deployment of applications. A containerized application relies on the kernel of the host OS. Sharing the kernel and avoiding the need for a guest OS typically yields better performance and a reduced footprint. One of the most popular implementations of containerization is Docker [7]. Docker supports process-level isolation, with two concepts from the kernel that are used: namespaces and cgroups. Namespaces are used to isolate certain resources: a network stack, its own processes tree and other resources. For example, a container cannot see or interact with the processes running in another container. The cgroups are used for specifying what computer resources a container should be able to access, such as certain amount of RAM, CPU cores, file system access and similar [26].

Docker containers can share namespaces. For example, sharing process ID (PID) namespace allows one container to access other containers’ processes. Sharing an inter-process communication (IPC) namespace means that containers use the same network stacks and memory region. The feature of shareable namespaces in Docker is key for ChaosOrca: it enables it to conduct experiments without modifying existing containers.

2.3 Observability

Observability [41] is the ability to collect values about some parts of the internal state of a system, based on its external behaviour. Increasing a system’s observability is extremely valuable for debugging and reasoning about failures. Observability leverages three types of strategies: well designed event logs, sufficient monitoring metrics and tracing. An event log can be in different forms, plain text, structured or binary. Monitoring metrics usually combine information are different levels. For example, CPU and memory usage are useful for finding out problems like infinite loops or memory leaks. Also, the ratio of successful HTTP requests is helpful to describe whether the system is running under acceptable status. Tracing is a technique that combines distributed but related events together, such as a user request which spans multiple micro-services. Tracing is hard as it requires each application in the request path to propagate tracing information [44].

2.4 Chaos Engineering

Chaos engineering is a technique that evaluates resilience in production environment. When an application is deployed into production, it faces uncertainties like unstable network, unavailable third-party services and unbalanced traffic. A well designed application should be able to bear these unanticipated scenarios without any serious impact for the users. Tools for chaos engineering actively inject different kinds of failures into a production system and try to learn how the system behaves under perturbation. If the system behaves as expected, such experiments improve the confidence the developers have about the resilience of the system. Otherwise the experiment points out weaknesses in the ability of the system to handle failures [27].

There are five principles in chaos engineering that describe how developers should apply such a technique into their production system:

1. The very first one is to build a series of hypotheses about the system’s steady state. The steady state is defined based on monitorable metrics. An hypothesis describes how the steady state is expected to change under a certain event which is triggered during chaos engineering experiments.
2. The second principle is that a failure injected by a chaos engineering experiment simulates a real-world event. For example, a simulated failure may be that the space of a disk is full, or a function returns with an exception.
3. As the goal of chaos engineering is to build confidence in a production system, chaos engineering experiments should be done after deployment, instead of in a testing environment.
4. Since it is impossible to predict every failure in a software system, chaos engineering experiments should be conducted continuously: automation is the fourth principle in chaos engineering.
5. The last but same important principle is “blast radius control”. The term “blast radius” describes the seriousness of impacts caused by chaos engineering experiments. Running chaos engineering experiments are done so as to minimize negative influences on end-user experience.

3 Design of ChaosOrca

ChaosOrca is addressing is an important problem: the lack of tools for developers to assess the resilience of containerized applications. ChaosOrca contributes to addressing this problem in the qualified domain of evaluating resilience of containerized application with respect to system call errors. ChaosOrca is designed to provide developers with observability and chaos engineering capabilities for an application container with respect to system call errors, without modification of the container. In other words, the core design principle is to gather sufficient insights about resilience, while causing minimal impact on the system under study. Thus, the requirements for designing ChaosOrca are as follows:

Requirement 1**.**

Black-box containers: ChaosOrca should consider containers as black-boxes. In a production system, the images specifying the containers are often optimized for performance and security, meaning that it is impractical, if even possible, to weave-in extra instrumentation in the image directly. All action by ChaosOrca should thus be implemented in a way that does not require instrumenting the containers. In other terms, ChaosOrca should consider containers as pure, untouchable black-box entities. A positive implication of this requirement is that it reduces the effort of developers to the minimum for using ChaosOrca: an application container is deployed normally and ChaosOrca is attached to it when developers need to run chaos experiments.

Requirement 2**.**

Monitoring: As mentioned in Section 2, observability is a foundation to analyze how the system behaviour differs during a chaos experiment. Given Requirement 1, the internals of a container cannot be directly observed. ChaosOrca must monitor the surroundings of the containers and gather information to assess resilience weaknesses. This includes monitoring the impact that the containers have on the physical resources (RAM, CPU), on the I/O (network activity), and how they use system calls on the host kernel.

Requirement 3**.**

System call perturbation: ChaosOrca is designed for analyzing resilience with respect to system calls. Thus, ChaosOrca should be able to perturb a specific type of system call. The form of perturbations also include 1) force a system call to directly return with an error code, 2) add a delay before a system call is actually completed, and 3) a combination of delay and error injection.

Requirement 4**.**

Minimizing blast radius: In a production system, a server may host hundreds of containers, it is thus critical for ChaosOrca to be able to target specific containers without influencing the others, so as to limit the blast radius of chaos experiments. Within this specific container, ChaosOrca should be able to only perturb one single process as well.

3.1 Overview of ChaosOrca

To fulfill the requirements listed above, ChaosOrca is divided into three components: 1) monitor, 2) perturbator and 3) orchestrator. The interactions among them are described in Figure 2. The monitor component is responsible for capturing the system behaviour at runtime. The perturbator injects different kinds of failures with respect to system calls. The orchestrator controls the monitor and perturbator components to conduct chaos experiments and generates reports.

3.2 The ChaosOrca Monitor Component

Observability is essential to control and analyze chaos experiments. The monitor component implements requirement 2. In order to connect low-level system call perturbations with application-level behaviour, the monitor component provides different levels of observability, respectively 1) container level, 2) system call level and 3) application level.

At the container level, the monitor records CPU time, memory usage, network traffic as follows. The monitoring component relies on shareable namespaces offered by Docker, it runs as a process which shares a specific namespace with the target container under study. This type of design fulfills requirement 1: the monitor provides strong observability with no modification or intrusion into the target container image.

For the system call level, the monitor component records the invoked system calls information including type, arguments and result. Regarding application level monitoring, the monitor cares about whether the application fulfills a user request, where the definition of fulfillment varies from one application to another. For example, for a web server, the monitor observes the HTTP response codes and the latency in handling HTTP requests.

3.3 The ChaosOrca Perturbator Component

The perturbator component is designed to inject system call failures at runtime. It addresses both requirement 3 and 1. A system call perturbation is defined as a tuple $(s,e,d)$ where $s$ represents the type of a system call, $e$ means the error code of this system call to be injected, and $d$ denotes the delay before this system call is actually invoked. In ChaosOrca, the perturbator works as follows, it intercepts an existing system call that is invoked by the application, then ask for commands from the orchestrator (cf. Section 3.4) about whether injecting a specific failure in this system call. Per requirement 3, the perturbator supports failures that could be either an error code, or a certain delay, or a combination of the both.

Let us consider the example of the TTorrent file download procedure shown in Figure 1. When TTorrent needs to save a piece of downloaded file on disk, the logic in method PieceStorageImpl/savePiece is executed. This leads the JVM (i) to invoke a pwrite system call, and (ii) to write bytes of data into a file through this system call. Our perturbator can target such types of system calls, it can intercept this pwrite system call and injects a failure in TTorrent.

For the perturbator component, it is also important to minimize the blast radius, according to requirement 4. Thanks to the shareable namespaces feature of Docker introduced in Section 2.2, ChaosOrca is able to accurately inject system call invocation failures into a specific container only to minimize the blast radius. Regarding over-consumption issues an injected failure may cause, the blast radius can be still controlled by limiting the resources (CPU, RAM, etc.) a container is allowed to use. [1]

3.4 The ChaosOrca Orchestrator Component

The orchestrator component acts as an interface between ChaosOrca and the fault injection strategies defined by the developers. It provides a command line interface for developers who want to apply ChaosOrca. Based on the configuration, the orchestrator controls the monitor and perturbator components to explore a set of system call perturbations in the target application. Such a procedure is described in Algorithm 1. Line $1$ and $2$ are executed to gather the application’s normal behaviour. The triple for-loops construct all possible system call failures and gather the perturbed behaviour for a comparison.

There are three phases for a fault injection experiment: 1) before perturbation, 2) during perturbation and 3) after perturbation as follows. First of all, before perturbation, the orchestrator attaches the monitor, the perturbator and itself to the target application. It feeds the application with a piece of workload in order to gather information about the normal behaviour of this application. After that, the orchestrator activates the perturbator to inject a specific system call failure defined in Section 3.3, this is the perturbation phase. Finally, the perturbator is turned off and the after-perturbation phase starts, the orchestrator keeps pulling monitoring information and computes the metric differences. With those three phases, ChaosOrca enables itself to detect whether a system call failure has an instantaneous influence (in phase during-perturbation) or it has a long lasting influence even after the perturbation (in phase during-after).

Regarding the behaviour analysis in the during- and after- phases in our experiments, ChaosOrca considers a mix of generic and domain specific dimensions. The generic dimension is 1) whether the application continues to run without a crash and 2) how the metrics under monitoring vary compared to nominal behaviour. ChaosOrca also considers domain specific dimensions. For example, for web applications, ChaosOrca compares whether the HTTP response code is $200$. A procedure of comparing an application’s perturbed behaviour with normal behaviour is called “Diff” in Algorithm 1.

3.5 The ChaosOrca Visualization Component

By default all metrics monitored by ChaosOrca are saved into a time-series database. The developers can query this data as a table and investigate them after fault injection experiments. ChaosOrca also has a separate component that visualizes all the monitored metrics as line charts.

For the convenience of resilience analysis, the following features are supported by the visualization component: 1) auto-refresh the charts in a configurable time rate, 2) highlight the perturbations, 3) aggregate the metrics with a developer defined configuration. For example, one chart can display the rate of all different system calls with a legend. In this way it helps developers to intuitively compare the rate among different system calls at a certain time point.

Figure 3 shows an example of the visualization component. The line chart describes how the CPU usage changes during fault injection experiments. A blue overlay indicates a period of system call perturbations, which also divides the figure into three phases: before, during, and after the perturbations. The green line explains that the CPU usage drops during the perturbations and catches up afterward. The ChaosOrca visualization component draws one line chart for each of the monitored metrics similarly.

3.6 Implementation

The monitor component uses cAdvisor111https://github.com/google/cadvisor to gather container-specific metrics like CPU usage, memory usage and network traffic metrics. For monitoring HTTP requests, the monitor combines PyShark222https://kiminewt.github.io/pyshark/, a python library for TShark to get HTTP response code and latency. For collecting the system calls the monitor uses the open-source tool Bpftrace333https://github.com/iovisor/bpftrace to observe the number and type of system calls. All the above monitoring information is pushed to a Prometheus time-series database444https://prometheus.io/.

Then ChaosOrca uses Grafana555https://grafana.com/ to visualize the data for developers. The perturbator component is mainly implemented with Strace 666https://strace.io/, which is a diagnostic, debugging and instructional userspace utility for Linux. The orchestrator component itself is written in Python. It provides a command line interface to developers to define system call perturbation arguments like type, error code and delay. ChaosOrca is publicly-available at https://github.com/KTH/royal-chaos/tree/master/chaosorca.

4 Evaluation

This section discusses the evaluation of ChaosOrca, which focuses on the following 4 research questions:

RQ 1**.**

What system call perturbations are of interest for containerized applications?

RQ 2**.**

What kind of resilience problems are identified with fault injection in containerized applications?

RQ 3**.**

How is multi-layer observability helpful to analyze the results of chaos experiments?

RQ 4**.**

What is the overhead of conducting chaos experiments using ChaosOrca?

4.1 Subject Programs

We experiment ChaosOrca on three different applications, in order to evaluate its ability at perturbing system calls and at detecting resilience weaknesses in containerized applications. We selected the applications according to the following criteria:

1. medium-sized, well implemented applications with source code available, 2) contain both client side and server side code 3) can be deployed as Docker containers and 4) it is possible to simulate production-like workloads in the laboratory.

The first application is TTorrent777https://github.com/mpetazzoni/ttorrent, a file downloading client that uses the BitTorrent protocol. The second application is Nginx888http://hg.nginx.org/nginx/file/tip, a popular and mature reverse proxy. The last one is Bookinfo999https://github.com/istio/istio/tree/master/samples/bookinfo, a micro-service based reference application which runs several containers written in different languages. The key descriptive metrics of those subjects are shown in Table 1 (e.g the number of containers, programming languages, lines of code). Note that ChaosOrca focuses on a subset of containers in a larger software system and leave the other containers running normally. To that extent, it would likely work on perturbing one single isolated container in a very large system.

4.2 Experimental Protocol

Protocol for RQ1: In order to select the most interesting system calls for chaos experiments, we consider two dimensions: 1) the prevalence of a system call during execution 2) the functionality of the system call under study. The first dimension is investigated by monitoring applications. The second one needs more domain knowledge. For example, an application which frequently writes files invokes $write$ a lot. So $write$ takes a large ratio in all happened system calls. Before invoking $write$, another system call open needs to be invoked. It opens the target file and returns the file descriptor. open system call may not be invoked lots of times, but it is still interesting since $write$, $read$ system calls rely on it.

Every application runs for $1$ minute under a simulated workload, and we perform $3$ rounds of executions in total for a statistical reason. In this experiment, ChaosOrca only monitors the applications and does not inject any failure. The workloads are as follows: TTorrent is executed to download an Ubuntu installation file from the internet, the file is 1.86GB in size. For Nginx, three types of valid GET requests including /, /abc and /abc?cache=bust are sent to the container. For Bookinfo, a collection of valid GET and POST requests are used to trigger the following features: the login logic; accessing the main page; accessing a product page and logging out. For all three applications, ChaosOrca calculates the rate of different types of system calls.

Protocol for RQ2: For each system call identified as relevant in RQ1, three different perturbation strategies are evaluated according to Algorithm 1: 1) injecting a system call error code without any delay 2) injecting a certain delay after which the system call succeeds and 3) injecting both an error code and a delay. Considering there are more than $100$ different error codes that can be returned by system calls [11], we carefully select $6$ error codes such that they are all related to resource and permission issues [8], namely EACCES, EPERM, ENOENT, EIO, EINTR and ENOSYS. We do so because these system call errors are representative of real-life failures. For example, error code EACCES means “permission denied”, indicating that the file permissions do not allow the attempted operation. Then we run chaos experiments with these $6$ error codes and $2$ different delay values ($1$s and $5$s). The workload for these applications are the same with RQ1. Both generic and domain specific dimensions described in Section 3.4 are used for behaviour comparison.

Protocol for RQ3: The experiment protocol is the same as for RQ2. However, for this research question, we focus on monitoring and data visualization. Consequently, RQ3 is discussed qualitatively. For example, by removing part of observability provided by ChaosOrca, such as closing HTTP response code monitoring, we analyze whether it is still possible to correctly detect resilience problems.

Protocol for RQ4: A study of the overhead of ChaosOrca is important to assess its applicability. To answer RQ4, we respectively evaluate the overhead caused by the monitor component and the perturbator component. For the network overhead, Nginx is used as a target application. Then, the tool Siege 101010Siege (https://github.com/JoeDog/siege) is an http load tester which generates requests at a fast rate is used to simulate a number of concurrent users sending requests for $3$ minutes. The number of users is respectively $5,10,25,200,1000,2000$ during the experiment.

For system call monitoring and perturbation overhead, in order to stimulate a high amount of system calls, dd command is used as a target application. dd is a Linux command which mainly coverts and copies files. It invokes a huge amount of $read$ and $write$ system calls. The command to test dd is dd if=/dev/zero of=/dev/null bs=1 count=500000k, which copies zero to the void for 500 million times.

4.3 RQ1: What system call perturbations are of interest for containerized applications?

The statistics of the invoked system calls are listed in Table 2. Each line describes one of the system call invoked during the execution. Each column shows the number and percentage of system call invocations in one containerized application. A blank cell means that a system call is never invoked by the corresponding application under the given workload. Some system calls are rarely used by the application, which are noted as $<0.01$. For example, the first row of the table tells that TTorrent invokes the accept system call $1780$ times every second, which takes about $9.92\%$ of the total. While Nginx never invokes the $accept$ system call, Bookinfo invokes it $293.45$ times per second which takes $3.90\%$ of the total.

From Table 2 we could see that the types of most used system calls differ in applications. For example, TTorrent invokes read and futex most frequently. This actually makes sense because TTorrent takes advantages of multi-threading to download files. System calls like read, futex are keys to support TTorrent’s functionality. Thus these most used system calls are of interest for evaluating TTorrent. Bookinfo invokes the most types of system calls during an experiment, because it is micro-service based and has a more complex business logic. For HTTP based applications including Nginx and Bookinfo, the system calls used in common are clone, getpid, getsockname, newfstat, newstat, open, recvfrom, setsockopt and write.

With the help of ChaosOrca, developers are able to focus more on the most critical system calls per application. This makes the following resilience evaluation more targeted and effective.

Over our three case studies, we observe a total of $44$ different types of system calls exercised by the containerized applications under study. By analyzing the prevalence and the functionality of each of them, $9$ types of system calls are identified as relevant for fault injection experiments in containerized applications, namely open, write, writev, read, readv, sendfile, sendfile64, poll and select.

4.4 RQ2: What kind of resilience problems are identified with fault injection in containerized applications?

According to the experiment protocol, we have conducted $9x(6+2+6x2)=180$ rounds of experiments. After finishing all the experiments, we qualitatively analyzed the influence caused by each perturbation. In the following, we discuss the most used system calls. The interested reader can access to the comprehensive experimental data at https://github.com/K TH/royal-chaos/tree/master/chaosorca.

4.4.1 Evaluations on TTorrent

Table 3 shows a sample of experiment results in TTorrent. As it is a file downloading client, system calls like open, write, read are the most used ones. In this table, each row describes one perturbation and TTorrent’s corresponding behaviour. Each cell is the result of the comparison of the application’s behaviour under and after perturbation.

From Table 3 it can be seen that only adding a delay to open, write, read system calls have a critical influence on network usage. Especially row $21$ and $22$ show that the $read$ system call perturbation causes the biggest dip in network IO. Recalling Table 2, this is consistent because the read system call is the most common one, it occurs almost $20$ percent of the total number of calls.

Another insight from the table is that for the open system call perturbations, network IO dips initially during the perturbation and then catches up after. For write and read system call perturbations, the network traffic stops completely and the number of system calls drops to nearly zero. This indicates that TTorrent is able to bear system call delays but is more sensitive to system call errors.

4.4.2 Evaluations on Nginx

Table 4 describes the most interesting findings from our experiments on Nginx. For Nginx, the perturbations that have an obvious effect are those on open, write, writev and sendfile. As an example, the first row records the Nginx behaviour under an open system call perturbation: adding a $1$ second delay with no system call error code. However, the HTTP response code becomes [math] and the latency increases to more than $50$ ms. The number of total system calls decreases during the perturbation compared to the normal execution. The network and CPU usage stay low while some spikes occur. The usage of memory increases during the experiment. This proves that Nginx is able to handle open system call delays and the usage of calculation resources stays acceptable.

Row $1$ to row $20$ shows that an open system call perturbation has different effects depending on the injected error code. An $EACCES$ error results in HTTP 403 Forbidden, an $ENOENT$ error leads to 404 Not Found instead. An $EPERM$, $EINTR$, or $ENOSYS$ results in HTTP 500 Internal Server Error. This could be considered as a good practice because developers designed a more detailed reaction strategies with respect to different system call errors.

Another interesting insight to look at is the read system call perturbations (row $26$ to row $30$). When such a perturbation is injected, the application still functions normally as the HTTP response code is still $200$. The trend of system calls and the network IO stay normal as well. According to these experiments, developers gain more confidence that Nginx is able to survive from read system call errors.

4.4.3 Evaluations on Bookinfo

Table 5 gives a representative sample of our experimental results. For the Bookinfo application, the system call perturbations that have a significant effect are open, write, read, poll and select. Compared to Nginx, when ChaosOrca injects delays into the open system call, the code of responses are divided into $200$ and $302$. Code $302$ means a redirection to another url. Considering the Bookinfo containers as a black box, we assume that this is because the injected delay causes a timeout exception in Bookinfo’s components. In order to lead users to a retry logic, Bookinfo redirects the request to an initial page. For the select system call perturbations, Bookinfo directly crashes if an error code is injected. This indicates that the application is sensitive to select system call errors, and its self-protection strategies are not able to pull Bookinfo back to normal when such a perturbation happens.

By performing fault injection in three different containerized applications, we show that two kinds of resilience problems are identified by ChaosOrca: 1) the application is not resilient to some system call failures: if certain system calls fail, the whole application crashes and 2) the application is sensitive to system call errors: if certain system calls fail, the application consumes significantly more system resources.

4.5 RQ3: How is multi-layer observability helpful to analyze the results of chaos experiments?

In order to improve the observability of target applications, the monitoring component in ChaosOrca records different levels of metrics: 1) system call level, 2) container level and 3) application level. In order to easily compare the changes under perturbation with a normal execution, all of these metrics are visualized by ChaosOrca in a Grafana dashboard.

For example, Figure 4 is a metrics visualization of an experiment on Nginx. In this series of experiments, ChaosOrca injects an EACCES error every time when the open system call is invoked. In the figure, a blue line stands for a specific system call perturbation. From the figure it can be seen that ChaosOrca monitors Nginx before any perturbations for $5$ minutes. Then it keeps intercepting open system calls and injecting an EACCES error in them. After the perturbator is turned off, ChaosOrca monitors Nginx for another $5$ minutes. Thanks to the monitor component, the $5$ metrics collected by ChaosOrca namely network IO, HTTP latency, CPU and memory usage, the amount of invoked system calls are observable by developers. The visualization also makes it easier for developers to analyze the three experiment phases defined in Section 3.4: the trend of metric changes before, during and after perturbations.

On the other hand, if the observability of target application is not sufficient, it brings more obstacles to distinguish abnormal behaviour. For example, during an OPEN system call perturbation in Figure 4, the rate of HTTP requests stays the same. However, most of requests are responded with a $403$ “Forbidden” code according to sub-figure (f). If ChaosOrca does not bring observability to web application’s HTTP response code, developers may think that everything is fine since Nginx still handles a proper amount of requests. But actually most of the responses are incorrect. Such a perturbation does have critical influence on end-users.

For a containerized applications, there are different locations in the stack that enable engineers to improve observability. ChaosOrca shows the feasibility of multi-layer monitoring at the system call level, the container level and the application level.

4.6 RQ4: What is the overhead of conducting chaos experiments using ChaosOrca?

4.6.1 The Overhead of the Monitoring Component

Figure 5 shows the overhead of monitor component when Nginx is the target application. The first sub-figure records the amount of HTTP requests per second during the experiment. The second sub-figure records the rate of system calls. The last two sub-figures are about the CPU and memory usage of the monitor components. Regarding dd command, the CPU and memory usage metrics are the same as Nginx’s ones.

For network monitoring, with the increase of HTTP request rate, sub-figure (a) in Figure 5 shows that the HTTP request monitoring even fails to monitor every request when the workload becomes too high. Sub-figure (c) describes that the CPU usage of network monitoring increases to almost two times of the application’s itself. The memory usage cost by network monitor ranges from 50MB to 200MB which is shown in sub-figure (d).

For system call monitoring, the overhead is quite low and stable. Even under a system call heavy tool like dd, the CPU usage of the system call monitor module (chaosorca.sysm) is almost zero (sub-figure c). The memory usage stays at 125MB and never increases (sub-figure d).

Overall for monitor component, the overhead of network monitoring becomes significant when the rate of user requests are higher than $50/s$ in the test environment. The overhead caused by system call monitoring always stays low. As the memory usage, it is stable and there is no memory leak during the experiment. The overhead of ChaosOrca can be considered as acceptable for a normal amount of workload.

4.6.2 The Overhead of Perturbator Component

When the perturbator injects system call failures, the key metrics are described in Figure 6 and Figure 7. For Nginx, it shows that the number of HTTP requests under a normal workload remains the same. For the system call heavy application dd, the perturbations have an significant impact on the number of system calls. Overall for the perturbator component, the usage of system resources remains small during experiments.

The overhead caused by ChaosOrca depends on system call usage and also on the workload. For the monitor component, the network monitor overhead increases with the workload; and the system call monitor overhead is low. As for the perturbator component, our experiment shows that the overhead is acceptable. Overall, our experiments suggest that the runtime overhead of ChaosOrca is acceptable for analyzing resilience of containerized applications.

5 Discussion

5.1 Threats to Validity

One threat to the validity of ChaosOrca is the perturbation search space has not been completely explored. We manually selected the $6$ most general and relevant system call error codes for our experiments. However, there are more than $100$ error codes that can be returned by different system calls. The interactions between a specific system call and its possible error codes have not been exhaustively analyzed. Future work will analyze the perturbation space with respect to all system call error codes.

Another threat is the generalization of the experimental results. ChaosOrca has been evaluated on three different types of applications, including a file downloading client written in Java, a reverse proxy server written in C and a polyglot micro-service based application which contains 6 containerized services. Though these applications are selected according to the criteria mentioned in Section 4.1, conducting more experiments on other kinds of applications may change some answers to the research questions. Future work is required to explore other application domains and evaluate the generalization of the experimental results in Section 4.

Lastly, in order to exclude accidental effects, we make $3$ rounds of experiments for each application (see Section 4.2). This is not enough for statistical analysis. It is an interesting area of future work to study statistically causal impact analysis on the fault injection results, which is little researched area.

5.2 Extensibility of ChaosOrca

As introduced in Section 3.6, ChaosOrca is implemented using specific features like namespace sharing in Docker. This limits ChaosOrca to be applied by only Docker applications. However, the design of ChaosOrca can be reused for other container systems like Singularity111111https://sylabs.io/. Furthermore, Singularity allows to interface with all of the resources, devices and network inside the container from the outside of the container [16]. Theoretically, by tuning the commands used by ChaosOrca’s orchestrator component and applying specific monitoring tools for Singularity, ChaosOrca can be used for the Singularity container system. Another well-designed container system is Shifter121212https://github.com/NERSC/shifter, which is particularly used for high performance computing. However, Shifter does not use network or process namespaces, nor cgroups. The implementation of ChaosOrca can not be easily adapted for Shifter. Future work is needed to evaluate the extensibility and overhead of ChaosOrca in other container systems.

5.3 Resilience Observation With or Without an Application Container

ChaosOrca aims at allowing observability and fault injection in containers, in order to assess resilience. All of the research questions and the corresponding experiments are designed based on a the presence of a container environment. It is assumed that the resilience observed inside and outside the container is similar. Future work is needed to systematically research the difference between the resilience observed inside and outside an application container, i.e. whether containerization itself is responsible for changing certain aspects of resilience.

6 Related Work

6.1 Fault Injection

Fault injection was originally studied at the hardware level, for example using radiations to force bit flips [32] or by interacting with pins to generate processor errors [40]. Later on, software-based approach were developed to emulate hardware errors [36] e.g. by injecting memory or network errors [33] or by perturbing the OS kernel [37, 38]. A number of tools now facilitates the injection of such kinds of errors, such as Gremlin [10], ChaosCat [6] or cpu-troll [20]. The network layer is another fault injection vector, which has been studied in depth. Quite a few open-source tools [34, 24, 22, 12, 21] and closed-source tools [10, 6] combine Iptables with the Traffic Control network emulation tool to inject different kinds of network failures, including latency and dropping a percentage of traffic.

More related to our work, a number of approaches propose to inject faults at the system call level, for example by introducing bit flip during the execution of system calls [25], by corrupting those system calls [43], or by overriding parts of the glibc library [39], which wraps system calls into a C API. Besides, approaches have been developed to better control and protect the access to those system calls [46]. Containerization, cgroups and namespaces, which we presented in Section 2, provide process-level isolation and allows to finely control which resources a given container can access. Yet the existing works above need to modify operating systems or core libraries, thus they are not applicable for a production environment. On the contrary, since ChaosOrca requires no modification at all and has a low overhead, it is applicable in production.

Chaos Engineering is an emerging discipline, building on those fault injection tools, but applying them directly on the production systems. Netflix for example created a tool called Chaos Monkey [34] to randomly disable production servers and to observe how their platform would handle such failures. Chaos Monkey has inspired a number of similar approaches for different production environments, such as Kubernetes [2, 18, 5], Docker [24, 19], AWS [4] or private clouds [13] or even the JVM [47] by forcing exceptions.

ChaosOrca is a novel Chaos Engineering approach, which proposes to perturb systems calls in a controlled manner. The blast radius of experiments conducted with ChaosOrca is controlled by cgroups and namespaces, which makes it possible to use it in production: only targeted containers will be affected.

6.2 Observability

Monitoring is a well researched area. For example, PyMon is a lightweight monitoring solution to collect monitoring metrics from the open-source tool monit [31]; Dargos is a decentralized architecture for monitoring cloud environments with a low overhead [42]

Moving to container-specific approaches, cAdvisor [9] by Google enables the collection and export of container metrics. It provides metrics about CPU, memory, disk and network usage, which can be for example used to decide when to scale in and out [29, 28]. Node_exporter [15] is a tool able to export many different metrics about hardware and operating system, examples include, CPU, disk, network and memory statistics.

In Chaos Engineering, metrics are necessary to evaluate the steady state of the system (before pertubation) and the current state of the system during an experiment. Netflix and Google have created a tool called Kayenta [17], which does automated canary analysis by comparing the key metrics of the new deployment version to the old one. If the canary degrades metrics too much it can automatically abort the canary [3].

ChaosOrca does not advocate for a specific monitoring approach provided that it can push its data into a Prometheus timeseries database. If not, an adapter needs to be implemented. Out of the box, ChaosOrca integrates with cAdvisor to resources (CPU, RAM, etc), Bpftrace to monitor the invocation of system calls and PyShark to monitor HTTP requests.

7 Conclusion

In this paper, we have presented ChaosOrca, a novel Chaos Engineering approach to actively inject system call failures into a containerized application in order to evaluate its resilience in face of such perturbations. Under the hood, ChaosOrca leverages Linux cgroups and namespaces to precisely control the blast radius of chaos experiments, ChaosOrca also integrates with different and complementary monitoring solutions to increase the observability of the system. ChaosOrca introduces perturbations by introducing errors or delays in the execution of the system calls, as specified in a resilience experiment.

Our evaluation on three different real-world applications shows that system call level perturbation is promising to detect weaknesses in self-protection and resilience mechanisms embedded in those applications. Our evaluation also shows that ChaosOrca improves the observability of a containerized application without modifying any source code or configuration in the target container. In the future, we plan to make ChaosOrca automatic in the context of a container orchestration framework such as Docker Swarm or Kubernetes.

Acknowledgements

This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.