Database Engines: Evolution of Greenness

Andriy V. Miranskyy; Zainab Al-zanbouri; David Godwin; Ayse; Basar Bener

arXiv:1701.02344·cs.SE·November 21, 2017

Database Engines: Evolution of Greenness

Andriy V. Miranskyy, Zainab Al-zanbouri, David Godwin, Ayse, Basar Bener

PDF

TL;DR

This study analyzes how the energy consumption and performance of MySQL database engines evolve over time, highlighting the correlation between software metrics and energy efficiency to guide greener database development.

Contribution

It provides an empirical analysis of energy consumption across multiple database releases and identifies software metrics linked to greenness and performance.

Findings

01

Energy consumption increases with database evolution.

02

Lines of Code correlates with energy use and execution time.

03

Practitioners can select greener database versions based on metrics.

Abstract

Context: Information Technology consumes up to 10\% of the world's electricity generation, contributing to CO2 emissions and high energy costs. Data centers, particularly databases, use up to 23% of this energy. Therefore, building an energy-efficient (green) database engine could reduce energy consumption and CO2 emissions. Goal: To understand the factors driving databases' energy consumption and execution time throughout their evolution. Method: We conducted an empirical case study of energy consumption by two MySQL database engines, InnoDB and MyISAM, across 40 releases. We examined the relationships of four software metrics to energy consumption and execution time to determine which metrics reflect the greenness and performance of a database. Results: Our analysis shows that database engines' energy consumption and execution time increase as databases evolve. Moreover, the…

Figures30

Click any figure to enlarge with its caption.

Tables10

Table 1. Table 1 : A list of the four major MySQL releases under study with their corresponding ten minor versions used in the experiments.

Major release	v.5.0	v.5.1	v.5.5	v.5.6
	v.5.0.15	v.5.1.30	v.5.5.10	v.5.6.10
	v.5.0.16	v.5.1.38	v.5.5.15	v.5.6.11
	v.5.0.20	v.5.1.40	v.5.5.20	v.5.6.12
	v.5.0.27	v.5.1.48	v.5.5.24	v.5.6.13
Minor releases	v.5.0.37	v.5.1.50	v.5.5.27	v.5.6.14
	v.5.0.67	v.5.1.59	v.5.5.30	v.5.6.15
	v.5.0.77	v.5.1.60	v.5.5.32	v.5.6.16
	v.5.0.83	v.5.1.65	v.5.5.35	v.5.6.17
	v.5.0.89	v.5.1.67	v.5.5.36	v.5.6.20
	v.5.0.96	v.5.1.72	v.5.5.39	v.5.6.21

Table 2. Table 2 : A list of the experiments with the corresponding memory buffer size; numbers in brackets represent the amount of raw data used in each experiment.

Memory buffer size	MyISAM	InnoDB
256MB	Experiment 1 (1GB)	Experiment 5 (1GB)
	Experiment 2 (3GB)	Experiment 6 (3GB)
1024MB (1GB)	Experiment 3 (1GB)	Experiment 7 (1GB)
	Experiment 4 (3GB)	Experiment 8 (3GB)

Table 3. Table 3 : Baseline energy and power measurements.

Baseline setup	Mean Energy Consumption (Wh)	Mean Power Consumption (W)
	$\pm$ Coefficient of Variation	$\pm$ Coefficient of Variation
OS alone	$10.7 \pm 0.7 %$	$63.9 \pm 0.7 %$
OS with MySQL v.5.6.20	$10.7 \pm 0.7 %$	$64.0 \pm 0.7 %$
OS with MySQL v.5.6.20	$10.7 \pm 0.5 %$	$64.2 \pm 0.5 %$
and Sysstat

Table 4. Table 4: Minimum energy consumption data per major release with relative percentages of difference between each pair of adjacent releases. Bold text highlights the greenest major release for a given experiment; italic text – the brownest (least green) one. The content of the first column is constructed based on the following template: ‘Experiment #: DB Engine Name, Memory Buffer Size (MB), Raw Data Size (GB)’.

	Minimum energy (Eq. 2) consumed				Energy consumption: relative
	by a minor release				difference (Eq. 3) between
	for a given major release (Wh)				previous and current release
Major release	v.5.0	v.5.1	v.5.5	v.5.6	v.5.1	v.5.5	v.5.6
1: MyISAM, 256, 1	23.13	26.40	27.80	33.00	14%	5%	19%
2: MyISAM, 256, 3	114.87	155.53	164.10	182.77	35%	6%	11%
3: MyISAM, 1024, 1	22.83	26.40	27.97	33.17	16%	6%	19%
4: MyISAM, 1024, 3	156.17	208.40	212.10	252.03	33%	2%	19%
5: InnoDB, 256, 1	23.00	26.30	20.93	31.67	14%	-20%	51%
6: InnoDB, 256, 3	92.30	310.43	246.40	448.07	236%	-21%	82%
7: InnoDB, 1024, 1	8.63	9.03	8.57	11.30	5%	-5%	32%
8: InnoDB, 1024, 3	101.00	318.07	307.10	473.77	215%	-3%	54%

Table 5. Table 5: Minimum execution time data per major release with the relative percentage of difference between each pair of adjacent releases. Bold text highlights the fastest major release for a given experiment; italic text – the slowest one. The content of the first column is constructed based on the following template: ‘Experiment #: DB Engine Name, Memory Buffer Size (MB), Raw Data Size (GB)’.

	Minimum execution time (Eq. 4) taken				Execution time: relative
	by a minor release				difference (Eq. 3) between
	for a given major release (Hr)				previous and current release
Major release	v.5.0	v.5.1	v.5.5	v.5.6	v.5.1	v.5.5	v.5.6
1: MyISAM, 256, 1	0.20	0.23	0.24	0.29	15%	6%	20%
2: MyISAM, 256, 3	1.21	1.75	1.83	2.09	45%	5%	14%
3: MyISAM, 1024, 1	0.20	0.23	0.24	0.29	16%	6%	19%
4: MyISAM, 1024, 3	1.83	2.55	2.58	3.14	39%	1%	22%
5: InnoDB, 256, 1	0.20	0.22	0.18	0.27	11%	-20%	52%
6: InnoDB, 256, 3	0.87	3.83	2.96	5.92	342%	-23%	100%
7: InnoDB, 1024, 1	0.07480	0.08	0.07479	0.10	5%	-5%	29%
8: InnoDB, 1024, 3	1.00	3.96	3.86	6.30	297%	-2%	63%

Table 6. Table 6 : Coefficient of variation in energy consumption (Eq. 1 ) of minor releases for a given major release.

	MyISAM				InnoDB
Experiment #	1	2	3	4	5	6	7	8
v.5.0	0.04	0.07	0.04	0.08	0.08	0.27	0.49	1.03
v.5.1	0.01	0.02	0.003	0.01	0.01	0.22	0.04	0.91
v.5.5	0.01	0.01	0.01	0.01	0.04	0.24	0.04	0.97
v.5.6	0.02	0.02	0.01	0.01	0.01	0.30	0.02	0.77

Table 7. Table 7 : Pearson correlation coefficient values between energy consumption and the software metrics. The values in brackets for Experiments #5–8 show correlation coefficient values for dataset without outliers.

	MyISAM				InnoDB
Experiment #	1	2	3	4	5	6	7	8
LOC	0.93	0.73	0.92	0.69	0.74	0.62	-0.06	0.12
					(0.76)	(0.60)	(0.89)	(0.15)
LOCC	0.00	0.02	0.01	0.02	0.03	0.15	-0.01	0.43
					(0.03)	(-0.22)	(-0.03)	(0.30)
MCC	0.53	0.21	0.49	0.31	0.82	0.49	0.26	0.03
					(0.80)	(0.52)	(0.76)	(0.35)
TCC	0.51	0.19	0.47	0.29	0.80	0.48	0.28	0.03
					(0.78)	(0.50)	(0.75)	(0.35)

Table 8. Table 8 : Pearson correlation coefficient values between changes in energy consumption and LOCC. The values in brackets for Experiments #5–8 show correlation coefficient values for dataset without outliers.

	MyISAM				InnoDB
Experiment #	1	2	3	4	5	6	7	8
LOCC	0.55	0.75	0.74	0.66	0.27	0.09	0.06	-0.01
					(0.29)	(-0.27)	(0.44)	(0.31)

Table 9. Table 9 : Pearson correlation coefficient values between time spent and the software metrics. The values in brackets for Experiments #5–8 show correlation coefficient values for dataset without outliers.

	MyISAM				InnoDB
Experiment #	1	2	3	4	5	6	7	8
LOC	0.93	0.71	0.92	0.67	0.75	0.63	-0.07	0.12
					(0.77)	(0.62)	(0.86)	(0.16)
LOCC	-0.01	0.03	0.01	0.02	0.03	0.15	-0.02	0.43
					(0.03)	(-0.23)	(-0.03)	(0.30)
MCC	0.53	0.20	0.49	0.30	0.84	0.48	0.26	0.03
					(0.82)	(0.52)	(0.75)	(0.34)
TCC	0.51	0.18	0.47	0.29	0.82	0.47	0.27	0.02
					(0.81)	(0.51)	(0.74)	(0.34)

Table 10. Table 10 : Pearson correlation coefficient values between changes in time spent and LOCC. The values in brackets for Experiments #5–8 show correlation coefficient values for dataset without outliers.

	MyISAM				InnoDB
Experiment #	1	2	3	4	5	6	7	8
LOCC	0.53	0.76	0.73	0.66	0.30	0.10	0.07	-0.01
					(0.31)	(-0.28)	(0.46)	(0.31)

Equations12

σ_{v} = σ / μ,

σ_{v} = σ / μ,

min (\frac{r _{1, 1} + r _{1, 2} + r _{1, 3}}{3}, \frac{r _{2, 1} + r _{2, 2} + r _{2, 3}}{3}, \dots, \frac{r _{10, 1} + r _{10, 2} + r _{10, 3}}{3}),

min (\frac{r _{1, 1} + r _{1, 2} + r _{1, 3}}{3}, \frac{r _{2, 1} + r _{2, 2} + r _{2, 3}}{3}, \dots, \frac{r _{10, 1} + r _{10, 2} + r _{10, 3}}{3}),

\frac{new_release_value - old_release_value}{old_release_value} \times 100.

\frac{new_release_value - old_release_value}{old_release_value} \times 100.

min (\frac{t _{1, 1} + t _{1, 2} + t _{1, 3}}{3}, \frac{t _{2, 1} + t _{2, 2} + t _{2, 3}}{3}, \dots, \frac{t _{10, 1} + t _{10, 2} + t _{10, 3}}{3}),

min (\frac{t _{1, 1} + t _{1, 2} + t _{1, 3}}{3}, \frac{t _{2, 1} + t _{2, 2} + t _{2, 3}}{3}, \dots, \frac{t _{10, 1} + t _{10, 2} + t _{10, 3}}{3}),

avg_energy_consumed_by_minor_release_N - avg_energy_consumed_by_minor_release_N-1

avg_energy_consumed_by_minor_release_N - avg_energy_consumed_by_minor_release_N-1

avg_time_spent_by_minor_release_N - avg_time_spent_by_minor_release_N-1 .

avg_time_spent_by_minor_release_N - avg_time_spent_by_minor_release_N-1 .

Peer Reviews

No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.

Videos

No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.

Full text

Database Engines: Evolution of Greenness

Andriy V. Miranskyy

Department of Computer Science

Ryerson University

Toronto, ON, M5B 2K3, Canada

[email protected]

Zainab Al-zanbouri

Department of Computer Science

Ryerson University

Toronto, ON, M5B 2K3, Canada

David Godwin

IBM Toronto Software Lab,

8200 Warden Ave.,

Markham, ON, L6G 1C7, Canada

Ayse Basar Bener

Department of Mechanical and Industrial Engineering

Ryerson University

Toronto, ON, M5B 2K3, Canada

Abstract

Context: Information Technology consumes up to 10% of the world’s electricity generation, contributing to CO2 emissions and high energy costs. Data centers, particularly databases, use up to 23% of this energy. Therefore, building an energy-efficient (green) database engine could reduce energy consumption and CO2 emissions.

Goal: To understand the factors driving databases’ energy consumption and execution time throughout their evolution.

Method: We conducted an empirical case study of energy consumption by two MySQL database engines, InnoDB and MyISAM, across 40 releases. We examined the relationships of four software metrics to energy consumption and execution time to determine which metrics reflect the greenness and performance of a database.

Results: Our analysis shows that database engines’ energy consumption and execution time increase as databases evolve. Moreover, the Lines of Code metric is correlated moderately to strongly with energy consumption and execution time in 88% of cases.

Conclusions: Our findings provide insights to both practitioners and researchers. Database administrators may use them to select a fast, green release of the MySQL database engine. MySQL database-engine developers may use the software metric to assess products’ greenness and performance. Researchers may use our findings to further develop new hypotheses or build models to predict greenness and performance of databases.

1 Introduction

Information Technology (IT) energy requirements are significant. Literature shows that 1500 Terawatt hours 111Terawatt hour $=10^{12}$ watt hours. (TWh) per year (or 10% of the worldwide energy generation) are consumed by IT [1]. With the adoption of the Cloud paradigm, data centers consume up to 23% (350 TWh [1]) of the overall amount of energy that is used by IT. In the US alone, it is expected that the data center energy consumption will grow from 91 TWh in 2013 to 139 TWh in 2020, with the energy bill rising from $9.0 billion USD to$ 13.7 billion USD [2].

Not only energy consumption is expensive, but it also affects the environment. The average monthly concentration of CO2 in atmosphere reached 400 parts per million in 2015 – the highest in the past 800,000 years [3]. It is estimated that world energy-related CO2 emissions will increase from 32.3 billion metric tons in 2012 to 35.6 billion metric tons in 2020 and to 43.2 billion metric tons in 2040 [4]. Data centers in US emitted 97 million metric tons of CO2 in 2013, it is expected that the emissions will grow to 147 million metric tons by 2020 [2]. Research in environmental sustainability focuses on economic, environmental, social, and human drivers that impact the environment and human beings. In this respect, IT, in general, and software, in particular, may contribute to environmental sustainability through the development of environmentally friendly systems. This may happen in different ways, e.g., using energy efficiently via decreasing the used resources, resulting, in turn, in the reduced CO2 emissions. Moreover, IT processes can be made more sustainable by decreasing the energy consumption and the negative emissions of companies and individuals. Green IT is focused on studying these issues. Formally, Green IT is a study and practice of efficient use of computing resources to decrease the negative impact on the environment [5], it can be applied to various high-tech domains, such as data centers, mobile computing and embedded systems [6]. However, in this paper, we will focus on the data center domain.

Obviously, the energy in the data centers is consumed by hardware. However, since hardware is driven by software, software is responsible for consuming energy as well [6]. The software side has received less attention than the hardware side and a few solutions (such as energy test suites [7, 8, 9]) have been put forth to comprehend energy efficiencies.

Software manufacturers are now paying more attention to making the enterprise software more energy efficient (greener). This is driven not only by external requests from their clients (that need to reduce maintenance cost of their data centers), but also by requests from within the software manufacturing organization [10]. The majority of software manufacturers (e.g., Amazon, Google, IBM, Microsoft, Oracle, and SAP) offer their products via Cloud using Platform as a Service (PaaS) or Software as a Service (SaaS) delivery models. Internal technology operations teams, managing the PaaS and SaaS offerings, are also adding the voice to the chorus of the clients, asking to reduce energy expenses for internal data centers.

Software development teams may benefit from making their own software greener too. Adoption of continuous delivery and integration practices, such as nightly builds and automatic unit and regression tests, based on literature [11, 12] and authors’ experience, requires hundreds of build and test servers per product. This computing power is required to build and run tens of thousands of test suites on multiple platforms to support multiple releases of a given product (maintained by the development team). Thus, by making their own software greener, developers can decrease their expenses, by reducing energy bill for the internal server farms.

Databases are considered to be the backbone in the software world; hence, they are responsible for a significant proportion of the overall software energy consumption. Therefore, we focus on understanding how to make databases more efficient (greener) by trying to identify the main factors that affect database energy consumption and execution time. In addition, a lot of database engines (especially relational ones) are mature products that have been developed for decades (e.g., first Oracle database was release in 1978, IBM DB2 – in 1983, and MySQL – in 1995). Multiple releases of these products are available and maintained by their development teams. Thus, it is important to understand how database engine greenness evolves from release to release.

Research objective: We aim to identify how the energy consumption and execution time of a database engine may change when the database evolves from one release to another, in addition to understanding how these changes are related to some database-associated properties such as raw data size, database engine type (MyISAM/InnoDB), and database memory size. Furthermore, this research investigates the software metrics that can have direct effects on the sustainability development in a database represented by its effect on energy consumption and execution time. By using such metrics, software developers can assess greenness of their products by measuring characteristics of their code, thus eliminating the need to execute a reference workload against a software product and to measure energy consumption directly (in turn, saving time and resources). In order to achieve our research objective, we focus on answering two research questions:

RQ1

How does the energy consumption and execution time of a database engine change as the product matures (from one release to another)?

RQ2

Which software metrics reflect energy consumption and execution time?

Answering RQ1 will help us identify the factors leading to green and, hopefully, fast databases. Answering RQ2 will help us build models that can predict software greenness and performance based on software metrics that can be easily extracted from source code (such as code size or code churn metrics). This information should be of interest to practitioners, since software vendors such as Apple, IBM and Microsoft are seeking more sustainable products with lower levels of energy consumption and execution time [13, 14, 10]. It should also be of interest to researchers, since the information can be used to build universal models of software’s energy consumption and performance.

In order to answer these research questions, in this paper, we study energy consumption and execution time across 40 different releases (shipped between 2005222The earliest available in MySQL archive. and 2014) of two database engines (MyISAM and InnoDB) of the MySQL database. MySQL is the most commonly used and most popular open source database in the world [15]. We chose MySQL as the software under study, because the MySQL database is a mature product (having been developed since 1995) with a large (approximately 1 million lines of code) codebase being actively developed. This gives us enough data to study the product’s evolution.

To answer RQ1, we study the relation between energy consumption (or execution time) for all the MySQL versions333In this paper, we use the terms ‘version’ and ‘release’ interchangeably. under study. To answer RQ2, we examine the relation between software metrics from one side and energy consumption (or execution time) from the other side.

This type of work required building a framework to automate all the necessary processes such as database installation, upgrade from version to version, executing the workload, reading and collecting the measurements from the special measurement meter and recording the measurements for all the MySQL releases used in addition to creating a database for all the experimental data results. It also required building a system to extract software metrics from the code base of MySQL so that the relation between the metrics on one side and energy consumption (or execution time) on the other side could be established. All these requirements are addressed in this work.

This is the first study examining the relation between different MySQL database releases and their energy consumption as well as execution times. This research differs from the previous research [13, 14] that has examined the link between different versions of Firefox web browser and their performance. In addition, to the best of our knowledge, this work is the first study to establish a link between MySQL databases’ energy consumption and their execution time from one side and the software metrics (namely Lines of Code (LOC), Lines of Code Changed (LOCC), and Traditional/Modified Cyclomatic Complexity (TCC/MCC)), from another side.

The rest of the paper is structured as follows. Related work is discussed in Section 2. Methodology of our experiments is explained in Section 3. Section 4 provides the results of our experiments. Threats to validity are given in Section 5. Finally, Section 6 concludes the paper.

2 Related Work

This section is structured as follows. Energy-related research, focused on computer parts and operating system (OS)-level software, is provided in Section 2.1. Energy-related benchmarks and frameworks are discussed in Section 2.2. Relevant database-related research is given in Section 2.3. Finally, research related to mining software repositories and energy consumption data is shown in Section 2.4.

2.1 Energy consumption: Hardware and OS-level Software

A number of researchers have focused on energy consumption in IT. Delaluz et al. [16] conducted a comprehensive study of software and hardware systems to determine the benefit of the DRAM mode control abilities for energy savings. They addressed an essential issue in energy saving for mobile and computing environments by specifically concentrating on the memory system, which consumes around 90% of the complete energy consumed by the system when ignoring input/output processes [16].

Tiwari et al. [17] presented the power usage of a single CPU. They defined an assessment-based instruction-level power analysis method, which provides an accurate and practical way of measuring the power cost of software and describes an assessment-based instruction-level power analysis method that makes it possible to effectively analyze software power consumption. Mittal et al. [18] presented an energy simulation tool that allows developers to estimate the energy use for their mobile apps on their development workstation itself.

There are several studies about the power consumption of devices. Bircher et al. [19] produced power models for the complete system depending on processor performance events. Greenwalt et al. [20] measured and modeled the power consumption of hard drives. The hard disk state model provides both the quantitative data and insight necessary to design an efficient power management system. Stemm et al. [21] studied two types of optimization (namely, transport-level and application-level) of network interfaces to decrease their energy consumption.

Li et al. [22] performed a quantitative analysis of the costs and benefits of spinning down a disk drive as a power management technique. The main idea behind the power consumption measurement movement is to be followed by suggestions or actions taken in order to find solutions to any undesirable outcomes. Selby et al. [23] applied methods to analyze the relationship between global variable usage and the efforts required by software maintenance and examined the effects of optimizations upon power usage. Fei et al. [24] employed source code change techniques to decrease the energy overheads accompanying application/OS connections and modified the source code changes and compiler optimizations in order to reduce power usage. Feng et al. [25] introduced a framework for studying the power-performance efficiency of the NAS parallel benchmarks on a 32-node Beowulf cluster.

2.2 Benchmarks and Frameworks

Some researchers have concentrated on the idea of benchmarking and examining power measurement. Asmel et al. [26] described a tool that approximates the energy consumption of software in order to help concerned consumers make knowledgeable decisions about the software they use. Gurumurthi et al. [27] introduced a complete system power simulator that represents the CPU, the hierarchy of memory and a low-power disk subsystem and calculates the power performance of both side applications and the OS.

Researchers have also developed frameworks for measuring and testing energy consumption. For example, Noureddine et al. [7] built runtime energy monitoring framework, enabling easy reporting on the energy consumption of system processes. Wilke et al. [8] created a generic framework for software energy profiling and testing.

2.3 Database-related research

Researchers have studied changes to the design of a database engine, but not to its energy consumption. For example, Shang et al. [28] investigated changes to the amount of communicated information passed to system administrators over multiple versions of the PostgreSQL database engine and the Hadoop data processing framework.

Researchers designed prototypes of energy-aware database management systems. Chen et al. [29] designed ReinDB database engine that, in the presence of renewable and non-renewable energy sources, distributes database workload to minimize usage of the non-renewable energy source. Liu et al. [30] created optimizer for execution plans of queries sent to the database engine; the optimizer minimizes energy consumption for a given query. In addition to these prototypes, Transaction Processing Performance Council created guidelines for measuring energy consumption of database workloads [9].

2.4 Mining Software Repository and Energy Consumption

The closest to our work are [13, 14, 31]. Gupta et al. [31] study focused on combining Mining Software Repository (MSR) [32] techniques with power performance and presented the first study from a software engineering perspective on energy awareness problems. The authors of [31] introduced a method for gathering and analyzing power data on mobile devices running Windows Phone 7. Their methodology describes and quantifies power consumption, detects differences in power consumption and predicts power consumption. The work by [31] is complementary to ours, because it focuses on examining the power consumption in different modules (a module is a part of a program) within the same software (Windows Phone 7) and finding which module consumes the most power. Moreover, it focused on finding the typical energy shape patterns of certain modules. We, on the other hand, are focusing on multiple versions of the same product (MySQL). Additionally, we concentrate on understanding the relation between energy consumption (or execution time) and the product development of MySQL.

Hindle [13, 14] demonstrated combining the MSR research and energy consumption by studying multiple versions of the Firefox web browser regarding characteristic energy consumption patterns of multiple modules of the web browser. He also examined the relation between the LOC and LOCC software metrics and energy consumption. These works are complementary to ours, because we focus on a different product (database instead of web browser) and study the effect of multiple software metrics (LOCC, MCC and TCC, in addition to LOC) on energy consumption and execution time.

The works cited above demonstrate the significance and importance of the study of power consumed by software in various areas of IT.

3 Methodology and Experiments

In constructing the experiment we followed guidelines of Wohlin et al. [33] in general and Hindle [13, 14] in particular, with minor variations. Design of the experiments capturing energy used, time spent, and system’s statistics is given in Section 3.1; extraction of software metrics – in Section 3.2; and analysis of the data – in Section 3.3.

3.1 Experimental Design

Product selection, as well as selection of a set of versions under study, is described in Section 3.1.1; testbed setup is given in Section 3.1.2; our test case is discussed in Section 3.1.3; product configuration is shown in Section 3.1.4; automation of test is depicted in Section 3.1.5; details of instrumentation and measurements are provided in Section 3.1.6; and effect of baseline energy consumption on the test case measurement – in Section 3.1.7.

3.1.1 Software Under Study

As discussed in Section 1, our software under study are 40 different releases (shipped between 2005 and 2014) of two database engines (MyISAM and InnoDB) of the MySQL database. Table 1 shows the list of the MySQL major releases used in our experiments along with a list of the minor MySQL releases under study. Shipping timeline of the releases is given in Figure 1.

We chose MySQL as the software under study, because the MySQL database is a mature and popular product [15] (having been developed since 1995) with a large (approximately 1 million lines of code) codebase being actively developed. These facts make MySQL a good candidate to study product evolution.

3.1.2 Testbed setup

The computer used in our experiments has two Intel Pentium 4 HT 630 3GHz CPUs, 3GB of RAM, 320GB of storage on a magnetic hard drive Western Digital WD3200AAKS. In order to eliminate effect of changing environmental conditions, this machine was rack-mounted in a data centre of the Department of Computer Science, Ryerson University. The data centre is thermostated at $29^{\circ}$ C (with 40% humidity) by industrial-grade air conditioners.

The operating system installed on the machine was Ubuntu Linux OS v.14.04 with v.3.13.0-32-generic x86_64 kernel, Server edition. We chose the Linux platform because it is a de facto standard server platform, and it is better designed for capturing computer-related statistics. Moreover, the server edition (unlike desktop edition) has smaller number of programs preinstalled [34]. For example, our installation did not have graphical user interface. This leads to a smaller number of programs that may run in the background and affect the results. The computer was dedicated to our workloads – no other tasks444Excluding limited number of crucial daemons ran by the OS; e.g., logrotate daemon performs archival and rotation of OS logs on a daily basis. were executed on this machine concurrently. We discuss effect of the daemons in Section 3.1.7. No monitor was attached to the machine; all communications with the machine happened remotely via ssh.

3.1.3 Test scenario

As a reference database workload, we used Transaction Processing Performance Council Benchmark H (TPC-H) version 2.17.0 [35]. It is considered as the standard benchmark for analytic workloads in the database community [35]. This benchmark has a set of 22 business-oriented ad-hoc queries. The data and queries simulate business practices and requirements. This benchmark mimics the decision support systems that use large volume of data, execute complicated queries (with relatively low volume of transactions), and answer business-related questions. We provide distribution of the number of operators in the queries in Figure 2. In-depth technical analysis of the queries is given in [36].

3.1.4 Product configuration

In our eight experiments, we loaded 1GB and 3GB of raw data into the database. Given the size of our hardware platform, 1GB dataset is used to mimic in-memory processing workload (since all the data can be stored in memory and no I/O operations have to be performed once the data are loaded into memory); the 3GB workload mimics workload operating on a large dataset (that cannot fit into memory and require intense I/O operations). The raw data is generated by DBGEN tool from the TPC-H package [35].

Different buffer pool sizes: Tuning the MySQL default installation is very important to improve its performance, and the key buffer cache is an essential element to be changed in this tuning process; a key buffer is used to cache the data from the hard drive into the memory [37, 38]. Theoretically, the more memory that is allocated to the cache, the faster the data processing. We set the buffer cache value to either 256MB or 1024MB.

Table 2 shows the eight different experimental setups with the corresponding memory buffer size; numbers in brackets represent the raw data size used in each experiment.

Other configuration parameters: in general, there exists infinite number of combinations of parameters. We followed best practices, as suggested by the official documentation (e.g., see [39]).

3.1.5 Test Automation

We designed a framework that automates experimentation process following, conceptually, the guidelines of Hindle [13, 14]. In particular, our framework automates the installation and measurement process: it installs the specific MySQL version (obtained from [15]), upgrades and configures (as discussed in Section 3.1.4) the database, and executes the 22 reference TPC-H queries 555We used TPC-H utility called QGEN to generate these queries. one by one. The framework measures the execution time, system state statistics (such as CPU and I/O load), and energy consumption. We provide pseudo code of the framework in Appendix A666Note that each experiment is repeated three times, increasing accuracy and precision, and reducing measurement error.. Details of the measurement process are given below.

3.1.6 Instrumentation and Measurements

The device we used to measure energy consumption is called “Watts up? PRO” [40]. The device allows direct reading of the measurements to a computer. Energy measuring device can measure both power and energy. We chose to measure energy (with resolution of 0.1 Wh), as per recommendations of the support personnel of the manufacturer of the device (obtained by e-mail). Based on the recommendation and the manual [41], the accuracy of the power (watts) measurement is $\pm 1.5\%$ (partially attributed to the shortest sampling interval of one second [42]). However, the cumulative energy measurement (cumulative watt-hours) is performed by the device continuously, by sampling the wattage measure 1000 times per second and then integrating the data to obtain cumulative energy usage, leading to higher accuracy. The device was connected to the computer running the workload via USB port and was controlled by automation scripts described in Section 3.1.5. Cumulative energy measurement readings were taken before and after execution of a given workload.

For measuring the system statistics (such as CPU and I/O load) we used Sysstat package [43]. The system measurements were taken asynchronously at one second interval during execution of the workload. Once a given workload execution was complete, the results were aggregated using summary statistics.

3.1.7 Baseline Measurement

As mentioned in Section 3.1.2, no other workloads were executed concurrently to ours. However, even though we tried to minimize the number of daemons ran by the OS, some of them have to be functional to ensure robustness of the OS. In addition, idle database engine and Sysstat package gathering system statistics may consume various amounts of energy.

To quantify the impact of such processes on energy consumption of the system, we measure energy consumption of the computer over 24 hour interval at 10 minutes increments (144 observations in total). The measurements were repeated for three different setups:

OS alone, 2. 2.

OS with MySQL v.5.6.20 installed and running (no connections to the database were made), 3. 3.

OS with MySQL v.5.6.20 and Sysstat collecting system attributes at 1 second interval.

The energy values were then converted to (average) power values to “standardize” the data. The conversion was done by dividing the amount of energy consumed by the time spent.

Results of the measurements are given in Table 3. To assess variability of baseline energy and power consumption, we compute coefficient of variation as:

[TABLE]

where $\mu$ and $\sigma$ are population mean and population standard deviation of the 144 observations for a given setup. The closer $\sigma_{v}$ to zero – the lower the variation. As we can see, OS alone consumes almost constant amount of energy through the day: $\sigma_{v}=0.7\%$ . MySQL consumes small amount of energy in idle state: extra 0.1W on top of the power consumed by OS alone. Systat consumes extra 0.2 W (or $0.3\%\leftarrow 0.2/64.2$ ) of power. However, the consumption throughout the day remains almost constant: $\sigma_{v}=0.5\%$ . This lack of variability is very desirable in our case, as the baseline will not affect our analysis (as discussed in Section 3.3).

3.2 Software Metrics Extraction

We gathered the source code for each of the MySQL versions under study by downloading them from the original MySQL website [15]. Then we created scripts to calculate the software code metrics for each MySQL version. In particular, we used CLOC tool [44] to extract size metric “total number of physical lines of code (without comments) in a given release”, denoted (LOC), and churn metric “total number of lines of code changed in the the current release in comparison with the previous release”, denoted (LOCC). We also computed two complexity metrics—“traditional cyclomatic complexity” (TCC) and “modified cyclomatic complexity” (MCC)—using the PMCCABE tool [45].

Before computing the source code metrics, we eliminated a number of source code files. First, we eliminated source code files of test cases, since the code from them is not included in the production binaries of the database engine. Second, we removed source code not written in C and C++, as they are excluded from the production binaries.

3.3 Analysis of the data

We analyzed the results (for a given product configuration) for the aggregate of all 22 SQL statements (queries) as one unit of work, simulating a single analytic workload, as discussed in Sections 3.1.3 and 3.1.5. Energy, time, system’s statistics, and software metrics data were stored in SQLite database [46]. R [47, 48] scripts (which obtained data from the SQLite database) were used to perform the analysis and produce tables and figures (discussed in Section 4).

We calculated the Pearson correlation coefficient777The Pearson correlation coefficient is a method to measure the linear relation (dependence) between any two variables $X$ and $Y$ . We chose linear regression to analyze our data; therefore, we chose the Pearson correlation over the Spearman correlation, because Pearson is designed to match the sign and magnitude of a linear regression slope [49]. between all the variables used in our experiments (such as energy consumed and time spent, energy consumed and lines of code changed, and time spent and modified code complexity).

We used the correlation data to answer the research questions, as explained in subsequent sections. We adopt the mapping between a value of correlation coefficient and the strength of correlation as per [49]. The mapping between the strength of correlation and a value of correlation coefficient is as follows:

a) Perfect: 1,

b) Strong: 0.7-0.9,

c) Moderate: 0.4-0.6,

d) Weak: 0.1-0.3, and

e) Zero: 0.

Note that Pearson correlation coefficient, denoted by $r$ , is invariant to linear transformations; i.e., given two vectors $x$ and $y$ , and a constant $c$ , $r(x+c,y)=r(x,y)$ . Based on the analysis in Section 3.1.7, our baseline energy consumption is almost constant. Therefore, our correlation analysis — involving energy consumption as one of the variables — will yield almost identical values of $r$ with baseline energy included or excluded from the energy consumption variable. In our case, we kept baseline energy included, as it represent the total amount of energy spent to complete an experiment.

4 Results of Experiments

This section is structured as follows. The effect of system state on energy consumption is given in Section 4.1. The relation between energy consumed and time spent is described in Section 4.2. System statistics analysis of the results of experiments, needed to answer research questions RQ1 and RQ2, are provided in Sections 4.3 and 4.4, respectively. Finally, the answers to RQ1 and RQ2 are provided and discussed in Section 4.5.

4.1 Energy consumption vs. System Statistics

In this subsection we will analyze the relation between system statistics and energy consumption for all the experiments, independent of the release number. This will help us understand general constraints of the system under study. The discussion on per-release evolution is given in Section 4.3-onward.

Figure 3 and 4 show888A discriminating reader may notice counterintuitive results: providing more memory to a database engine does not yield expected performance improvement. We discuss this phenomenon in Section 4.5.1. the relation between energy consumption, CPU utilization, and I/O load. We compute CPU utilization and I/O load as follows.

CPU utilization is computed as a sum of CPU utilization that occurred while executing at user level, user level with nice priority, and at the system level [43]. The CPU utilization data in this figure is computed by averaging out per-second data gathered by Sysstat. The higher the number – the more utilized the CPU is. Note that our computer has two CPUs. Therefore, 50% utilization means full load of one CPU.

Transfers per second shows the number of I/O requests (of indeterminate size) to the hard drive. The higher the number the more data active our reads and writes to/from the hard drive. The transfers per second data in this figure is computed by summing up per second data collected by Sysstat.

As shown in Figures 3, 4(a), and 4(c), the more I/O operations one has to do, the more CPU has to idle waiting for the I/O operations to complete. This leads to increased time spent and energy consumption, as the CPU idles, waiting for the data to be read from (and written to) the hard drive.

As seen from Figure 3, 4(a), and 4(c), the setup with 1GB of raw data being loaded into the database requires minimal amount of I/O operations, because the whole database is cached into memory. Once the data are cached – all operations happen in memory, no access to the hard drive is required. In this case the workload often becomes CPU-bound, as the architecture of both engines (for the releases under study) cannot effectively parallelise processing of a single query. The setup with 3GB of raw data being loaded into the database is I/O-bound: the database engine cannot load all the data into memory and has to wait for I/O operations to complete.

InnoDB engine has a number of outliers with very high energy consumption ( $>500$ Wh) for Experiments 6 and 8 (where 3GB of raw data are loaded into the database), as seen in Figures 3(b), 4(c), 4(d) and Figures 8(b), 8(d). These outliers have a very high number of I/O operations associated with them (sum of transfers per second increases from $\approx 2.0\times 10^{6}$ to $\approx 1.2\times 10^{7}$ ). This large number of I/O operations can be explained by suboptimal decisions made by query optimizer. For example, it can decide to do a full table scan (i.e., read the content of every row in a table) rather than relying on indexes of a given table. Typically, this is considered a performance defect. Such defects are quickly noticed and reported by users (“My query took 10 minutes on a previous release; and now, after engine upgrade, it takes 10 hours!”) and are, usually, fixed quickly by the database engine developers.

4.2 Execution Time vs. Energy Consumption

Figure 5(a) shows the relation between energy consumption and execution time for all eight experiments. We can see that the strong positive linear relation between these two variables: coefficient of determination, $R^{2}=0.9975$ and the slope value, represented by $b$ in Figure 5(a), is positive. The Pearson correlation coefficient between these two variables is equal to $0.9986$ , signifying an almost perfect correlation (as defined in Section 3.2).

This strong correlation comes from a trivial fact: the more time computer spends running the workload, the more energy it consumes. But what is the root cause of this observation? To get a better understanding of the situation, we plot time spent vs. average power consumed in Figure 5(b). The average power consumed is computed by dividing the amount of energy consumed by the time spent. From hereon, for the sake of brevity, we will use the term ‘power’ instead of ‘average power’. As the figure shows, the more time we spend – the less power we consume. This can be explained by differences in the amount of power consumed by CPU and HDD.

Both CPU and HDD require different amount of power while idling or under load. However, the amount of power (in absolute numbers) is vastly different: our HDD consume 5.6W while idling and 6.0W under load [50]; each of our two CPUs, on the other hand, has thermal design power999Thermal design power is not equivalent to power consumed by the CPU. It gives the amount of power that CPU dissipates while being fully active. However, this value give us an understanding of the magnitude of power consumption. of 84W [51].

The higher the CPU utilization is – the higher its power consumption. From Figure 3 we know that experiments which consumed the least amount of energy (and thus finished in the shortest period of time) had high CPU utilization. The figure also shows that these workloads had almost no I/O activity, since all the data was loaded into memory. CPU did not have to idle waiting for I/O operations to complete and was able to maintain high utilization rate. These experiments correspond to data points in the bottom-right corner of Figure 5(b), where power consumption was in the range of 110-121W.

As mentioned above, HDD consumes significantly smaller amount of energy than CPU (compare 6W with 84W). This becomes important for experiments causing high I/O utilization. Hard drive power consumption will remain at $\approx$ 6W, while CPU utilization will drop significantly (as shown in Figure 3), as CPU idles waiting for I/O operations to complete. The cases of experiments with idling CPU are represented by data points in top-left corner of Figure 5(b), where power consumption was $\approx$ 67W, which is quite close to the baseline power consumption of 64.2W. Even though power consumption in these cases is low, the energy consumption is high, since we have to integrate low power consumption value over prolonged time interval (when the CPU waits for I/O operations to complete). Similar behaviour was observed in the past with IBM DB2 database management system [52, 53].

Keep in mind that each experiment yields the same result, the only difference is in the amount of “work” needed to obtain this result. To summarize, the experiments that consumed the most amount of time and energy are the I/O-bound ones, requiring high number of I/O operations.

4.3 RQ1: Experiments

In this section, we will analyze the results of our experiments to answer RQ1: ‘How does the energy consumption and execution time of a database engine change as the product matures (from release to release)?’

4.3.1 Description of Equations, Tables, and Figures

In this subsection we describe tables (along with formulas needed to populate them) and figures that are used in subsequent Sections 4.3.2 and 4.3.3.

Figures 7 and 8 show box plots of the energy consumed in MySQL MyISAM and InnoDB versions, respectively. Each figure contains four subplots – one per experiment (listed in Table 2) for a given engine. Note that we represent the data using box plots, since each experiment was conducted three times to ensure reproducibility.

Similarly, Figures 9 and 10 contain box plots of the energy consumed in MySQL MyISAM and InnoDB versions, respectively. Because energy consumption and execution time are correlated almost perfectly, as was shown in Section 4.2, energy- and time- related pairs of figures are visually similar (Figures 7 and 9 for MyISAM and Figures 8 and 10 for InnoDB).

By eyeballing the plots, we can observe trends in the data. To formally confirm our observations, we plot linear trend lines (computed using linear regression). We map release names to natural numbers in order to compute linear regression. The first minor release v.5.0.15 is mapped to 1, the second minor release v.5.0.16 is mapped to 2, etc. The $p$ -values of the trend lines for all but one product configuration are less than $0.001$ . The remaining configuration (Experiment 8: InnoDB engine with 1GB of memory and 3GB of raw data shown in Figures 8(d) and 10(d) ) has a $p$ -value of $\approx 0.05$ .

In subplots, solid line depicts a trend line obtained using linear regression; the dashed lines show the 95% confidence interval of the trend line. The dotted lines represent a trend line (obtained using linear regression) per major release, and the dot-dashed lines show the 95% confidence interval of these trend lines per major release. Vertical long-dash lines show boundaries between major releases.

Figures contain a lot of information; in order to summarize this information, we construct the following tables. Table 4 summarizes the energy consumption for the MyISAM and InnoDB engine: Experiments #1-4 in the table were conducted on MyISAM, Experiments #5-8 – on InnoDB. In this table, we show the average energy consumption of a minor release (average of three runs) that consumes the least amount of energy within a set of minor releases belonging to a given major release. Formally, for a given major release, the energy reading displayed in the tables is computed as follows:

[TABLE]

where $r_{i,j}$ represents energy consumption for the $j$ -th run of the $i$ -th minor release belonging to the major release of interest. The average of three runs per minor release in Eq. 2 is taken to minimize the measurement error and to reduce the effect of outliers for a given minor release; then the minimum amount of energy from the ten average readings is chosen. Furthermore, the relative percentage of difference between each two adjacent releases is calculated using the following formula:

[TABLE]

To assess variability of minor releases for a given major release, we use Equation 1, where $\mu$ and $\sigma$ are population mean and population standard deviation of ten $(r_{i,1}+r_{i,2}+r_{i,3})/3$ terms. The $\sigma_{v}$ values, shown in Table 6, portray the amount of variability in relation to $\mu$ . The closer the $\sigma_{v}$ to [math] – the lower the variation.

The timing results per major release given in Table 5 are computed using a formula that is structurally similar to that of Eq. 2, as shown in Eq. 4:

[TABLE]

where $t_{i,j}$ represents workload execution time for the $j$ -th run of the $i$ -th minor release belonging to the major release of interest. The relative percentage of difference between each two adjacent releases is computed using Eq. 3. As in the case of Table 4, Experiments #1-4 in the table were conducted on MyISAM, Experiments #5-8 – on InnoDB. Given that energy consumption and execution time are correlated almost perfectly (see Section 4.2 for details), timing results are similar to energy ones.

4.3.2 MyISAM

We performed four Experiments (#1–4) on the MyISAM engine. The analysis of the per-run data, collected during these four experiments, suggests that both the energy consumption and execution time of the MyISAM engine increased as the engine matured. In essence, on average, the newer major releases are slower and less green than the older ones.

Table 4 summarizes the energy consumption for the MyISAM engine. The oldest release, v.5.0, is the greenest (of all major releases), because it consumed the least amount of energy to execute the same reference workload. On the other hand, the newest release, 5.6, is the brownest (least green), because it consumed the largest amount of energy, as shown in Table 4.

All four experiments showed a consistent increase in energy consumption between any two major consecutive releases (all the difference values in Tables 4 are positive, representing this increase). For example, Experiment 1 shows that release v.5.1 is 14.12% less efficient than v.5.0, release v.5.5 is 5.30% less efficient than release v.5.1, and release v.5.6 is 18.71% less efficient than release v.5.5 – see Table 4 for details.

Figure 7 shows that, overall, the energy consumption increased as MySQL matured. By eyeballing the plots, we can see that this trend is clearly pronounced as we move from one previous major release to the next one. This is also confirmed by linear models: they explains most of the data variability (based on $R^{2}$ values): in the Experiments #1-3 from 83% to 87% of variability is explained (see Figures 7(a), 7(c), and 7(b)), while in the case of Experiment #4 only 61% of variability is explained (see Figures 7(d)).

The increase in variability for Experiments #4 and, to a degree, for Experiment #2 (see Figures 7(d) and 7(c), respectively) can be explained by the fact that Experiments #1 and 3 process 1GB of raw data, all of which can be fitted in memory. However, in Experiments #2 and 4, we cannot fit all 3GB of raw data into memory. Database memory manager and OS file caching mechanism need to derive sophisticated probabilistic strategies to efficiently load the data to memory from the hard drive (e.g., by predicting which data pages will soon be needed and proactively loading them into memory [54]). We will discuss this observation for both database engines in Section 4.5.

As we can see from Table 6, the variation of energy consumption within three major releases v.5.1, v.5.5, and v.5.6 remains low ( $\leq 0.02$ ). The results are further confirmed by eyeballing Figure 7 and examining relatively flat trend lines for these releases. However, the variation for v.5.0 is higher (between 0.04 and 0.08). Moreover, the trend line in Figure 7(d) suggests that energy consumption increases from the earliest minor release v.5.0.15 to the latest minor release v.5.0.96 under study. We conjecture that this anomalous behaviour may be related to the fact that v.5.0 is the first major release in a series of v.5.x releases; it could have had “infantile diseases” that were outgrown in subsequent releases.

The timing results per major release are given in Table 5 and Figure 9. Since energy consumption and execution time are correlated almost perfectly, the timing results are very close to the energy consumption results. In other words, execution time increases as the engine matures, with the v.5.0 release being the fastest and v.5.6 being the slowest. We conjecture that this behaviour of the MyISAM engine can be explained by the fact that the inclusion of additional functionality to the MyISAM engine in each subsequent release requires additional computational resources, leading to an increase of time and energy consumption.

4.3.3 InnoDB

We conducted four Experiments (#5–8) on the InnoDB engine. The summary of the energy consumption (computed using Eq. 2) for these experiments is given in Table 4 along with the relative percentage of difference between each pair of adjacent releases, calculated using Eq. 3. We can see that the results for the InnoDB engine are less monotone (in comparison with the MyISAM engine). Examination of Table 4 reveals that energy consumption increased from v.5.0 to v.5.1, decreased from v.5.1 to v.5.5, and increased again from v.5.5 to v.5.6 for all four experiments. For example, for Experiment 5 energy consumption increased by 14.53% from v.5.0 to v.5.1, decreased by 5.51% from v.5.1 to v.5.5, and, finally, increased by 19.73% from v.5.5 to v.5.6. This non-linear dynamics is confirmed by low levels of variability explained by linear trend lines in Figure 8: $R^{2}$ for the lines ranges between 0.03 and 0.31.

As in the case of MyISAM, the least green major release is the most recent one (namely, v.5.6). However, we obtained different results for the greenest engine: in the case of Experiments #6 and 8, the greenest version is the oldest one (v.5.0), while, in the case of the remaining Experiments, #5 and 7, the greenest engine is the intermediate v.5.5. Experiments #5 and 7 cache all the data in the memory (using 1GB raw data size); Experiments #6 and 8 have to read the data from the hard drive (using 3GB raw data size). This implies that the functionality of v.5.5 may be better suited for handling an I/O-intensive workload.

Variability between minor releases of InnoDB is also significantly higher than in the case of MyISAM, as shown in Table 6: e.g., compare $\sigma_{v}=0.08$ for v.5.0 in Experiment 4 with $\sigma_{v}=1.03$ for v.5.0 in Experiment 8. The variability for Experiments #5 and 7 is $<0.08$ (with the exception of v.5.0 with $\sigma_{v}=0.49$ ); the variability for Experiments #6 and 8 is significantly higher with $\sigma_{v}$ ranging between $0.22$ and $1.03$ .

Moreover, examination of Figure 8 reveals that variation between minor releases can be significant (as we already discussed in Section 4.1): e.g., in Experiment #8, based on Figure 8(d), v.5.6.15 consumed $\approx 260\%$ more energy than its predecessor v.5.6.14 or its successor v.5.6.16 (average energy readings are 1748.93, 479.56, and 476.56, respectively). IT personnel maintaining a database running on InnoDB engine should be mindful of this fact: an untested upgrade to a new release may lead to significant increase in energy consumption (and performance degradation, since these two variables are strongly correlated).

Table 5 and Figure 10 summarize the execution time for Experiments #5–8. Due to correlation of time and energy consumption the timing data mirrors the energy data: v.5.6 is the slowest in all four experiments. The fastest release in Experiments #6 and 8 is v.5.0, while v.5.5 is the fastest for Experiments #5 and 7.

We will discuss the difference in behaviour between the MyISAM and InnoDB engines in Section 4.5.

4.4 RQ2: Experiments

In this section, we will show the analysis of the data focusing on RQ2: ‘Which software metrics reflect energy consumption and execution time’? In order to answer RQ2, we analyzed the relations between energy consumption and software metrics (namely, LOC, LOCC, TCC and MCC) in all eight experiments. In Section 4.4.1 we describe equations, figures, and tables that are used in analyzing the relations in Section 4.4.2.

4.4.1 Description of Equations, Tables, and Figures

We provide exploratory analysis of the data, namely, the distributions of the software metrics and correlations among the software metrics in Figures 6(a) and 6(b), respectively. We list Pearson correlation coefficient values between energy consumption (or time spent) and various software metrics in each experiment in Tables 7 and 9, respectively.

We introduce two additional response variables: change in energy consumption and change in time spent. They are defined as:

[TABLE]

and

[TABLE]

The variables represent the difference between energy consumed (or time spent) by a given release and previous release. We will use these variables to identify relations between energy consumed (or time spent) and software metrics.

Table 8 shows the Pearson correlation values between changes in energy consumed (Eq. 5) and the LOCC metric. Table 10 gives the Pearson correlation values between changes in time spent (Eq. 6) and the LOCC metric. Given the perfect correlation between energy consumed and time spent, the resulting numbers in these two tables are very close: the difference between the correlation values is less than 0.03.

4.4.2 The Relations

The distributions of the metrics in Figure 6(a) show that LOCC has higher variability in comparison with the other three metrics. By examining correlation among the software metrics (see Figure 6(b)), we found that LOC is strongly correlated with MCC and TCC. This behaviour has been observed in other software products in the past [13, 14, 55]. MCC and TCC are perfectly correlated, by construction, since their formulas are similar. LOCC is weakly correlated with the other variables.

We investigated the relation between energy consumption (time spent) and each software metric graphically, using corresponding scatter plots, and numerically, by analyzing the trend lines obtained using linear regression and correlation between variables.

By examining correlations between energy consumption (or time spent) and various software metrics in each experiment (given in Tables 7 and 9), we see that the results are identical due to an almost perfect correlation between energy and time.

These tables show that there exists a moderate to strong positive correlation between consumed energy (or time spent) and codebase size, measured in LOC, for Experiments #1-6 for both database engines: MyISAM and InnoDB. However, for Experiments #7 and 8 on InnoDB engine, the correlation between energy consumed (or time spent) ranges between none to weak. This may be explained by the high variability of the results in these two Experiments (as per Table 6 and Figure 8), “masking” the correlation.

In order to test this hypothesis, we removed the outliers (i.e., those data points whose energy consumption and performance were anomalously high, probably due to performance defects, as discussed in Section 4.1) and recalculated the correlation for Experiments #5–8 (see values in brackets in Tables 7 and 9). As we can see, removal of the outliers does yield strong correlation between consumed energy (or time spent) and LOC for Experiment #7. However, for Experiment #8 the correlation remains weak.

In the case of MyISAM, there is no correlation between LOCC and consumed energy (or time spent); while in case of InnoDB there is none to weak correlation (with the exception of Experiment 8, where correlation is moderate). The exception can be explained by high variability in the data and is, probably, a statistical artifact. The removal of the outliers does not change the picture significantly: the correlation for Experiments #5-7 remains weak to none, and for Experiment #8 the correlation drops from moderate to weak.

The correlation between energy consumed (or time spent) and code complexity metrics (MCC and TCC) ranges from none to strong, suggesting that it cannot be used as a consistent predictor of consumed energy. Removal of the outliers, increases correlation strength to the ‘weak to strong’ range, but does not affect the conclusion. Peculiarly, an examination of Figure 6(b) suggests that the relation between LOC and complexity metrics (MCC and TCC) is strong. Nevertheless, MCC and TCC are less reliable predictors than LOC.

We also computed the correlation between changes in energy consumed (Eq. 5) — or time spent (Eq. 6) — and the software metrics LOC, MCC, and TCC. We do not show Pearson correlation coefficient values for the sake of brevity, but the correlation for all experiments was none to weak.

The correlation data in Tables 8 and 10 show that the change in energy consumption (or change in time spent) between releases is correlated with the code churn, measured by LOCC; the correlation for the MyISAM engine ranges between moderate and strong, while the correlation for InnoDB engine ranges between none and weak. Removal of the outliers increase InnoDB correlation range from weak to moderate. This suggests that changes in energy consumption or time has some correlation with the amount of changes to the code; however, the size of code attribute has stronger relation.

4.5 Discussion

4.5.1 RQ1

The answer to RQ1: ‘How does the energy consumption and execution time of a database engine change as the product matures (from release to release)?’ is as follows.

As shown in Tables 4 and 5 and Figure 7, the overall energy consumption for MyISAM engine increases as the product matures, suggesting that the additional functionality added with every new release consumes additional resources. In the case of InnoDB (based on data in Tables 4 and 5 and Figure 8), the earliest major release v.5.1 is the greenest in 50% (2 out of 4) of the experiments, while the latest major release v.5.6 is the least green in 100% (4 out of 4) experiments. In the case of the two remaining experiments, the greenest title is claimed by an intermediate release v.5.5.

The results of execution time (performance) are similar to energy consumption results (due to the almost perfect correlation between energy consumed and time spent, see Section 4.2): in MyISAM case, the overall execution time increases as the product matures, i.e., newer releases are slower than the older ones. InnoDB execution time findings are identical to energy consumptions findings as well.

This is different from the results of the experiments on the Firefox web-browser [13, 14], where energy consumption decreased as the product matured. This suggests that, depending on the product and its domain, the results may vary. This difference can potentially be explained by the fact that the two products’ (Firefox and MySQL) application domain, construction methods, and coding styles are different.

Let us discuss additional difference in behaviour between the MyISAM and InnoDB engines, shown in Sections 4.3.2 and 4.3.3.

Greenness and performance: based on Tables 4 and 5 data, the results for greenest and fastest results of InnoDB are better than those of MyISAM. However, this rule is not universal. For example, a database user (whose data cannot fit into memory) may require some features present only in the latest version v.5.6 of MySQL. In this case, the user may have to select MyISAM engine instead of InnoDB, since it will be 61% greener (182.77Wh vs. 473.77Wh) and 67% faster (2.09Hr vs. 6.30Hr), see results of Experiments 2 and 8 in Tables 4 and 5 for details.

Variation within major release: based on the data from Table 6 and Figures 7 and 8, MyISAM’s results are less volatile than InnoDB ones. We conjecture that this can be explained by the fact that MyISAM was designed to handle analytic workloads, while InnoDB was originally designed for transactional ones [56]. Since the developers have not focused on satisfying requirements of analytic workloads, the performance and energy consumption results for the current analytic workload are volatile.

Peculiarly, before major release v.5.5, MYSAM was a default database engine for MySQL; starting from v.5.5, InnoDB became the default one [15]. Database administrators interested in executing an analytic workload should take into account this difference while setting up their databases. This has to be taken into consideration during migration to a new version of the engine or an upgrade to a new release, since it may lead to a spike in energy consumption and performance degradation (as discussed in Section 4.3.3).

Effect of Raw Data Size: based on Tables 4 and 5, we can see that in Experiments 1 vs. 5 and 3 vs. 7, dealing with 1GB of raw data that can fit into computer’s memory, InnoDB engine is greener and faster than MyISAM engine for all releases. In the case of Experiments 2 vs. 6 and 4 vs. 8, dealing with 3GB of raw data that cannot fit into memory, MyISAM engine is greener and faster than InnoDB engine with one exception: InnoDB’s early release v.5.0 (in Experiment 6) is greener and faster than MyISAM’s one (in Experiment 2).

This suggests that InnoDB may be better suited for in-memory processing than MyISAM. However, for larger datasets that cannot fit into memory (which is more common for analytic workloads), MyISAM may be a better choice (as mentioned above, MyISAM was designed for analytic workloads).

Effect of Memory Buffer: Tables 4 and 5 show that in the case of in-memory processing (for Experiments # 1 vs. 3 and 5 vs. 7, dealing with 1GB raw data) increase of the memory buffer from 256MB to 1024MB leads to marginal (less than 1%) improvement in greenness and performance for MyISAM engine. In the case of InnoDB the improvement is significant (approximately 60%). This can be explained by the fact that MyISAM has a rudimentary memory manager, relying mainly on OS mechanism for file caching, while InnoDB has a more sophisticated memory manager [38, 37].

In the case of processing data that cannot fit into memory (experiments 2 vs. 4 and 6 vs. 8, dealing with 3GB of raw data) increase of the memory buffer leads to significant (29% to 38% for MyISAM and 2% to 25% for InnoDB) degradation of greenness and performance for both MyISAM and InnoDB engines. This can be explained by the fact that OS mechanism for file caching outperforms database engines memory manager for all major releases.

4.5.2 RQ2

The answer to RQ2: ‘Which software metrics reflect energy consumption and execution time?’ is as follows. Consumed energy is governed mainly by the size of the code base. As shown in Section 4.4, the code size LOC metric serves as a moderate to strong predictor of energy consumption for both database engines (with the exception of InnoDB’s Experiment 8, where correlation is weak). The code churn (LOCC) and complexity (MCC and TCC) metrics results are weaker. This conclusion holds for predicting energy consumed as well as change in energy consumption.

The results for execution time (performance) are similar (due to the almost perfect correlation between energy consumed and time spent, see Section 4.2): The time is governed mainly by LOC; LOCC, MCC and TCC have a lesser effect on performance. These results suggest that the amount of consumed energy and time spent are governed by the sheer volume of code to execute rather than the amount of changes introduced or code complexity. If we treat high energy consumption as a defect [57], our results differ from the results seen for functional defects, where LOCC acts as a better predictor of defects than does LOC [58, 59]. This result is also different from the findings of Hindle [13, 14], who found that, in the case of the Firefox web browser, LOC is not correlated with power consumption.

Note that LOC cannot work as a predictor of energy consumption or time spent when a performance defect, causing major performance degradation, is present in the code base. However, such cases are rare (as we observed and discussed in Section 4.1).

5 Threats to validity

Here we present threats to validity, classified as per [60, 33, 14].

**Conclusion validity **: There are a number of threats related to reliability of measures. In order to avoid fluctuation of energy consumption due to fluctuation of ambient temperature (higher temperature may lead to higher energy consumption by computer’s hans), the computer was placed in thermostated environment of the data center.

Baseline energy consumption may fluctuate over time. In order to address this threat, we installed server version of the operating system with minimal amount of software preinstalled. We also performed extensive measurements of the baseline, making sure that it remains nearly constant over time.

Construct validity: To make comparison of different releases representative, we chose a subset of ten minor releases from all the available major MySQL releases (v.5.0, v.5.1, v.5.5, and v.5.6), providing representative “picture” for each major release, helping to gain a broad and clear idea about MySQL’s conduct.

The limitation in MySQL behaviour (response) if using a single configuration. To mitigate this threat, we chose two different database storage engines types (MyISAM and InnoDB) in order to get a more general idea about the database engine’s behaviour. Also we chose different key buffer sizes (256MB and 1024MB), which helped in examining the various situations for each database engine. Moreover we chose different raw data sizes (1GB and 3GB), which helped to gain a clear idea about the different responses of the database engine when the raw data was less than the available memory (all data can be cashed in the case of 1GB) or exceeded the available memory (in the case of 3GB).

To ensure consistency in the workload used, we ran the same standard TPC-H workload to be sure that we had a consistent execution in all experiments.

To maintain accuracy and precision of results and reduce measurement error, we ensured that all the required measurements (such as energy consumption and execution time) were calculated automatically. Moreover, each experiment was repeated three times, increasing accuracy and precision, and reducing measurement error.

Internal validity: In order to reduce the threat to validity related to human errors, we automated the process of data gathering and analysis, reducing the risk of human error. We created Python [61] scripts for gathering code metrics from the source code of the database engine. We also created a Python script that profiled workload execution. The results of the experiments as well as the source code metrics were stored in a SQLite database. We then automatically generated the figures and correlation tables using R [47, 48] scripts (which accessed data from the SQLite [46] database).

External validity: As described by Wieringa and Daneva [62], software engineering studies suffer from the variability of the real world, and the generalization problem cannot be solved completely. As they indicate, to build a theory we need to generalize to a theoretical population and have adequate knowledge of the architectural similarity relation that defines the theoretical population. We studied two database engines. The generalization to other database engines or other software products is, obviously, not possible. However, the software under study represents a critical case [60] of a relational database management system. In this study we do not aim build a theory, rather we would like to have a deeper understating of energy footprint in a case study. However, our experimental framework can be applied to other projects with well-designed and controlled experiments.

6 Conclusions and Future Work

In this research our aim was to explore and have a deeper understanding of the impact of energy efficiency on database applications. We performed a case study, measuring the energy consumption and execution time of two MySQL database engines across 40 releases on a reference analytic workload TPC-H. To achieve this goal, we developed a framework to measure the energy consumption and execution time of a database workload by extracting software metrics from each release of the software product.

Answering RQ1—‘How does the energy consumption and execution time of a database engine change as the product matures (from one release to another)?’—our study shows that the MySQL MyISAM engine becomes less green and less efficient as the product matures in all four experiments (leading to increased energy consumption and higher CO2 emissions). In the case of MySQL InnoDB engine, the earliest major release is the greenest and fastest in 50% (2 out of 4) of our experiments, while the latest major release is the least green and efficient in 100% (4 out of 4) of our experiments.

This is different from the results of the experiments on the Firefox web-browser [13, 14], where energy consumption decreased as the product matured. This difference suggests that, depending on the product and its domain, energy consumption and execution time may “evolve” differently.

Answering RQ2—‘Which software metrics reflect energy consumption and execution time?’—we show that consumed energy and performance are governed mainly by the size of the code base. The code size LOC metric serves as a moderate to strong predictor of energy consumption and performance for both database engines (with the exception of one InnoDB experiment). The smaller the code base, the greener and more efficient the database engine is. The code churn (LOCC) and complexity (MCC and TCC) metrics results have a lesser effect on energy consumption and performance.

This implies that the amount of consumed energy and time spent are governed by the volume of the code to execute rather than the amount of changes introduced to the code base or the code complexity. If we treat high energy consumption as a defect (as per [57]), our results differ from the results seen for functional defects, where LOCC acts as a better predictor of defects than does LOC [58, 59]. This result also differs from the finding of Hindle [13, 14], who found that, in the case of the Firefox web browser, LOC is not correlated with power consumption.

LOC cannot work as a predictor of energy consumption or time spent when a performance defect, causing major performance degradation, is present in the code base. However, such cases are rare, as they are quickly exposed by the users (as observed and discussed in Section 4.1).

To summarize, our answers to the research questions suggest that, depending on the product and its domain, the results may vary. For example, the difference between Hindle’s [13, 14] and our findings may be explained by the differences in the application domain, construction methods, and coding styles of the products under study (Firefox and MySQL).

Our findings may give insights to both practitioners and researchers. Database administrators may use our findings to select a green and fast release of the MySQL database engine. Developers of MySQL database engines may assess the greenness and performance of their product with the help of software metrics. The findings may also be of interest to researchers, as they lay a foundation for models predicting the greenness and performance of databases, which, in turn, would aid in developing green software.

Going forward, we plan to expand our work by studying other relational, NoSQL, and NewSQL database engines as well as study other enterprise products, such as middleware servers. We also would like to collect requirements data to better understand the release content and its impact on energy consumption.

Appendix A Test Automation: Algorithm

In this appendix we provide pseudo code for automatic execution of test cases and gathering energy, time, and systems statistics for various releases and configurations of the database engine. The main procedure is given in Algorithm 1. The main procedure, in turn, calls procedure to setup test database (Algorithm 2) and function to execute test case, also known as workload, (Algorithm 3). Note that we do not explicitly define procedures for installing, uninstalling, starting, stopping, and restarting a database engine, as these procedures are platform-specific.

Note that we recommend to rejuvenate [63] energy meter on a regular basis via reboot of its software (see Alogrithm 1, line 20), which can be done programmatically. We have noticed that after a few weeks of continuous log collections, the meter may start to behave erratically; rejuvenation fixes this issue.

Appendix B Supplementary material for Results of Experiments

This appendix contains Figures used in Section 4. Figures 7 and 9 show energy consumed and time spent by experiments executed against MySQL MyISAM engine. Figures 8 and 10 – against MySQL InnoDB engine.

Bibliography63

The reference list from the paper itself. Each links out to its DOI / PubMed record.

1[1] Mills MP. The cloud begins with coal. Technical Report , Digital Power Group 2013. URL http://www.tech-pundit.com/wp-content/uploads/2013/07/Cloud_Begins_With_Coal.pdf .
2[2] Whitney J, Delforge P. Data center efficiency assessment. Technical Report , Natural Resources Defense Council, New York City, New York 2014. URL http://www.nrdc.org/energy/files/data-center-efficiency-assessment-IP.pdf .
3[3] Jackson RB, Canadell JG, Le Quéré C, Andrew RM, Korsbakken JI, Peters GP, Nakicenovic N. Reaching peak emissions. Nature Climate Change 2015; URL http://dx.doi.org/10.1038/nclimate 2892 . · doi ↗
4[4] Conti J, Holtberg P, Diefenderfer J, La Rose A, Turnure JT, Westfall L, Adams G, Aloulou F, Aniti L, Boedecker E, et al. . International energy outlook 2016. Technical Report DOE/EIA-0484(2016) , U.S. Energy Information Administration May 2016. URL http://www.eia.gov/forecasts/ieo/pdf/0484(2016).pdf .
5[5] Murugesan S. Harnessing green IT: Principles and practices. IT professional 2008; 10 (1):24–33.
6[6] Kern E, Dick M, Johann T, Naumann S. Green software and green it: An end users perspective. Information Technologies in Environmental Engineering . Springer, 2011; 199–211.
7[7] Noureddine A, Bourdon A, Rouvoy R, Seinturier L. A Preliminary Study of the Impact of Software Engineering on Green IT. First International Workshop on Green and Sustainable Software (GREENS) , 2012; 21–27.
8[8] Wilke C, Götz S, Richly S. Joule Unit: a generic framework for software energy profiling and testing. 2013 Workshop on Green in Software Engineering, Green by Software Engineering (GIBSE) , 2013; 9–14.