Explainability as a Requirement for Hardware: Introducing Explainable Hardware (XHW)

TL;DR

This paper advocates for explainability in hardware, especially microchips, to enhance security and trust, proposing a framework inspired by explainable AI and software, supported by expert surveys.

Contribution

It introduces the concept of Explainable Hardware (XHW), developing a framework with stakeholder requirements and explainability approaches, and identifies research gaps through expert surveys.

Findings

Framework for XHW with stakeholder requirements

Identification of potential research gaps in XHW

Application of the framework in real-world scenarios

Abstract

In today's age of digital technology, ethical concerns regarding computing systems are increasing. While the focus of such concerns currently is on requirements for software, this article spotlights the hardware domain, specifically microchips. For example, the opaqueness of modern microchips raises security issues, as malicious actors can manipulate them, jeopardizing system integrity. As a consequence, governments invest substantially to facilitate a secure microchip supply chain. To combat the opaqueness of hardware, this article introduces the concept of Explainable Hardware (XHW). Inspired by and building on previous work on Explainable AI (XAI) and explainable software systems, we develop a framework for achieving XHW comprising relevant stakeholders, requirements they might have concerning hardware, and possible explainability approaches to meet these requirements. Through an exploratory survey among 18 hardware experts, we showcase applications of the framework and discover potential research gaps. Our work lays the foundation for future work and structured debates on XHW.

Full text

explainability, explainable hardware, explainable systems, explainable artificial intelligence, trustworthiness, stakeholders, desiderata, technical approaches

††copyright: none††booktitle: XXX††ccs: Hardware Integrated circuits††ccs: Security and privacy Human and societal aspects of security and privacy††ccs: Computing methodologies Philosophical/theoretical foundations of artificial intelligence

AI Artificial Intelligence API Application Programming Interface ASIC Application-Specific Integrated Circuit ASML Advanced Semiconductor Materials Lithography BEOL Back-End-Of-Line BSI Bundesamt für Sicherheit in der Informationstechnik CMOS Complementary Metal–Oxide–Semiconductor CPU Central Processing Unit CRT Chinese Remainder Theorem DARPA Defense Advanced Research Projects Agency DL Deep Learning DLL Dynamic Link Library DRM Digital Rights Management DSP Digital Signal Processor ECC Elliptic-Curve Cryptography EDA Electronic Design Automation EU European Union FEOL Front-End-Of-Line FIPS Federal Information Processing Standards FPGA Field-Programmable Gate Array GCD Greatest Common Divisor GDPR General Data Protection Regulation HCI Human Computer Interaction HDL Hardware Description Language HLEGAI High Level Expert Group on AI

IC Integrated Circuit IEC International Electrotechnical Commission IEEE Institute of Electrical and Electronics Engineers IoT Internet of Things IP Intellectual Property IRB Institutional Review Board ISO International Organization for Standardization MATE Man-at-the-End MCAS Maneuvering Characteristics Augmentation System MITM Man-in-the-Middle ML Machine Learning NIST National Institute of Standards and Technology PC Personal Computer PCB Printed Circuit Board PDK Process Design Kit PKI Public Key Infrastructure RAM Random-Access Memory RTL Register Transfer Level SEM Scanning Electron Microscope SoC System on a Chip TSMC Taiwan Semiconductor Manufacturing Company UK United Kingdom USA United States of America XAI Explainable AI

1. Introduction

In recent years, researchers have started questioning the ethical implications of digital technologies. In particular, software incorporating opaque Artificial Intelligence (AI) algorithms has led to intense discussions about fairness and accountability (Panesar, 2019; Lepri et al., 2018; Mittelstadt et al., 2016) in a wide range of technical systems, from autonomous vehicles to social networks. A common criticism of AI algorithms is that they are inscrutable black boxes—many people do not understand how these algorithms make decisions, despite their consequential impact on human life and well-being. This has led to the emergence of the rapidly growing field of Explainable (XAI), which is described by Langer et al. as being “concerned with developing approaches to explain and make artificial systems understandable to human stakeholders” (Langer et al., 2021, p. 1).

Explainability provides many benefits, one of the most often cited is its contribution to people’s trust in a system (Kästner et al., 2021). However, if our objective is to increase people’s trust in technical systems through explainability, the current discussions focusing only on algorithms appear too narrow. No algorithm, including every AI implementation, is in practice computed as a mere mathematical construct—it must actually run on a physical computer, that is, on a piece of hardware. This observation has important implications for explainability, especially as it relates to trust. If we are to trust algorithms, we must also necessarily trust the underlying hardware on which they are executed.

When it comes to hardware trust, consequential initiatives about Integrated Circuit (IC) manufacturing capabilities are currently taking shape around the world, e. g., the European Chips Act (European Comission, 2022) and the US CHIPS and Science Act (Senate of the United States, 2022). Through these multi-billion dollar investments, the governments behind them seek to gain more control over the production and supply of ICs and become less dependent on foreign manufacturers. Another prominent example is the ban of certain 5G communication systems from foreign manufacturers in the USA (Webster, 2019). The underlying concern is that adversarial actors can modify hardware in such a way that the algorithms running on top of it can be manipulated at will. Indeed, scientific studies have demonstrated that malicious hardware manipulations can have catastrophic consequences, potentially leading to a complete loss of security or incorrect algorithmic decisions. Such manipulations include the insertion of a kill switch to render military hardware inoperable under specifiable conditions (Adee, 2008), manipulation of Machine Learning (ML) accelerators (Clements and Lao, 2018; Li et al., 2018; Liu et al., 2017), or the compromising of hardware security primitives (Becker et al., 2013).

Motivated by hardware trust, we focus on the explainability of the very foundation of digital hardware, namely the ICs from which every computer system is built. Just as XAI seeks to make AI models comprehensible to various stakeholders (Arrieta et al., 2020; Langer et al., 2021; Chazette et al., 2021), the goal of hardware explainability is to make the hardware being used comprehensible.

From the XAI literature, we know that AI explainability can even provide a range of other benefits beyond trust, such as accountability, safety, legal compliance, and verifiability (Arrieta et al., 2020; Langer et al., 2021; Lipton, 2018; Chazette et al., 2021). The same might hold for hardware explainability, as illustrated by the following examples: First, the engine manipulations in the Volkswagen diesel emissions scandal revealed that the behavior of complex systems can be extremely difficult—if not practically impossible—to verify even for experts, making it hard to attribute accountability (Barthe et al., 2016; Baum, 2016; D’Argenio et al., 2017). Second, the two fatal accidents involving the Boeing 737 MAX (Cruz and de Oliveira Dias, 2020) resulting from erroneous sensor responses have shown that people have a valid concern about the safety and legal compliance of everyday systems like airplanes. Third, end users exhibit a lack of awareness of hardware-based camera indicators on laptops and smartphones; these indicators should serve as privacy safeguards in theory but fail to ensure users’ feelings of safety against webcam spying in reality (Portnoff et al., 2015). In all three cases, explainability might help.

However, achieving hardware explainability is no easy feat. Similar to the opacity of AI systems (Burrell, 2016), modern ICs are mostly opaque and thus difficult to explain. ICs have become increasingly complex, culminating, e. g., in the Apple M1 Ultra with 114 billion transistors (Shankland, 2022). In parallel to the probabilistic training processes of AI systems, non-deterministic design tools automate vast parts of the hardware design process, sometimes even relying on AI themselves. Meanwhile, IC supply chains are globally distributed and subject to increasing geopolitical tensions (Mozur et al., 2022), resulting in opaque manufacturing processes. The resulting non-explainability affects not only downstream stakeholders, such as end users (i. e., consumers or system operators) who interact with the systems but also experts who design them.

As a remedy, we take the first step towards a hardware explainability framework. Our main contributions are:

• •

Explainability for Hardware and Systems (Section 3). We transfer the notion of explainability to the hardware domain and motivate the need for hardware explainability through legislation related to the trustworthiness of modern hardware components. We argue that hardware explainability is an essential component to achieve explainability on a system level, building on existing models for AI explainability.

• •

A Comprehensive Framework for Hardware Explainability (Section 4). We conceive a framework for hardware explainability that unites stakeholders, their explainability needs (i. e., desiderata), and approaches from hardware manufacturing to achieve a more comprehensive understanding of hardware-based systems.

• •

Applying the Framework (Section 5). By analyzing real-world scenarios (i. e., the three examples from above) and transferring concepts from XAI, we illustrate how our framework can be used to address the explainability needs of specific stakeholders and identify directions for developing novel explainability approaches.

2. Working Definitions

To lay a common ground for this work, we introduce relevant terms and processes of the hardware ecosystem.

A Broad System Definition

In this work, we refer to a system as something that consists of different hardware and software components to serve a dedicated purpose. In other words, a system can be anything from a smartphone, a car, or a PC. A system may also incorporate smaller subsystems, thereby introducing a hierarchy of systems. We define hardware as physical electronic components such as chips (i. e., ICs). Software is executed on hardware and includes drivers, operating systems, user applications, and algorithms such as AI.

A Rough Sketch of Hardware Design and Manufacturing

Modern hardware is specified and designed using high-level Hardware Description Languages. The resulting schematics are mapped to a technology library that describes all circuit elements available for realizing the design. This process is known as synthesis. The technology libraries used in this process are typically provided by large (contract) manufacturers, also called foundries.

The implemented design, stripped of all high-level information like hierarchy, labels, and comments, is passed on to the manufacturer. They produce the actual chip using the chosen manufacturing technology in one of their production facilities, i. e., a semiconductor manufacturing plant also referred to as fab (Geiger et al., 1990). In the last step, the fabricated chip can be integrated into various systems that are then employed in diverse areas of our digital society, such as telecommunications, goods and passenger transport, public infrastructure, or military and space applications.

3. The Necessity for Hardware Explainability

The design and manufacturing of high-end semiconductors has increasingly become a political issue. Both the United States of America (USA) (Senate of the United States, 2022) and the European Union (EU) (European Comission, 2022) have pledged 52 billion USD and 43 billion EUR, respectively, to invest in domestic semiconductor manufacturing and reduce reliance on foreign chip manufacturers (Clark, 2021).

One primary concern with reliance on foreign-made semiconductors is implanted backdoors especially in secure hardware components used by the military or in space (of Texas, 2010; Board, 2005; European Comission, 2022). Therefore, trustworthy hardware requires built-in security and safety, even if it comes at the cost of higher complexity and power consumption (European Comission, 2022, p. 44). According to the European Chips Act, this requires “a solid understanding of a chip’s architecture” (European Comission, 2022, p. 44).

Towards meeting these political demands, we explore ways to transfer the notion of explainability to the hardware domain. We first explore the root causes for the opaqueness of contemporary hardware. This opaqueness makes the hardware susceptible to malicious modifications that are hard to detect after manufacturing due to the complexity of the chips and the manufacturing processes involved. Thereby, we derive the necessity of hardware explainability to achieve explainability on the system level. To represent this necessity most clearly, we make use of a model (Langer et al., 2021; Hoffman et al., 2018) that makes explicit how explainability can help satisfy peoples’ different goals or interests. This model serves as a foundation for a hardware explainability framework that we introduce in Section 4.

3.1. Causes of Hardware Opaqueness

To understand why hardware is largely opaque today, it is worth taking a brief look at the history of hardware design.

Since the invention of the transistor by Jack Kilby in 1958, technological progress has led to a steady increase in efficiency of ICs. While in the 1970s, hardware developers often had to design circuits by hand, today sophisticated Electronic Design Automation (EDA) toolchains take over (Weste and Harris, 2010). In particular, the introduction of HDLs starting in the 1980s simplified the hardware design process, as they decouple logical design from the actual circuit, allowing for modular hardware designs. At the same time, shrinking transistor sizes enabled more complex and deeply integrated hardware systems, e. g., Systems on a Chip, which are also less expensive to manufacture.

As another factor, vertical integration and globalization of the supply chain have fostered innovation in each step of the complex manufacturing processes. Meanwhile, these efforts have led to an oligopoly of a very few, highly specialized companies. In particular, the production of the most advanced ICs requires investments in the double-digit billions (Reuters Staff, 2017), which can only be raised by a few manufacturing companies such as Taiwan Semiconductor Manufacturing Company (TSMC) or Samsung that are typically subsidized by governments.

Over time, the players making up the globally distributed supply chain have built up know-how that gives them considerable advantages over new competitors. For example, Advanced Semiconductor Materials Lithography (ASML) Holding N.V. is the only company in the world that manufactures the machines needed for making leading-edge ICs. Designing and manufacturing a modern IC using such machines, which can cost up to 150 million USD a piece, requires up to 1500 processing steps and takes more than six months (European Comission, 2022, p. 11).

While disruptive innovations in hardware manufacturing are shaping our everyday life, ICs are largely opaque to everyone involved today. This opaqueness or un-explainability is caused by the very same innovations which enabled the technological progress in the first place. Adopting the three types of opacity for ML algorithms suggested by Burrell (Burrell, 2016), we now explore these causes for hardware opaqueness.

First, hardware is opaque to most people due to technical illiteracy (Burrell, 2016), even to some experts. The ever-growing complexity and shrinking feature sizes make physical inspection and verification of manufactured integrated circuits a challenging task, mastered by only a few highly specialized companies or government agencies worldwide.

Second, modern synthesizers are based on efficient heuristic or even AI-based algorithms, which (for complex circuits) leads to different results for every synthesis run. This correlates to what Burrell calls opacity characteristic of ML (Burrell, 2016).

Third, developers and manufacturers use obfuscation to protect their Intellectual Property (IP) and thus their investments, which relates to opacity as intentional corporate secrecy (Burrell, 2016). The situation is further complicated by conflicting interests of nation states: they demand openness of foreign hardware while striving to protect domestic systems from unauthorized access to the highest extent (European Comission, 2022; Board, 2005).

3.2. From System Explainability to a Definition of Hardware Explainability

In Section 3.1, we identified a diverse set of factors that make contemporary hardware mostly opaque. Interestingly, there have been parallel ongoing discussions in the AI community: modern AI systems are also considered opaque, despite the consequential societal impacts these systems could generate. Consequently, in recent years there have been growing discussions on Explainable (XAI), aiming to open up the black box of modern AI and make it understandable to different addressees (Langer et al., 2021; Páez, 2019) to ensure properties like trust, accountability, and safety. The question now is to what extent the concept of explainability can be transferred to the hardware domain, with the analogous goal of making hardware understandable, to ensure similar properties.

If we consider hardware as part of a system, this transfer can take place quite straightforwardly. In fact, recent work has extended explainability from AI to software or even systems holistically (Brunotte et al., 2021; Köhl et al., 2019; Chazette et al., 2021; Brunotte et al., 2022). AI can exhibit a high degree of opacity—the same is true for the increasingly complex computing systems used today. Accordingly, researchers have emphasized the connection between explainability and understanding not only AI, but more general aspects of a system. Chazette et al.  (Chazette et al., 2021) cast this connection into a definition:

Definition 0 (System Explainability).

A system $S$ is explainable with respect to an aspect $X$ of $S$ relative to an addressee $A$ in context $C$ if and only if there is an entity $E$ (the explainer) who, by giving a corpus of information $I$ (the explanation of $X$), enables $A$ to understand $X$ of $S$ in $C$.

In the definition above, the aspect $X$ to be explained may well be the hardware, since it is a constituent—and, thus, certainly also an aspect—of a system $S$. With this finding, a glaring gap in prior research becomes apparent: To make a system truly explainable, its hardware components must be made explainable as well. Merely making an AI—or the software that implements it—explainable is not sufficient.

To the best of our knowledge, however, there is no prior research on the explainability of hardware, despite it being an essential constituent of systems. Furthermore, stakeholders’ interests cannot be fully met if we only focus on explainability at the algorithmic level. To see why this is the case, let us consider trustworthiness again as one of the goals of explainability. Kästner et al. propose the following operationalization of trustworthiness for systems (Kästner et al., 2021):

Definition 0 (Trustworthiness).

A system $S$ is trustworthy to a stakeholder $H$ in a context $C$ if and only if

1. (a)

$S$* works properly in $C$, and* 2. (b)

$H$* would be justified to believe that (a) if $H$ came to believe that (a).*

For a system to work properly and thus be trustworthy, we have to factor in all its constituents—software and hardware. Furthermore, to find out whether the hardware works properly—and be justified in believing that it does so—we need to understand how the hardware is composed and how it operates. That is where explainability comes into play according to Kästner et al.  (Kästner et al., 2021): by means of explaining a certain aspect of the system (e. g., its hardware), the understanding of a stakeholder is facilitated, letting them justifiably judge whether this aspect works properly.

Against this background, we can define hardware explainability building on Definition 3.1:

Definition 0 (Hardware Explainability).

A piece of hardware $W$ is explainable with respect to an aspect $X$ of $W$ relative to an addressee $A$ in context $C$ if and only if there is an entity $E$ (the explainer) who, by giving a corpus of information $I$ (the explanation of $X$), enables $A$ to understand $X$ of $W$ in $C$.

3.3. A Model of Explainability

As we have seen in the previous section, explainability promises to help facilitate trustworthiness. The same is true for various other desirable properties of systems, be it their debuggability, safety, security or the like. These desirable properties to be achieved by explainability are often referred to as desiderata (Langer et al., 2021; Lipton, 2018).

Coming back to the definition of explainability in the context of systems (see Definition 3.1), all desiderata are downstream goals. In other words, gaining an understanding of a system through explainability facilitates the satisfaction of a desideratum for a stakeholder. Langer et al.  (Langer et al., 2021) and Hoffman et al.  (Hoffman et al., 2018) propose similar models of this process (see Figure 1 for a simplified depiction). Founded in these models, we develop our framework for hardware explainability.

Roughly speaking, the models propose that explainability approaches (i. e., ways of achieving explainability) provide explanatory information to facilitate a stakeholder’s understanding. This understanding, in turn, affects the satisfaction of certain desiderata, either directly or indirectly.

On the one hand, the relation is direct, when the satisfaction of a desideratum, by obtaining the explanatory information, is facilitated. On the other hand, the relation is indirect, when, with the help of the obtained explanatory information, the stakeholder can judge whether a desideratum is satisfied. In case the desideratum is not satisfied, the stakeholder can optimally initiate steps to counter this state. For example, when a system is not safe, more information about how it works, on its own, will not make the system safer. However, this information can be presented to stakeholders who can then improve the system such that it becomes safer (e. g., in the case of system designers) or steer away from unsafe systems (e. g., in the case of end users).

4. A Framework for Hardware Explainability

Based on the model for explainability introduced in Section 3.3, we developed a framework of hardware explainability. The framework highlights the individual desiderata of the stakeholders interacting with a hardware-based system and how to achieve them using existing approaches from the hardware domain (see Figure 2 for a high-level illustration).

Through literature review, we identify relevant stakeholders (Section 4.1), desiderata (Section 4.2), and—in absence of established approaches to hardware explainability—unify existing methods and techniques from the hardware design and manufacturing domain under the notion of explainability (Section 4.3). Figure 3 presents an overview of the components included in our framework.

4.1. Stakeholder Ecosystem

Various stakeholders interact with a hardware-based system throughout its lifetime. Depending on their background, these stakeholders have individual desiderata concerning the explainability of hardware. To address these individual desiderata, we introduce stakeholder categories to classify entities that interact with the hardware by developing, manufacturing, integrating, regulating, or using it. These categories are adapted from existing work by Langer et al.  (Langer et al., 2021) and Tomlinson et al.  (Tomlinson et al., 2022): ($i$) designers, ($ii$) manufacturers, ($iii$) system integrators, ($iv$) policymakers and watchdogs, and ($v$) end users.111An entity (a person, company, organization, or government agency) may belong to multiple stakeholder categories at the same time. Below, we outline each stakeholder category as well as the interactions between the stakeholders and the system (see Figure 4). Having a clear understanding of the stakeholders, their relationships, and desiderata is a prerequisite for our analysis of existing methods and how they contribute to the explainability of hardware.

Designers

Hardware is specified and conceptualized by designers who describe the desired functionality in a high-level language. A designer commonly has a good understanding of the hardware components they have worked on and the ones that their components are interacting with. However, especially for large designs, they have limited insights into the components that they have not been working on. Once completed, the design is passed on to the manufacturer that produces the actual hardware (i. e., the IC).

Manufacturers

Following design, the hardware must be produced using highly specialized tools and equipment. This step is performed by the manufacturers. A manufacturer may, for instance, be a fab run by TSMC or Samsung, two of the most advanced foundries world-wide. Manufacturers rarely have access to high-level descriptions of the hardware they produce—they only receive data such as circuit diagrams of already synthesized hardware designs.

System Integrators

An integrator takes hardware such as an IC and integrates it into a complete system together with other hardware and software components. Depending on the complexity of the final product, there may be different tiers of integrators along the value chain, each building upon the (sub)systems of the others to create increasingly complex systems such as cars or airplanes. Through data sheets and user manuals, system integrators have access to the specification of hardware components or subsystems, especially in terms of their functionality and connectivity. However, they usually do not know implementation details. This is because to compose a system, the integrator must primarily establish interactions between different hardware components and subsystems, which only requires a superficial understanding of these components.

Policymakers & Watchdogs

Policymakers set the legal framework in which a system operates by introducing laws and standards to which designers, manufacturers, and system integrators must adhere. While the legal framework is created by legislators in parliaments and governments, standards are commonly set by industry bodies and government agencies such as the Institute of Electrical and Electronics Engineers (IEEE), the National Institute of Standards and Technology (NIST) in the USA, or the Bundesamt für Sicherheit in der Informationstechnik (BSI) in Germany. Policymakers receive demands and concerns about existing or future legislation from industry and consumer groups, but have virtually no insight into the actual hardware or system. In addition, they must protect national interests, e. g., by introducing export control measures and securing critical infrastructure against foreign interference.

Watchdogs can be certification bodies that attest adherence to specific standards and norms. They also include government agencies with enforcement capabilities regarding non-conformity to existing legislation and patent or IP infringements. Therefore, such watchdogs may run their own analysis laboratories to conduct extensive analyses on a given hardware component or system. Watchdogs can, in addition, request detailed information on a device from the designers, manufacturers, or system integrators.

End Users

Following Tomlinson et al.  (Tomlinson et al., 2022), we define end users as consumers or operators of a system. A consumer directly uses the system such as a car or a smartphone. An operator (also called “deployer”) offers a service that is directly dependent on the system. Examples of operators are railway operators and telecommunication providers.

Commonly, consumers know little about technical details of the system—including its hardware and software—and do not understand the relationships between its individual components. For example, an average consumer may know how to download and use apps for a smartphone, but they are unlikely to know about the existence of the ICs featured in their phone. An operator potentially has more insights into the general architecture as they need to be able to maintain and service the system, but still only on a high abstraction level.

4.2. Desiderata

The goals and purposes of explainability—also referred to as desiderata—strongly depend on the perspectives and requirements of the stakeholders involved. Depending on each stakeholder’s viewpoint, some aspects of explainability may be more relevant than others. Below, we give a concise overview of the desiderata most relevant in the context of hardware explainability by adapting existing work on XAI and explainable software by Chazette et al.  (Chazette et al., 2021) and Langer et al.  (Langer et al., 2021). Our selection of desiderata is based on a literature review as well as expert knowledge.

Safety

Reliable and safe operation of hardware-based systems is crucial to prevent harm to people and businesses (Jeon et al., 2011; Grimm et al., 2018; Tumer and Smidts, 2011). Failure of such systems can impact revenue or even lead to loss of life. Explainability can help identify and address physical safety hazards, by facilitating the understanding of how to operate a system in the right way.

Accountability

In the event of a system failure, attributing responsibility (Nissenbaum, 2020; Kroll, 2021) is crucial not only for legal reasons, but also to identify the failure’s root cause and who is in charge of repairs. However, as systems become more complex, a responsibility gap has emerged (Raji et al., 2020; Matthias, 2004) as it becomes more difficult to assign responsibility. Explainability can restore responsibility by helping to understand an error, facilitating the assignment of fault to the parties involved.

Debuggability

Due to its growing complexity, modern hardware is increasingly hard to design and maintain. Hence, debuggability (Rath, Dominic, 2008) of hardware is desirable to, e. g., identify, trace, and correct bugs in order to prevent potential hardware malfunction (Tehranipoor and Wang, 2012). Explainability improves debuggability as additional information on a component or a system help developers identify and fix mistakes (Adadi and Berrada, 2018).

Legal Compliance

Laws and regulations provide a framework for all stakeholders to operate within (European Comission, 2022). An ever more versatile hardware market, increasing opaqueness of systems, and accelerating development cycles complicate verifying compliance with existing legislation. Explainability contributes to legal compliance by enabling stakeholders to investigate whether a system or hardware component meets legal requirements and standards.

Security

Given our society’s increasing reliance on digital systems, exploitation of security vulnerabilities can undermine confidentiality, integrity, and availability of data (Karri et al., 2010; Lipp et al., 2018; Kocher et al., 2019; European Comission, 2022). Therefore, it is critical to ensure security on all system levels, particularly hardware (Tehranipoor and Wang, 2012). Explainability bridges the gap between perceived and actual security (Pieters, 2011), helping users understand the actual security mechanisms in systems and adjust their behavior accordingly. Furthermore, explainability helps assess the security of hardware designs as well as products and warrant appropriate certifications.

Verifiability

In the presence of a globally distributed—and thereby opaque—hardware supply chain, verification of (parts of) a system is vital to check for the absence of unintended bugs or faults, test correct operations, and rule out targeted manipulations (Lippmann et al., 2020; Jha and Gupta, 2002; Gupta, 1992). Explainability aids verifiability by making the inner workings of hardware components more comprehensible, making it easier to judge whether a system is working properly.

Trustworthiness

In the hardware domain, trustworthiness (European Comission, 2022) is described as “the degree to which the security behavior of the [hardware] component is demonstrably compliant with its stated functionality” (Benzel et al., 2005). This aligns with the definition of trustworthy systems by Kästner et al.  (Kästner et al., 2021), see Definition 3.2: the hardware must (a) work properly and (b) come with the means to demonstrate that it is doing so. For the latter in particular, explainability is paramount as it provides stakeholders with the appropriate tools and resources to show that the hardware is functioning as intended.

Trust

While trustworthiness describes a property of a hardware-based system, trust is “the degree to which the user or a component depends on the trustworthiness of another component” (Benzel et al., 2005). Undertrust can result in users’ interference with the system, thereby degrading efficiency, and overtrust may result in inadequate or even insecure system usage (Lee and See, 2004; Parasuraman and Riley, 1997; Hoff and Bashir, 2014; Parasuraman and Manzey, 2010). Explainability allows stakeholders to appropriately calibrate their trust (Hoffman et al., 2018).

4.3. Leveraging Approaches from Hardware Design and Manufacturing for Explainability

In this section, we examine approaches currently employed in the hardware design and manufacturing processes with regard to their potential for enhancing hardware explainability. The intended purpose of the individual approaches varies—some address trustworthiness and security, whereas others enhance debuggability and verifiability. Furthermore, we illustrate where these existing approaches have their own limitations.

Trusted Manufacturing

In 2005, the US Defense Advanced Research Projects Agency (DARPA) proposed trusted manufacturing to prevent targeted manipulations in military-grade ICs by a malicious foundry (Board, 2005). Trusted manufacturing aims to retain the necessary manufacturing capabilities at domestic companies.

A variant of trusted manufacturing is split manufacturing, in which the most complex and advanced Complementary Metal–Oxide–Semiconductor (CMOS) manufacturing steps are outsourced to untrusted high-end facilities. The remaining steps, primarily the interconnect wiring of an IC, are conducted by a trusted foundry using older technologies (Vaidyanathan et al., 2014; Imeson et al., 2013; Rajendran et al., 2013). The idea behind split manufacturing is that the untrusted entity cannot reconstruct the hardware design given only the limited information it is provided with.

Trusted manufacturing can be viewed as a hardware explainability approach. With respect to Definition 3.3, the explained aspect $X$ of the hardware is its manufacturing process, and the information provided is that critical steps of this process have not been subjected to malicious manipulation by third parties.

Unfortunately, trusted manufacturing is expensive, as the investment cost for a modern foundry is high, while only a few advanced foundries are in demand worldwide. In addition, techniques such as split manufacturing can be error-prone if production processes are not precisely aligned.

Standards & Certifications

The first standards for machines were established in the 19th century to ensure operator safety. While the standardization process was traditionally dominated by the largest economies, today institutions such as the International Organization for Standardization (ISO) define global standards. Ideally, a new hardware standard is developed, refined, and implemented in close collaboration with industry stakeholders. Over the past decades, numerous standards have been established in the hardware and semiconductor sector. These standards include product certifications at various stages of the supply chain as well as site certifications, specifically for (trusted) foundries.

Standards and certifications facilitate explainability as conformity with a standard provides additional information on the hardware. The nature of that information depends on the scope of the standard, but can include design and manufacturing best practices, implementation details, and safety as well as security assertions.

While the concept of standards appears to have many benefits, standardization processes can often be lengthy and require a great deal of resource commitment (e. g., time, expertise, willingness to compromise) from all involved stakeholders. It is possible that standards will end up at the lowest common denominator of industry and policymakers, will not be properly executed or even be flawed (Speith et al., 2022; Chhotaray et al., 2017), or in the worst case, contain maliciously built-in backdoors (Checkoway et al., 2014).

Open Hardware

Today’s commercial hardware systems are based on closed and proprietary IP. In contrast, open hardware strives for transparency throughout the supply chain. Open hardware in the context of ICs includes open-source HDL implementations (e. g., OpenTitan (lowRISC CIC, 2019), OpenRISC (Community, 2018), OpenCores (Oliscience, 1999), or LibreCores (Free and Foundation, 2018)), EDA tools (Wolf, 2016; Ajayi et al., 2019), and manufacturing techniques (Nangate, 2014; Ansell, 2020), many of which are united under the umbrella of the Chips Alliance (The Linux Foundation, 2019).

Open hardware can facilitate explainability, as it is designed transparently such that it already provides information to help at least hardware experts understand its architecture. Furthermore, it allows for verification of tools and manufacturing processes and thereby evokes trust in the manufactured hardware, e. g., by assuring its security.

Limitations of open hardware include a possible lack of long-term support and funding: successful open hardware projects are often supported by and dependent on major corporations. There are also potential quality issues due to the superiority of proven proprietary EDA tools and technologies, as well as liability and warranty issues.

Testing & Verification

To evaluate the correctness of a hardware design or manufactured IC, both testing and verification are extensively used throughout the entire manufacturing process (Geiger et al., 1990). Early identification of bugs, faults, and defects is attempted by simulation (Kleijnen, 1999; Nayani and Mollaghasemi, 1998), testing (Geiger et al., 1990; Maly, 1987), and formal verification (Kuehlmann et al., 1995; Althoff et al., 2011; Kern and Greenstreet, 1999). Both simulation and testing target the dynamic execution of the hardware and hence facilitate an understanding beyond just a static circuit description. Formal verification can provide guarantees on the equivalence of an implementation and its specification.

Testing and verification support explainability by providing information on passed (or failed) tests or verification procedures. The explained aspects are the properties that were tested or verified.

However, such techniques are limited in their applicability. Simulation and testing can only take into account a small subset of all possible inputs, hence some circuit states may never be reached (Kern and Greenstreet, 1999). Formal verification is computationally expensive such that it can only be applied to smaller subcircuits (Kern and Greenstreet, 1999), but not to more complex designs such as advanced Central Processing Units or SoCs.

Physical Analysis

Invasive or even destructive physical analysis is used to verify the correctness of an IC and check for undesired information leakage. To rule out malicious modifications, a specialist can check whether the fabricated chip matches specification, design, or implementation files by performing hardware reverse engineering (Torrance and James, 2009; Azriel et al., 2021; Fyrbiak et al., 2017). Furthermore, side-channel evaluation (Kocher, 1996; Agrawal et al., 2002; Kocher et al., 1999) and fault injection (Antoni et al., 2000; Hsueh et al., 1997; Ziade et al., 2004) are leveraged to detect or even provoke unintended runtime behavior, e. g., the leakage of protected data like cryptographic keys.

As it aims to make (parts of) an IC understandable, reverse engineering promotes explainability. The extent of this understanding depends on the goals of the reverse engineering efforts, but can reach up to a full high-level description of the entire circuit. Side-channel and fault analysis provide insights into the runtime behavior of an IC, e. g., by analyzing power consumption or electromagnetic emissions, and thereby extract information that facilitates understanding.

Reverse engineering is limited in applicability due to the associated costs, the required effort, and the fact that the analyzed IC is destroyed in the process. Just the equipment required to perform such analyses, e. g., an ion mill and a special-purpose Scanning Electron Microscope (SEM), can easily cost multiple million USD, not even taking into account the know-how required to operate the machines and investigate their outputs. Furthermore, running side-channel or fault injection attacks is always only a best-effort approach as full coverage of the entire attack space is infeasible.

5. Application of the Hardware Explainability Framework

Having presented our hardware explainability framework, we now discuss its possible applications. On the one hand, our framework is suitable to determine which explainability approach a stakeholder needs to satisfy a desideratum of their choice. On the other hand, the framework can help determine what kind of novel explainability approach is required if no suitable method is available. Below, we showcase these two applications.

5.1. Applying the Framework to Analyze Real-World Scenarios

Our framework lays the foundation for determining which explainability approach a stakeholder needs to satisfy a specific desideratum. To illustrate this point, we discuss the three examples presented in Section 1, referring to approaches that (if applied) might have helped mitigate the issues at stake. We also reflect on limitations of the approaches, highlighting how they may fall short in addressing the desiderata for certain stakeholders. Our suggestions should be interpreted as preliminary ideas that need to be empirically verified in future work.

In the VW engine manipulation case, open hardware could have ensured accountability, legal compliance, and trust for policymakers and watchdogs. With information obtained by analyzing open hardware designs, members of this stakeholder class could have gained a sufficient understanding to notice and prevent malicious tampering. However, open hardware would most likely fail to address the need for explainability among end users as they generally do not have the expertise to understand the technical details of the system even when they are made transparent. Even worse, standards and certifications as well as testing and verification were actively circumvented and outsmarted by VW (Contag et al., 2017).

Concerning the crashes of two Boeing 737 MAX planes in 2019 and 2020, proper testing procedures and rigorous certification processes could have mitigated safety concerns and ensured legal compliance. If sufficient efforts of testing and verification had been performed in advance, the airplane operator could have obtained insights into the status of hardware components and assess the risks of malfunctioning. In this case, authorities did not enforce the required level of explainability, which ultimately resulted in fatalities. However, there seems to be an explainability gap for end users as well in this case: none of the existing approaches works to unpack the technical details of a complex system like an aircraft in a format suitable for end users, such that these can be (re-)assured of the aircraft’s safety.

Lastly, in the case of hardware-based camera indicators, whether the indicator indeed fulfills its privacy promise (i. e., there is no covert data collection when the LED light is off) can be ensured through physical analysis by experts such as watchdogs or system integrators. Here, the recovery and analysis of circuit schematics facilitates an understanding of the employed hardware and thereby supports satisfaction of the respective desiderata. While end users lack the equipment and expertise to apply physical analysis, knowing that the hardware conforms with certifications can be a first step for them to get assurance and build trust.

A unifying theme among the examples above is that end users’ need for explainability and the various associated desiderata are potentially not fulfilled in most cases. A possible reason is that the approaches we summarize in our framework—as techniques from hardware design and manufacturing—do not have end users as the target audience.

The second application of our framework is exactly for such cases, that is, to find out what kind of novel explainability approach is required if it turns out that there is no suitable method available. With this idea, we follow Langer et al.  (Langer et al., 2021) and Belle and Papantonis (Belle and Papantonis, 2021), who assume that certain properties of explainability approaches can guide which approach is required for a specific aim. To use this idea, we transfer existing findings from XAI to hardware explainability.

5.2. Transferring Basic Concepts From XAI to Hardware Explainability

In particular, we discuss how properties of XAI methods can be transferred to classify hardware explainability methods. For this, we leverage the comprehensive review by Speith (Speith, 2022), which classifies existing approaches and surveys the most relevant terminology in XAI. According to Speith, approaches that facilitate the understanding of AI systems can be classified by the stage at which they are applied, their applicability to different models, and their scope.

Stage

In XAI, there is a distinction between directly training explainable AI models (ante-hoc explainability) and explaining an opaque model after training (post-hoc explainability) (Speith, 2022; Arrieta et al., 2020; Belle and Papantonis, 2021; Langer et al., 2021; Sokol and Flach, 2020; Arya et al., 2019; Adadi and Berrada, 2018). We could similarly categorize hardware explainability approaches. Those approaches that take explainability into account from the very beginning of the hardware design and manufacturing process facilitate ante-hoc hardware explainability (e. g., open hardware and trusted manufacturing). Approaches such as physical analysis strive to explain already manufactured hardware, hence attempting to achieve post-hoc hardware explainability. For other approaches such as standards and certifications or testing and verification, this distinction is not as clear. For example, standard and certification requirements may have ante-hoc effects on hardware explainability by influencing early design stages, but post-hoc effects as well due to the downstream certification process.

Applicability

Furthermore, explainability approaches for AI are distinguished by their applicability. Approaches that are specific only to certain kinds of AI models are referred to as model-specific while those that apply to AI models more generally, independent of their internal architecture, are called model-agnostic (Speith, 2022; Guidotti et al., 2021, 2019; Langer et al., 2021; Sokol and Flach, 2020; Ribeiro et al., 2016). In XAI, this differentiation is only applied to post-hoc explainability approaches. However, for hardware explainability it makes sense to consider the applicability of ante-hoc approaches as well. For example, trusted manufacturing (especially split manufacturing) may not be applicable to all hardware architectures and technologies and must thereby be considered hardware-specific, while open hardware is rather hardware-agnostic. Some post-hoc approaches such as testing and verification or physical analysis are largely dependent on the specific type of hardware and hence should be considered hardware-specific, while others such as standards and certifications can be applied independently of the hardware type and are therefore hardware-agnostic.

Scope

In XAI, explainability approaches can be applied locally or globally, i. e., only for a single prediction of a model or for the model in its entirety (Speith, 2022; Guidotti et al., 2019; Langer et al., 2021; Sokol and Flach, 2020; Ribeiro et al., 2016). Again, this notion can be adapted to hardware explainability. Approaches facilitating the explainability of a single piece of hardware, e. g., a chip, can be considered local, while those that target the hardware architecture may be referred to as global. Local approaches include physical analysis as well as testing and verification, since their goal is to gain insights into a specific piece of hardware and are highly dependent on the properties of that target. Trusted manufacturing, open hardware, and standards and certifications are considered global as they influence the design and manufacturing process rather than the hardware itself and thereby affect not just a single piece of hardware.

5.3. Identifying Directions for Novel Approaches

With these concepts in place, we now illustrate our framework’s second possible application by identifying directions for new approaches benefiting end users: if the goal is to make hardware understandable to end users (so that the potential explainability gaps identified in Section 5.1 are closed), what would new approaches look like?

Let us assume that safety is the primary desideratum of interest to end users. Drawing on the transferred concepts (Section 5.2), we anticipate that a hardware-agnostic, post-hoc, and global approach could be the best kind of explainability approach for end users. End users might struggle to make sense of a hardware component’s architecture, thus ante-hoc approaches that aim to make them more accessible will likely miss their goal. For a similar reason, hardware-agnostic approaches may be more effective for end users, as they do not require expert knowledge and are not tied to specific hardware architectures. Finally, global approaches to hardware explainability are expected to be more useful for end users, who are typically not interested in the properties of individual components.

An explainability approach that fulfills these criteria could be hardware labels similar to those on foods and beverages. Inspiration for the criteria depicted on these labels could be drawn from standards and norms such as ISO 26262 (for Standardization, 2018) and IEC 61508 (Brown, 2000) for hardware safety, FIPS PUB 140-3 (of Standards and Technology, 2019) for hardware security, and privacy and security labels for IoT devices (Naeini et al., 2020) and smartphone apps (Cranor, 2022). These labels could provide end users with the necessary and comprehensible information to facilitate their understanding of a hardware-based system. This, in turn, affects the satisfaction of their safety desideratum by empowering end users to make justified decisions for or against a system’s use.

6. Limitations and Future Work

The exploration of hardware explainability offers great potential for rendering hardware more comprehensible to all stakeholders. However, it is a complex and multifaceted subject that cannot be fully addressed in a single publication. In this section, we discuss the limitations of our work and propose possible directions for future research.

Choice of Framework Components

While our framework (see Section 4) provides a novel and comprehensive approach to hardware explainability, the components we selected for stakeholders, desiderata, and explainability approaches may not encompass the entire hardware landscape. We leveraged prior work to cover a wide range of components, transferring and compiling ideas from various fields. For instance, many of the explainability approaches have their roots in the hardware manufacturing domain, but possess the potential to serve a wide range of explainability needs. As technology and explainability-related debates evolve, we anticipate the need for further refinement and expansion of our framework components, emphasizing the significance of continued research in this area.

Evaluating the Framework’s Effectiveness

We applied our hardware explainability framework to three examples, discussing the promises as well as limitations for existing approaches to tackle the desiderata at stake, see Section 5.1. However, to truly understand each approach’s practical effectiveness, empirical studies, such as surveys or stakeholder interviews, are needed. Such studies should evaluate the suitability of the selected explainability approaches in meeting the desiderata of each stakeholder. Additionally, the evaluation process may bring forth novel explainability approaches.

Insights from XAI

We focused on reviewing basic concepts in XAI and transferred them to explainable hardware, see Section 5.2. However, there are other advanced concepts and classifications that are discussed in XAI literature, e. g., in the work of Schwalbe and Finzel (Schwalbe and Finzel, 2021), Sokol and Flach (Sokol and Flach, 2020), and Speith (Speith, 2022). Future research could further evaluate and adapt existing XAI research to develop new explainability approaches.

Trade-Offs

Analogous to similar debates in XAI (Wang and Kurz, 2022), disclosure of hardware designs and layouts may result in major revenue losses, as hardware Intellectual Property (IP) is a billion-dollar market. Open hardware, for example, may require companies to sacrifice profits in the interest of explainability. However, greater explainability does not necessarily require IP disclosure. Other options, such as trusted manufacturing or standards and certifications can provide explainability without revealing proprietary information.

The trade-off between performance and explainability is an intensely discussed challenge in the field of XAI (Rudin, 2019). In order for AI systems to become explainable, additional resources may be required, which can potentially reduce the overall performance of the system (Gunning and Aha, 2019; Angelov et al., 2021). In the context of hardware, this trade-off can even be more pronounced, as open-source hardware often lags behind the performance of proprietary ICs manufactured using state-of-the-art processes. Moreover, efforts to simplify hardware designs to enhance explainability can sometimes result in reduced performance, as evidenced by certain countermeasures for hardware security vulnerabilities (Prout et al., 2018). However, it is important to note that explainability approaches such as physical analysis or testing and verification do not impact the performance of the hardware as they do not interfere with the design. Future versions of our framework could take such trade-offs into account and thus enable fine calibration of the approaches.

7. Conclusion

In this work, we introduced the concept of explainable hardware, which is crucial to achieve explainability on system level. Based on models from XAI, we have developed a framework for hardware explainability that considers the perspectives of different stakeholders, their desiderata, and existing explainability approaches from the hardware design and manufacturing domain. Finally, we reflected on limitations of our research and identified future research directions.

We present hardware explainability as a new and important area of research. Future work could draw on literature and empirical methods to refine existing approaches and develop novel approaches for hardware explainability, with an emphasis on addressing the needs of underrepresented stakeholders. Explainable hardware paves the way for holistically explainable systems that can become an important building block for a self-determined modern digital society.

Acknowledgements.

Work on this paper was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy—EXC 2092 CASA—390781972, through the DFG grant 389792660 as part of TRR 248, and by the Volkswagen Foundation grants AZ 98509 and AZ 98514 “Explainable Intelligent Systems” (EIS). The Volkswagen Foundation and the DFG had no role in preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication. The authors declare no other financial interests.