Architectural Middleware that Supports Building High-performance, Scalable, Ubiquitous, Intelligent Personal Assistants

TL;DR

This paper introduces a middleware architecture that simplifies building high-performance, scalable, and ubiquitous intelligent personal assistants by abstracting complex low-level details and demonstrating its effectiveness through a conversational IPA case study.

Contribution

The paper proposes a platform-agnostic middleware architecture for IPAs and provides a concrete implementation that supports high-level development and large-scale deployment.

Findings

Middleware reduces development complexity and effort.

System scales to hundreds of thousands of IPAs without extra effort.

Analysis shows improvements in software metrics like complexity and performance.

Abstract

Intelligent Personal Assistants (IPAs) are software agents that can perform tasks on behalf of individuals and assist them on many of their daily activities. IPAs capabilities are expanding rapidly due to the recent advances on areas such as natural language processing, machine learning, artificial cognition, and ubiquitous computing, which equip the agents with competences to understand what users say, collect information from everyday ubiquitous devices (e.g., smartphones, wearables, tablets, laptops, cars, household appliances, etc.), learn user preferences, deliver data-driven search results, and make decisions based on user's context. Apart from the inherent complexity of building such IPAs, developers and researchers have to address many critical architectural challenges (e.g., low-latency, scalability, concurrency, ubiquity, code mobility, interoperability, support to cognitive services and reasoning, to name a few.), thereby diverting them from their main goal: building IPAs. Thus, our contribution in this paper is twofold: 1) we propose an architecture for a platform-agnostic, high-performance, ubiquitous, and distributed middleware that alleviates the burdensome task of dealing with low-level implementation details when building IPAs by adding multiple abstraction layers that hide the underlying complexity; and 2) we present an implementation of the middleware that concretizes the aforementioned architecture and allows the development of high-level capabilities while scaling the system up to hundreds of thousands of IPAs with no extra effort. We demonstrate the powerfulness of our middleware by analyzing software metrics for complexity, effort, performance, cohesion and coupling when developing a conversational IPA.

Tables

Table 1. TABLE I: Latency Comparison. AMIPA vs. VHT

IPAs	AMIPA	AMIPA	VHT	VHT	Performance
	(ms)	(ms/msg)	(ms)	(ms/msg)	(Rate %)
${10}^1$	12	0.0012	29	0.0029	58.62%
${10}^2$	147	0.0015	738	0.0074	80.08%
${10}^3$	1,895	0.0019	19,867	0.0199	90.46%
${10}^4$	24,604	0.0025	328,975	0.0329	92.52%
${10}^5$	27,4681	0.0027	5,665,592	0.0566	95.15%

Table 2. TABLE II: Metrics Comparison of AMIPA vs. VHT.

Metric	AMIPA	VHT	Performance Rate
CC	5.2	12.6	58.73%
MHF (%)	63.87	35.62	79.30%
AHF (%)	95.32	89.12	6.95%
ILF + EIF (FP)	17	17	0%
EI + EO + EQ (FP)	15	23	34.78%
TFP (FP)	182	345	47.24%
EPD (person/day)	391.30	741.75	47.24%
CBO	3.44	3.78	8.99%
CF (%)	12.76	28.43	55.11%
LCOM	0.63	2.84	77.81%

Full text

I Introduction

The advances verified in areas such as natural language processing, semantic web, machine learning, and artificial intelligence, combined with the huge amount of available information made accessible by the Internet, has enabled the creation of Intelligent Personal Assistants (IPAs). IPAs are mobile, autonomous, and software agents capable to perform tasks or services on behalf of humans [1], from answering general questions to recommend restaurants nearby, hear incoming messages and notifications, get directions, process automated subscription content like weather updates, and customize communications like receipts, shipping notifications, and live automated messages, just to name a few. As smartphones computational capabilities increase, IPAs such as Apple Siri, Google Now, Amazon Alexa, Microsoft Cortana, Facebook M, etc. aim to radically disrupt the way people search and consume information on the internet, as well as the manner they communicate and interact with the world. A relatively new trend in the development of IPAs is to allow the agent to dwell not only in the user’s smartphone but also in other devices such as wearables, tablets, personal computers, cars, household appliances, etc., and interact with the surrounding environment. This process can be seen as natural interaction among people, environment, and machines, creating a scenario of ubiquitous computing, that is, a scenario where technology is so integrated with users that they are unaware of the existence of the technological functions that surround them [2]. This trend entails several architectural challenges that must be addressed in order to create reliable, efficient, engaging, real-time, and ubiquitous IPAs. Most of these architectural challenges are intrinsically related to architecturally significant requirements such as high-performance, low latency, interoperability, scalability, extensibility, ubiquity, concurrency, mobility, support to reasoning and cognitive services, among others. However, addressing these challenges is a very time consuming, tedious, and burdensome task for researchers and developers (r&d from now on), which divert them from their main goal: building IPA’s. A convenient solution for separating the underlying complexity of the architectural requirements from the IPA agent implementation is the inclusion of a middleware, however, there is a lack of available open-source IPA-oriented middlewares that fulfill all the aforementioned architectural requirements. Therefore, we propose an IPA architectural middleware called AMIPA (Architectural Middleware for building Intelligent Personal Assistants), which aims to simplify the development of high-performance, ubiquitous IPAs. This paper is organized as follows: Section III presents the motivation and related work. Section III describes the architecturally significant requirements for building the proposed architecture. Section IV discusses the architectural design considerations necessary for building the middleware. Section V addresses a concrete implementation of the architecture. Section VI presents the evaluation of our architectural middleware. Section VII summarizes the conclusions and addresses future directions for this work.

II Motivation and Related Work

II-A IPAs Architectures

The usage of IPAs was originally promoted by projects like the PAL Program [3] (Personalized Assistant that Learns) proposed by DARPA, with contributions from SRI (the predecessor of SIRI) and several other laboratories with CALO (Cognitive Assistant that Learns and Organizes) project [4]. The technological revolution has enabled the creation of new gadgets with higher computing capabilities and new features, such as smartphones. Some smartphones provide a personal digital assistant as one of their main features. Apple’s Siri [5], Google Now [6], and Microsoft’s Cortana [7] are examples, however, there are little insights about how their system architectures were conceived. IPAs can normally interact equally with other intelligent objects in the environment (human or machine) to obtain knowledge about different domains. These scenarios are referred to as MAS (multi-agent systems) [8]. MAS are composed of multiple heterogeneous interactive intelligent agents within an environment, enabling parallel processing inside the system and making it less prone to failures. Li and Chen [9] presented a middleware for an IPA based on MAS and case-based reasoning (CBR) for manufacturing. Their MAS middleware allows the creation of a collaborative environment among intelligent agents, decentralizing the processing in the system. In [10] is proposed an IPA to execute services on behalf of users, using natural language interfaces to process the users’ requests, and using OMAS as their multi-agent middleware [11]. OMAS was designed for building cognitive agents and provides three types of agents: service agents (SA), transfer agents (XA) and personal assistant agents (PA). These agents are organized around a single net local loop and share messages using broadcast mode (UDP). The communication between agents is P2P meaning that there is no central directory, a valuable feature for service-oriented architectures. In [12] is proposed an IPA that can be integrated into ubiquitous computing environments in an Internet of Things (IoT) context, by using the CoAP protocol that allows REST architectures to be used in IoT applications, and XMPP message protocol which allows entities to communicate inside a network. Some other approaches [13, 14] present some theoretical insights about important aspects to develop IPAs, such as natural language and speech interfaces, but they do not focus on aspects such as the architectural considerations necessary to develop those IPAs. On the other hand, there some other approaches that use well-defined MAS middlewares such as JADE [15] (Java Agent Development Framework, a FIPA-compliant multi-agent system) to develop and execute IPA agent-based applications. CoolAgent RS [16] is a context-aware IPA system implemented with JADE that allows contextual information to be freely distributed among agents so that the meaning of that information can be shared and understood. Though we initially consider JADE as a middleware for building IPAs, we realized some limitations of this framework, such as the lack of support for latest versions of Android-based devices (e.g, Android 6 and above) [17, 18] and some issues regarding performance [19]. The strongest middleware candidate we took into consideration to develop IPAs was VHT (Virtual Human Toolkit) [20]. VHT is a collection of modules, tools, and libraries designed to aid and support researchers with the creation of virtual human conversational assistants. Though its architecture define proper mechanisms for extensibility (through loosely coupled modules) it doesn’t scale very well because the latency footprint increases dramatically, there is no proper mechanism to manage the concurrency, and the messaging system is based on a pre-defined configuration of ActiveMQ that doesn’t allow to create customized communication patterns. We will discuss how VHT compares to AMIPA on Section VI.

II-B Cognitive Services (CS)

CS are a set of REST APIs based on machine learning algorithms intended to create the next generation of applications powered by AI. Microsoft Cognitive Services [21] enables its Bot Framework and the Cortana (IPA) Intelligent Suite to perform Facial and object Recognition (FR), Automatic Speech Recognition (ASR), Natural Language Understanding and Generation (NLU and NLG), Knowledge Base access (KB), and Semantic Searches (SS). IBM Watson [22] provides CS for intelligent searches, deployment of conversational chatbots and virtual agents (NLU/ASR), and visual recognition. And Google Actions [23] allows to extend Google Now IPA by providing hooks to conversational CS that trigger actions implemented by developers. The two most evident disadvantages of these approaches are: CS are tight to the IPA platform, e.g., Google actions are only accessed through Google Now IPA and cannot be invoked from other apps; and they cannot be easily replaced by other CS (from other providers), that is, r&d have to re-implement the whole set of CS when switching the provider.

III Requirements

In this section we describe the Architecturally Significant Requirements (ASReqs) that we took into consideration during the development of AMIPA. According to [24], ASReqs are requirements that: (1) play an important role in determining the architecture; (2) need to be satisfied before the architecture is considered “stable”; and (3) affect the architecture in measurably identifiable ways. So AMIPA should address:

• •

ASR01 Interoperability: it should define common “contracts” of how heterogeneous components can exchange data.

• •

ASR02 Multi-session: it should allow managing multiple simultaneous sessions that share some kind of resources.

• •

ASR03 Scalability: it should perform efficiently under situations such as an expanding number of connected IPAs.

• •

ASR04 Low-latency: it should be latency-sensitive, that is, the speed at which the system responds to an asynchronous stream of independent and diverse events that result from interactive user input should not exceed 100 ms [25].

• •

ASR05 Abstraction: it should exhibit a gradual increase in the level of representation by replacing existing detailed information with information that emphasizes certain aspects important for r&d while hiding the least relevant aspects. As a result, the system’s complexity and size should decrease.

• •

ASR06 Distributed and decentralized: it should define flexible mechanisms to easily build robust distributed IPAs.

• •

ASR07 Fully portable: it should allow further implementation of cross-platform multi-language IPAs.

• •

ASR08 Concurrency: it should exhibit at least the following features proposed by [26]: (a) process creation/destruction should be a lightweight operation; (b) asynchronous message passing is the only way for processes to interact; (c) copying message-passing semantics (share-nothing concurrency); (d) processes are strongly isolated; and (e) remote processes appear largely the same as local processes.

• •

ASR09 Pluggability/Extensibility: it should provide mechanisms to make IPAs more modular, customizable, and easily extensible. It must allow to extend the functionality of the IPA by easily adding new features and components.

• •

ASR10 Mobility: components should be able to run code that is migrated from one machine to another.

• •

ASR11 Event-driven and Service-oriented: it should allow service management and communication through a loosely-coupled asynchronous messaging mechanism.

• •

ASR12 Ubiquity: it should provide mechanisms to run the middleware in a wide range of devices (e.g., phones, tablets, wearables, etc.) anytime, anywhere and using any format.

• •

ASR13 Reasoninga and Orchestration: it should reason over the data collected by different devices and orchestrate the interaction between services and components.

• •

ASR14 Cognitive Services: it should define reusable contracts for connecting CS that can be replaced any time through a lightweight integration mechanism.

We have identified the most critical requirements to create our architectural middleware, however, there are so many other requirements that we are not covering for now (e.g, security, privacy, maintainability, etc.) but expect to do it in the future.

IV Architectural Model

AMIPA’s architecture follows the basic principles of a Clean Architecture [27] guaranteeing better separation of concerns and better modularization of the IPA project. In a nutshell, the Clean Architecture defines concentric circles that represent different areas of software, where the outer circles are mechanisms and the inner circles are policies. The overriding rule that makes this architecture work is the Dependency Rule which says that source code dependencies can only point inwards, nothing in an inner circle can know anything at all about something in an outer circle. Following this rule, AMIPA’s layers can only point from the upper layers towards the bottom layers and not the contrary. Figure 1 depicts a high-level architecture for AMIPA with four identifiable macro layers: Communication, Decision Making, Management, and Cross-Cutting. Though the model resembles a monolithic architecture in the sense that the design is self-contained and their components seem to be interdependent rather than loosely coupled, it is not, this is only a high-level representation of the system’s architecture, which is appropriate for our purpose of capturing the vocabulary. On the contrary, AMIPA is inspired by the microservices architectural style. According to [28, 29, 30]: microservices is an architectural style that breaks an application down into smaller, cohesive, independent, loosely coupled, elastic, resilient, and scalable collection of services, each running in its own process and communicating with lightweight mechanisms. AMIPA is not purely oriented to microservices, only some layers support this style, as we describe later. In software development, architecture and middleware are different concepts, while a software architecture define the system’s high-level behaviors such as components, connectors, communication rules, connection rules, etc.; middleware provides an implementation, generally based on services, that sits between an application and the OS or the programming language, and where the architectural concerns are secondary. According to [31], the relationship between architecture and middleware and their respective shortcomings suggest the possibility of coupling architecture modeling and analysis approaches with middleware technologies in order to get the best of both worlds. Next, we describe the structural and behavioral viewpoints of AMIPA and some architectural views. The concerns to be addressed were described in section III and stakeholders for all viewpoints is the r&d team.

IV-1 Structural Viewpoint

AMIPA components communicate with each other through architecture-level software connectors that are implemented later using a middleware. This approach preserves the properties of the architecture-level connectors while leveraging the beneficial capabilities of the underlying middleware. Listing 1 presents an excerpt of AMIPA’s architectural description using ACME-ADL [32]. This oversimplified description will be enough to identify the main architectural elements of AMIPA. Let’s focus on the most relevant components of this architecture description. There are six components, one represents the client (i.e., a smartphone, smartwatch, tablet, laptop, etc.), two are in charge of managing the user sessions (Session and SessionManager), and the other three are responsible for decision making (ProcessOrchestrator, PluggableComponent, and CognitiveService), as we will discuss in detail in further sections. There are also three main connectors in charge of linking different components as described in the Attachments block: the Broker connector allows the IPA-client component to connect to both SessionManager and Session components; the Blackboard connector mediates the communication between a Session component and a ProcessOrchestrator component, and between the ProcessOrchestrator and a set of PluggableComponents; and the CommunicationController connector, which supports multiple communication protocols, allows the communication between a PluggableComponent and a CognitiveService. Finally, there are defined two architectural constraints in terms of Heuristics (a Heuristic constraint is taken to be a rule that should be observed, but may be selectively violated), one for low-latency and another for scalability. The first heuristic states that the total latency (the sum of all the latencies for the decision making components) should ideally not exceed 100 ms, as proposed in section III. The second heuristic says that when the system scales up to one million sessions, it should still keep the latencies in the range of 100-120 ms (i.e, the performance should not be affected when scaling the system up).

IV-2 Communication Viewpoint (ASR01, ASR05, ASR06, ASR07, ASR10, ASR11, ASR12)

Messaging: We propose an extension of the Asynchronous Majordomo Specification [33], a Service-Oriented Reliable Queuing pattern that defines a dialog between a set of client applications (those issuing requests, e.g., an IPA client on user’s phone), one or multiple broker devices (for routing, service discovery, task distribution, etc.), and a pool of worker applications (those processing the requests which can register for specific services (e.g., an ASR service), as depicted in figure 2. The DEALER and ROUTER patterns allow AMIPA to send multiple requests and multiple replies simultaneously and asynchronously, which means that workers are not passive modules that only listen for IPA client requests, but active and autonomous processes that can proactively communicate with IPA clients anytime. Blackboards use a PUB-SUB pattern to broadcast messages to specific pluggable components. IPA apps can communicate with each other either directly through a PAIR-PAIR pattern or indirectly through a broker. It is worth mentioning that all communication patterns in this architecture (i.e., ROUTER, DEALER, PAIR, PUB/SUB) are design-level constructs that realize architecture-level connectors, meaning that they define common contracts that can be further realized using different AMIPA implementations. This layer supports communication between co-located services trough INPROC protocol (thread to thread within a single process), IPC protocol (messages between local processes), TCP protocol (data exchange between IPA applications), and RPC (remote procedure calls that allow cognitive services to collaborate). The messaging layer is based on the following microservices patterns [30]: Messaging, Domain-specific Protocol, and Remote Procedure Invocation patterns for inter-service communication, and Circuit Breaker pattern for reliably handling partial failure.

Blackboard System: A blackboard system is better explained through the following metaphor [34]: a group of specialists work cooperatively to solve a problem by watching the blackboard and looking for an opportunity to apply their expertise to the developing solution. Such opportunities arise when an event occurs (a change is made to the blackboard) that enables the specialist to act. This process continues until the problem has been solved. AMIPA’s blackboard system is composed of three main modules: (1) Pluggable Components (or Specialist Knowledge Sources according to the metaphor), independent modules that contain the knowledge needed to solve the problem and can be widely diverse in representation and inference techniques; (2) a global working memory (the blackboard) containing input data, partial solutions, and other data that are in various problem-solving states; and (3) a Process Orchestrator (the control module in the metaphor) that keeps problem solving on track to insure that all crucial aspects of the problem are receiving attention, and to balance the stated importance of different specialist’s contributions. Pluggable Components (PC) subscribe to specific messages with the blackboard, so one PC may subscribe to multiple messages and multiple PCs may subscribe to the same message. On the other hand, the Process Orchestrator (PO) intercepts all the messages broadcasted by the blackboard. The blackboard receives messages from two sources: from external components (e.g., IPA apps, remote services, etc.) by using the underlying communication layer, and from internal components through an event-driven mechanism (i.e., PCs that post a partial solution of the current problem). When a new message is received, the blackboard notifies all the subscribers about that specific message, and then subscribers may extract the data that come wrapped in a BlackboardEvent object. Although multiple PCs may be activated by the same message, it is the PO who decides which PC can post to the blackboard. The Blackboard define a set of connectors that allow heterogeneous PCs to interact with each others through message-passing mechanisms. This layer uses the Messaging pattern of the microservices style.

IV-3 Concurrency Viewpoint (ASR03, ASR04, ASR08)

Processes that runs on single threads do faster when compared to multi-threaded processing, because it involves no context switching and no synchronization/locking. Therefore, in order to build a really efficient concurrency model, AMIPA does not use mutexes, locks nor semaphores meant to orchestrate the parallel processing. Instead, each object lives in its own thread, and threads do not share objects and only communicate through an inter-process protocol.

IV-4 Pluggability and Extensibility Viewpoint (ASR03, ASR05, ASR09, ASR14)

Pluggability is an essential feature in AMIPA, which allows r&d team to build IPAs that are modular, customizable, and easily extensible. In AMIPA, Pluggable Components (PC) are organized around IPA capabilities, that is, they encapsulate the interaction with self-contained cognitive services such as ASR, NLU, NLG, KB, etc. that may run either locally or remotely. When running locally and within the same process boundary, PCs behave as components (i.e, units of software that are independently replaceable and upgradeable [29]) that communicate through inexpensive in-process calls, whereas when crossing process boundaries, they behave as (micro)services that use well-defined remote interfaces to exchange information among distributed components using a mechanism such as a web service request or remote procedure call. Implementing a PC as a component or a service is left at r&d team discretion and depends on IPA’s context. In Figure 3 is depicted a view that illustrates the semantics of the Pluggability and Extensibility Viewpoint. The abstract class PluggableComponent exposes some common functionality, so r&d can then link against AMIPA in their applications and include the appropriate implementation by extending this abstract class. It is worth mentioning that the PluggableComponent class is the first-class construct of AMIPA, it defines a set of functions and mechanisms for registering callback functions for different types of messages, defines lifecycle methods for starting-up and shutting-down the component/service, and implements both the Pluggable and BlackboardListener interfaces which define the contracts of how PCs should behave when using either a direct-invocation or an event-driven approach. In the former approach, r&d are responsible for calling each component in the desired order (synchronously or asynchronously) by invoking the execute() method through a ProcessOrchestrator. On the other hand, for the latter approach, the PC’s onEvent() method is automatically triggered each time the blackboard is modified (e.g., insertion and deletion of elements). Moreover, in order to simplify the implementation of a PC, subclasses of PluggableComponent may use annotations for both subscribing to specific messages (BlackboardSubscription annotation) and defining the PC state (StateType annotation). More specifically, the behavior of a PC is determined by any of these three types of states: (1) Stateful PCs keep a representation of user’s state, so a new PC is instantiated each time a session is created for each user; (2) Stateless PCs do not keep the user’s state, so they are instantiated as singletons; and (3) Pool PCs that can create a fixed number of instances that split the work among them through a load-balancing algorithm. Additionally, the Module class defines high-level abstraction methods for the runtime creation, configuration and binding of a set of PCs. Now, the PluginModule class exposes methods to register different kind of PCs and orchestrators in an internal registry (for service discovery). Equally important is the PluginFactory class, which dynamically instantiates specific PCs contained in the PluginModule’s registry by using reflection and dependency injection. If a PC behaves as a local component then it only has to extend the PluggableComponent class, as CustomizedComponent class does, otherwise, if it behaves as a remote service, it registers with the PluginModule class and an ExternalComponent instance is automatically created to serve as a proxy between AMIPA and the remote service. The ExternalComponent class implements the ResponseListener interface, so when asynchronous responses from remote services are received by AMIPA, then the callback process() is invoked. The idea behind this viewpoint is that r&d can build high-level IPA functionality on top of PCs without having to worry about where their PCs will be running on, that is, it is completely transparent for them whether PCs are running locally or as remote services.

IV-5 Orchestration and Reasoning Viewpoint (ASR05, ASR13, ASR14)

Process Orchestrators are in charge of arbitrating the interactions and dynamics among PCs. As we discussed on Section IV-4, AMIPA’s orchestrators provide two types of control of execution: direct-invocation and event-driven. When using an event-driven approach, the execution of components is mostly based on events that are triggered by the blackboard system; then events are intercepted by the orchestrator, which will decide if any kind of data transformation is needed; and finally, these events are delivered by the orchestrator to the corresponding PC. Conversely, the direct-invocation allows r&d to explicitly define the way that components may interact with each other. For that purpose, AMIPA provides two mechanisms to define the control of execution: a workflow controller (decideWithWorkflow() method) and a rule-based system (decideWithRules() method). For the former mechanism, r&d can design interaction workflows using Petri net-like semantics, that is, directed bipartite graphs where nodes are PCs and edges are triggering conditions that define the transitions among those PCs. For the latter mechanism, r&d can create production rules expressed as IF-THEN structures, where conditions correspond to a set of premises in the blackboard that must be true, and actions are execution triggers such as posting new premises on the blackboard, invoking one or several components, etc. Listing 2 presents a rule-based orchestration script that controls the interaction among 7 PCs (and therefore Cognitive Services) for a user-IPA conversational interaction. In a nutshell, there are at least two main scenarios that the orchestration process can cover: 1) an ideal scenario where user speaks, the voice is recorded and processed by ASR, then NLU finds a corresponding user-intent for user’s utterance, next the Dialogue Manager (DM) generates a system-intent (a system action), then NLG generates a system utterance, and finally TTS (Text-To-Speech) converts the text into a spoken voice output (rule sequence: [01, 02, 03, 04, 05]); and 2) an exceptional scenario where NLU cannot match the user’s utterance with a user-intent (i.e., confidence value is low), so the system asks the user for clarification, next if the system cannot find a proper user-intent yet then it passes the user utterance to the Question&Answering component (QA), which in turn if does not find an answer then passes the user utterance to a chatbot powered by crowdsourcing (CCS), and finally, TTS sinthesizes the CCS’ output (rule sequence: [01, 02, 03, 05, 01, 02, 03, 06, 07]). The reasoning process is not only performed by the orchestrator but also by Cognitive Services that can use diverse kind of reasoning mechanisms depending on its own nature (e.g., a Semantic Search CS typically uses semantic inference to extract new facts from a KB).

IV-6 Dependency Resolution and Service Discovery Viewpoint (ASR05, ASR09, ASR11)

In order to support pluggability and flexibility features, AMIPA uses two common design patterns that assemble components from different sources into a cohesive application: a Resource Locator and a Dependency Injector. Both components decouple system implementation from its dependencies, so these dependencies can be replaced or updated with no change to the system implementation [35]. The Resource Locator is a singleton registry that contains references to different kind of AMIPA’s resources (e.g., components, remote services, databases, etc.) and encapsulates the logic that locates them. On the other hand, the Dependency Injector removes internal dependencies from AMIPA by allowing dependent objects to be injected into the classes/methods by an external caller. The Dependency Injector provides automatic instantiation and lifecycle management of classes the developer registers such as pluggable components, orchestrators, sessions, etc. Both the Resource Locator and the Dependency Injector are mechanisms for flexibility that maximizes dependency on interfaces while minimizing dependency on specific implementation. Consequently, AMIPA can support “pluggable” implementation classes that are used depending on the circumstances. The components in this viewpoint are inspired by the microservice chasis, externalized config, log aggregation, service registry, and distributed tracing microservice patterns [30].

IV-7 Session Management Viewpoint (ASR02, ASR03, ASR12)

This viewpoint focuses on the “multi-user”, “session-sharing” and “cross-IPA task coordination” capabilities of AMIPA. The multi-user capability is possible thanks to the concurrency and extensibility properties of AMIPA’s underlying layers, which allow the system to be easily scaled up to hundred thousands IPAs instances with no extra development effort and minimal latency footprint. The session-sharing and the cross-IPA task coordination capabilities are accomplished by two fundamental components: Session and SessionManager. There exist a unique session per user/IPA that carries out the following tasks: (1) it keeps the state of user’s interaction; (2) it controls session inactivity timeouts and decides whether to automatically reconnect itself; (3) it processes all messages sent from different user’s devices and then passes them to the corresponding process orchestrator; and (4) it provides a mechanism to guarantee that multiple ubiquitous user devices communicate through the same session so they can share information and collaborate in a distributed, inexpensive and robust fashion, supporting this way the session-sharing capability of the system. On the other hand, the SessionManger is responsible for: (1) handling the entire life-cycle of sessions (i.e., create, destroy, pause, resume, reconnect, etc.); (2) moderating the interaction among multiple IPAs so they can share information and collaborate to perform tasks, supporting this way the cross-IPA task coordination capability; and (3) controlling the interaction of multiple instances of AMIPA, that is, developers may define a master/slave configuration where a master AMIPA instance manage all the IPAs sessions, whereas one or multiple slave AMIPA instances may be deployed on a distributed environment to perform specific computations and therefore avoid overloading the master instance. Figure 4 shows an example of how multiple IPAs, co-existing in multiple ubiquitous devices, may interact with each others. Assume there are two users, Alice and Bob, who live in the same apartment. Bob’s IPA (IPA-1) co-exists in two devices, a phone and a tablet, and a new session is created for IPA-1 when Bob connects to his phone. When subsequent IPA-1 instances are connected (e.g., Bob’s tablet IPA) they are bound to the existing session (steps 1-7 in Figure 4). Then, Bob wants to check the weather app on his phone, which requires his location to filter the forecasting results, however, the location option is turned-off on his phone but turned-on on his tablet. Thus, the weather PC running on the IPA-1 phone instance sends a request through the SessionManager to the location PC running on the IPA-1 tablet instance, which in turn finds out Bob’s current location and returns this value to the weather PC on the phone. All this interaction among IPA instances is completely transparent for Bob (steps 8-18). After some time, Alice opens her IPA on her phone, and a new session is created (IPA-2) when her IPA connects to AMIPA. Alice and Bob need to coordinate who will do the groceries and what needs to be shopped. IPA-1 shares Bob’s location, who is closer to a grocery store, however, it is Alice who has the shopping list, so IPA-2 shares the list with IPA-1 and both IPAs (i.e., their process orchestrators) come to the conclusion that Bob should do the shopping. All this coordination and resource sharing is done through the interaction between Sessions, services and the SessionManager (steps 19-30).

V Implementation

In this section, we provide some details of AMIPA’s middleware implementation111See our GitHub repo: https://github.com/ojrlopez27/multiuser-framework, that is, the prefabricated structure that deals primarily with scaffolding, meaning that it provides pre-built modules that are easy to use/extend and, therefore, releasing r&d from having to create them from scratch. We decided to implement the concurrency and messaging layers by using ZeroMQ (ZMQ) [36], a high-performance asynchronous messaging library aimed at use in distributed and concurrent applications with minimal latency footprint. There are some bench-markings [36, 37] that demonstrate why ZMQ performs significantly better than other messaging/concurrency frameworks. The selection of ZMQ also aligns with the microservices principle of building smart endpoints and dump pipes [29], that is, messaging should be implemented over a lightweight message bus, where the infrastructure chosen should be typically dumb (dumb as in acts as a message router only, and implementations such as ZeroMQ fits well because they don’t do much more than provide a reliable asynchronous fabric) while the smarts still live in the endpoints that are producing and consuming messages: in the services. Furthermore, ZMQ enabled us to address several non-functional requirements such as: (a) it allows a wide range of ubiquitous devices (e.g., phones, tablets, raspberry pi’s, Microcontrollers, etc.) to connect to AMIPA thanks to it defines a socket-based API that supports different kind of unicast and multicast protocols (INPROC, IPC, TCP, TIPC, PGM, EPGM, NORM, SOCKS5) as well as M2M communication; (b) its non-blocking nature dramatically minimizes message delivery latency by dispatching, delivering and queuing messages in parallel to the regular processing performed by the system; (c) thanks to ZMQ, AMIPA is highly inter-operable, portable, cross-language and cross-platform, so it is possible to connect either a 32KB embedded chip or a z/OS mainframe running IBM dialects of Unix, using any of the most known programming languages [38]. According to [39, 40], even IoT applications can get access to ubiquitous data in rich sensing pervasive environments by using a M2M messaging mechanism based on ZMQ; (d) it uses CurveZMQ, a protocol for secure messaging across the Internet; and (e) since ZMQ uses the same API for inter-machine, inter-process and inter-thread communication, AMIPA can scale up seamlessly from a one-process-per-core scenario to a grid fabric of thousands of machines. Based on the foregoing, the proposed ZMQ-based communication topology (see Figure 2) enables AMIPA to scale to large numbers (thousands) of workers (services) and clients (IPA apps), where a single broker thread can switch several million messages per second, and multi-threaded implementations (offering multiple virtual brokers, each on its own port) can scale to as many cores as required. We implemented AMIPA’s middleware on both Java and Python programming languages and defined a set of custom annotations (in Java) and decorators (in Python), that serve as syntactic metadata that ease and simplify the implementation of system modules and components. Pluggable Components can communicate with each others using two mechanisms: an even-driven approach based on ZMQ and a mobile-code approach based on Mobility-RPC [41], a Java library that allows pluggable components to autonomously migrate from one host to a different host during their execution, and spontaneously update IPAs (running on ubiquitous devices) with new functionality or context-dependent program code. For message formatting we used protobuf [42], a Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data. The Dependency Resolution layer was implemented using Google Guice [43], an open source generic framework for dependency injection using Java annotations, which also supports AOP (Aspect Oriented Programming). Using guice, r&d can easily and quickly extend AMIPA’s features by alleviating the need for factories and promoting the creation of injectors and method interceptors that efficiently and unobtrusively perform cross-cutting logic (e.g., logging, error handling, etc.). We also used Google Guava [44], a set of core libraries and API’s to control the sessions lifecyle and build the orchestrator’s workflows, among other things. The former functionality is carried out by a ServiceManager component that monitors session’s state transitions, and the latter uses guava’s graphs, a library for modeling graph-structured data where nodes correspond to pluggable components and the edges between them correspond to conditional transitions. For the orchestrator’s rule-based control of execution mechanism, we used the Java Rule Engine API, a light and fast rule engine in compliance with the JSR94 specification. Figure 5 illustrates an example of how AMIPA can be deployed on a distributed environment. AMIPA provides a server library tha can be deployed on one or multiple host computers (e.g, Server 1 and Server 2) and a lightweight client library that can be deployed in a wide range of ubiquitous devices (e.g, smartphones, tablets, smartwatches, TVs, thermostats, etc.). If the user has installed multiple IPA instances on different devices (let’s say IPA-1 which is installed in a smartphone, a tablet, and a thermostat) then they will share the same session (session-1) to send and receive messages to/from different components and services. While the server library support all layers, the client library only supports the concurrency, messaging, pluggable components, cross-cutting and (partially) session management layers.

VI Evaluation

In this section we present an empirical metric-based comparison between AMIPA and VHT (as we mentioned in Section II, VHT was our initial candidate as an IPA middleware before deciding to build our own middleware). To this purpose, we compared the implementation of an IPA using both middlewares. In a nutshell, the IPA was a conversational agent that generated different kind of recommendations (e.g., movies, news, etc.) while keeping user’s engagement through social dialogue. The IPA had access to 7 different CS deployed on 3 remote servers: Google ASR, Multisense for non-verbal behavior recognition [45], Microsoft NLU, NLG, Dialogue Manager [46], Social Intention Recognition [47], and a Social Reasoner for making decisions about conversational strategies [48]).

VI-1 Measuring Latency and Performance

In order to measure the latency of both AMIPA and VHT, we conducted an experiment222Servers hardware configuration: EC2 t2.2xlarge AWS instance, Canonical, Ubuntu, 16.04 LTS, 8 cores, amd64 xenial image build on 2017-04-14 where we measured the round trip time (in milliseconds) for 1,000 messages to flow through all 7 CS and for the reply to be received, using $10^{1}$, $10^{2}$, $10^{3}$, $10^{4}$, and $10^{4}$ IPA clients. Each experiment was performed 10 times, and then their harmonic mean (which mitigates the impact of large outliers) was calculated. Finally, we calculated the speed in milliseconds per message sent (ms/msg) and the performance improvement rate between both approaches, which was estimated using the equation: $(VHT-\textsc{AMIPA})/VHT$. The results are shown in Table I. In general, AMIPA’s performance surpassed VHT’s performance in a high rate (from 58.62% to 95.15%). It is worth noting that, when scaling from 10 to $10^{5}$ IPAS, AMIPA’s latency had very little footprint in comparison with VHT, going from $12\times 10^{4}$ milliseconds per message (with 10 IPAs) to $27\times 10^{4}$ ms/msg (with $10^{5}$ IPAs), while VHT went from $29\times 10^{4}$ to $566\times 10^{4}$ ms/msg. We also ran a one-way ANOVA test to analyze the difference among the means of the 2 groups of data, and given that the obtained p-value was less than the significance level ($p=0.0049\leq 0.5$) then we could conclude that there was a statistically significant difference between AMIPA and VHT results. The performance experiments revealed a clear correlation between the number of IPAs and the performance rate of AMIPA vs. VHT: the more messages were sent the better AMIPA performed in comparison with VHT. This is due to the optimization of AMIPA’s concurrency and communication modules: we drastically reduced the message-passing latency by using low-level thread manipulation and zero-copy techniques instead of using the typical concurrency model based on locks, mutexes, and semaphores.

VI-2 Measuring Abstraction, Pluggability and Extensibility

There exist different approaches to measure software abstraction, but the general consensus is that the more abstract an application is, the less complex and effort-consuming its development is [49, 50]. So our initial hypothesis was: AMIPA should significantly decreased the complexity, size and effort to build the proposed scenario in comparison with VHT. For complexity we used the Cyclomatic Complexity metric (CC), defined as the number of linearly independent paths within a graph that represents the source code flows, and is calculated as: $M=E-N+2P$, where $E$ is the number of edges of the graph, $N$ is the number of nodes, and $P$ is the number of connected components [51]. We used a software metrics tool called MetricsReloaded [52] and collected the results shown in Table II. We deduced that both AMIPA and the VHT have low complexity (according to [51], high complexity is over 15), however, the improvement rate demonstrated that AMIPA reduces the complexity on $\cong 59\%$ in comparison with VHT. We also measured the level of abstraction in terms of the minimum amount of implementation details that were exposed to the developer without loosing information content. To this purpose, we used two MOOD metrics (Metrics for Object Oriented Design) [53]: the Method Hiding Factor (MHF) and the Attribute Hiding Factor (AHF) which were calculated across all classes in the system. It is worth noting that while AMIPA improved method hiding in a $\cong 79\%$, it only improved attribute hiding in a $\cong 7\%$, which is not particularly a significant difference between the two middlewares. In terms of size metric, we used Function Points (FP), a widely accepted industry standard (ISO/IEC 20926:2009) for functional sizing. FP are estimated in terms of the amount and functional complexity of data functions ( i.e., Internal Logical Files ILF and External Interface Files EIF); and transaction functions (i.e., External Inputs EI, External Outputs EO, and External Inquiries EQ). Based on the results, we observed that the main difference between both middlewares was an increment of 8 FP’s (for transaction functions) in VHT, which means that AMIPA reduced the amount of transactional functionality to be developed in $\cong 35\%$. The Total Function Point measure (TFP) reflects that AMIPA is $\cong 47\%$ smaller in functionality (this means that less functionality has to be implemented to meet the same system’s requirements) in comparison to VHT, which in turn represents a drop in effort in the same proportions. The Effort Person/Day (EPD) estimation is computed as $EPD=TFP/DR*DPM$, where DR is the delivery rate (in average, a java developer can implement 10 FP’s per month [54]) and DPM is days per person-month (21.5 business days per month). EPD can be better understood in terms of time and number of persons required to develop the IPA, let’s say we have a team of 5 persons, using VHT it would take 148.4 days (741.75/5) while using AMIPA would take 78.2 days (391.3/5), which means a reduction of $\cong 47\%$ of the required effort when using our approach. However, our empirical study revealed that development effort can be reduced more than 60% when using AMIPA, and this discrepancy with EPD may be due to unconsidered elements during the estimation. Measuring pluggability and extensibility can be achieved by calculating the amount of coupling and cohesion in the system: the more loosely-coupled and high-cohesive the system is the more pluggable and extensible. We used 3 different metrics [55]: Coupling Between Object classes (CBO), Lack of Cohesion Methods (LCOM), and Coupling Factor (CF). In CBO, two classes are coupled when methods declared in one class use methods or instance variables of the other class; LCOM defines the number of different methods within a class that reference a given instance variable; and CF is the ratio of the maximum possible number of couplings in the system to the actual number of couplings not imputable to inheritance. The results are presented in Table II. According to [55] a CBO $>$ 14 is too high, therefore both approaches have loose coupling, however, using CBO we demonstrated that AMIPA only reduced coupling in $\cong 6\%$. We obtained a more significant difference between both approaches when using CF, this time, AMIPA reduced the coupling at a rate of $\cong 55\%$. Our approach lessens coupling due to the use of dependency injection patterns and event-driven communication instead of direct invocations to classes. In LCOM metric, a result equals to 0 indicates a cohesive class, higher than 0 indicates that the class needs or can be split into two or more classes. Therefore, the results suggest that both approaches have certain lack of cohesion, however, AMIPA seems to be $\cong 78\%$ more cohesive than VHT. The main reason of this improvement is due to the architectural class decomposition into high-cohesive classes with a clear separation of concerns (e.g., communication, service discovery, components, orchestrators, etc.). Nevertheless, the presented metrics are, in some way, biased and extremely sensible to the style of programming.

VII Conclusions and Future Work

In this paper, we have presented an architectural middleware solution called AMIPA that supports the construction of high-performance, distributed, ubiquitous, and scalable Intelligent Personal Assistants. The solution consisted of two parts: 1) a universal architecture that exposes the most critical design aspects, by using different levels of abstraction and design patterns, that can be further concretized using any programming language; and 2) a concrete high-performance, high-scalable middleware implementation using a Java-based approach. AMIPA hides the underlying complexity of the environment; insulates the IPA applications from explicit protocol handling, disjoint memories, data replication, network faults, and parallelism; masks the heterogeneity of operating systems, programming languages, networking technologies, and distributed ubiquitous devices to facilitate IPA programming and management; and provides a better abstraction that allows researchers and developers to build IPAs with less code and less programming errors. Our future work will focus on several aspects: we are planning to make AMIPA completely opensource so the developer and research community can take advantage of its powerfulness. Next steps will be to use software standards for modularity, extensibility and pluggability such as OSGi (Open Service Gateway initiative) or a fully-oriented microservice architecture style. Also, we have identified the need of creating a semantic layer on top of middleware to improve the service discovery process, provide more accurate and relevant information to higher-level layers, and make inferences about user’s context.

Acknowledgment

This research was supported in part by Yahoo! and Verizon through the CMU-Yahoo InMind project.