Tool Macgyvering: Tool Construction Using Geometric Reasoning

TL;DR

This paper introduces a formal framework for tool Macgyvering, focusing on constructing tools from available parts using geometric reasoning, validated through robotic experiments.

Contribution

It formalizes the tool Macgyvering problem, defines complexity levels, and presents a novel geometric reasoning algorithm for tool construction.

Findings

Successfully constructed three tools with a 7-DOF robot arm

Validated the effectiveness of the geometric reasoning approach

Established a formal problem domain for tool Macgyvering

Abstract

MacGyvering is defined as creating or repairing something in an inventive or improvised way by utilizing objects that are available at hand. In this paper, we explore a subset of Macgyvering problems involving tool construction, i.e., creating tools from parts available in the environment. We formalize the overall problem domain of tool Macgyvering, introducing three levels of complexity for tool construction and substitution problems, and presenting a novel computational framework aimed at solving one level of the tool Macgyvering problem, specifically contributing a novel algorithm for tool construction based on geometric reasoning. We validate our approach by constructing three tools using a 7-DOF robot arm.

Figures

Full text

I Introduction

Intelligence is often best expressed through creative problem solving - a skill that humans frequently depend on, especially in high-stress or time constrained scenarios. In the Apollo 13 incident of 1970, a carbon dioxide filter creatively constructed out of a sock, a plastic bag, book covers, and duct tape helped save the lives of the three astronauts on board [1]. Solving problems by constructing new tools from available objects is colloquially referred to as “Macgyvering”. The term originated from the popular TV series Macgyver, which featured a secret service agent who used common objects available to him in the absence of required tools in order to escape difficult situations. Unlike humans, today’s robots are limited to using predefined available tools, although some prior work has explored tool substitution as a means of problem solving and adaptation [2, 3, 4]. Our research focuses on tool construction, contributing a computational framework that enables a robot to construct, or Macgyver, tools out of parts available in the environment. To the best of our knowledge, this is the first work to demonstrate tool construction on a physical robot.

Tools are defined as objects that extend the physical influence of the agent [5], and the construction of tools to solve problems is a particularly interesting challenge for intelligent robotic systems because it demonstrates a more sophisticated level of intelligence than simple tool use. Existing work in psychology has shown that tool-making emerges notably later than tool use in children, and that physical tool making is preceded by the step of imagining some “canonical tool” suitable for the task [6]. In our work, we refer to this conceived tool as the reference tool. Given a 3D model of a reference tool, our system reasons about the geometric properties (e.g., shapes, sizes, attachment points) of available parts to construct a substitute tool (Fig. 1). Our work makes two contributions: 1) we formalize the overall problem domain of tool Macgyvering, introducing 3 levels of complexity for tool construction and substitution problems, and 2) we introduce a computational framework aimed at solving one level of the tool Macgyvering problem, specifically contributing a novel algorithm for tool construction based on geometric reasoning. We validate our approach by constructing three tools using a 7-DOF robot arm.

II Related Work

Tool-making and tool use have been widely studied in animals. Mammals, such as chimpanzees [7] and bonobos [8], and birds, such as rooks [9] and blue jays [5], have been shown to create and use simple tools to retrieve food. In [6], the authors identify two key aspects of tool-making: tool innovation and tool manufacturing. Tool innovation is the process of imagining the ideal tool required for a given task, and tool manufacturing is the process of physical transformation of materials/parts into a tool.

In the context of robotics, prior work in tool substitution [2, 3] has used visual reasoning for finding tool substitutes. Further, recent work has explored Macgyvering and the use of environmental objects for problem solving. Sarathy and Scheutz [10] proposed a theoretical formulation of Macgyvering; their work differs from ours in that it does not reason about visual/physical properties of objects in tool creation. In [11], the authors introduce techniques for reasoning about construction of functional structures for navigation. Further work explored use of environmental objects as simple machines [12, 13]. In [14], the authors focus on creation of novel tools using 3D printing. More recently, Choi et al. [15] extended the cognitive architecture ICARUS to support the creation and use of tools in abstract planning scenarios. Our work differs from these approaches in that we consider the geometric properties of tools, and demonstrate robot tool construction from diverse environmental objects, including physical validation of the construction.

One of the key design choices within our framework is the underlying representation used for tools. We adopt the tool representation proposed by Abelha et al. [2], who demonstrated the superior performance of Superquadrics (SQ) over other techniques in the context of object substitution. Superquadrics refer to the family of geometric shapes that includes quadrics, but allows for arbitrary powers instead of just power of two [16]. Fig. 2 shows an example of a SQ model of a hammer, where both the hammer head and handle are represented by a class of SQs called Superellipsoids. In our work, we show that the SQ representation of tools allows the robot to select and attach geometrically appropriate pieces when constructing new tools.

III Levels of Tool Macgyvering

The problems of tool use, tool substitution and tool construction are all related to the representation of tool affordances. Below, we first define and characterize tool affordances, and then show that complexity levels of the tool Macgyvering problem can be formalized based on the extent to which the tool affordance representation must be adapted.

III-A Macgyvering as a function of object affordances

Problems that are solved using tool construction and substitution are defined by their two primary constituents: a task goal $G$ and a solution affordance $S$. We define goals as persistent relationships between objects, such as “isAttached(painting, wall)”, and affordances as action possibilities available to the agent for a given object [17]. Computationally, affordances are defined as unique relationships between objects $(O)$, the actions those objects can be used for $(A)$, and the effects $(E)$ of applying the actions with the objects [18]. We use the tuple $S=(O,A,E)$ to denote an affordance solution for goal $G$.

Prior work on affordance representations [19], has defined object equivalence ($O^{eq}$) as occurring when the same action applied to two different objects generates an equivalent effect. Similarly, object-action equivalence ($OA^{eq}$) is defined as occurring when different actions applied to different objects result in equivalent effects. We introduce a third equivalence class, object-action-effect equivalence ($OAE^{eq}$), as occurring when different actions applied to different objects generate different effects, but that those effects accomplish the same task goal $G$. Table I presents an example of all three levels of equivalence for the task goal G=isAttached(board1, board2) and reference solution $S_{R}=(screwdriver,turn,tighten\_screw)$. $OAE^{eq}$ occurs when the goal of attaching the boards is achieved, but without using the original tool, action or effect, such as by tying the boards together with a rope instead of tightening the screw. Below, we show that tool Macgyvering problems can be broken down into three classes, based on which elements of the reference solution $S_{R}$ are assumed to remain unchanged.

III-B Macgyvering Levels

We define three levels of Macgyvering based on the above equivalence classes, and show how the relative complexity of tool-based problems can be assessed directly from the affordance equivalency representation. Each Macgyvering level consists of two variants based on whether reference objects are substituted or constructed. Tool substitution, denoted by the subscript S, refers to the case in which an existing tool can be used to replace a reference tool. Tool construction on the other hand, denoted by the subscript C, refers to the case in which no substitute for the reference tool exists, and a new tool must be constructed. The definitions below, also summarized in Table II, are in reference to a task with goal $G$ and a reference solution defined by $S_{R}=(O_{R},A_{R},E_{R})$.

Level 1: object equivalence solutions. The first level addresses tasks with solutions that can be achieved either through object substitution ($O_{S}^{eq}$) or object construction ($O_{C}^{eq}$) alone, where the newly found or created object shares the action and effect of the reference solution. Thus, the solution affordance $S$ for Level 1 is $(O_{S},A_{R},E_{R})$ or $(O_{C},A_{R},E_{R})$. As shown in Table I, in the case of $O_{S}^{eq}$ a knife can be used in place of a screwdriver to turn and tighten a screw, and in the case of $O_{C}^{eq}$, a clothespin and a coin can be combined to create a tool identical in function to the screwdriver.

Level 2: object-action equivalence solutions. The second level addresses tasks for which no viable object equivalence solution exists, and the goal is achieved by substituting, $OA_{S}^{eq}$, or constructing, $OA_{C}^{eq}$, a tool with a non-reference action where the object-action pair yields the same effect as the reference solution. Thus, for Level 2 $S=(O_{S},A_{S},E_{R})$ or $S=(O_{C},A_{C},E_{R})$, where $A_{S},A_{C}\neq A_{R}$. In our example of attaching two boards, in the case of $OA_{S}^{eq}$, a hammer can be used to push the screw in with a hitting action instead of using the screwdriver with a turn action. Both actions accomplish the effect of tightening the screw. In the case of $OA_{C}^{eq}$, the same hitting action would be used, but the hammer would need to be constructed, such as by combining a stick and a rock.

Level 3: object-action-effect equivalence solutions. The third level addresses tasks for which no viable object-action equivalence solution exists, and the goal $G$ is achieved by substituting, $OAE_{S}^{eq}$, or constructing, $OAE_{C}^{eq}$, a tool with a non-reference action and effect. For Level 3, $S=(O_{S},A_{S},E_{S})$ or $S=(O_{C},A_{C},E_{C})$, where $A_{S},A_{C}\neq A_{R}$ and $E_{S},E_{C}\neq E_{R}$. In our running example, in the case of $OAE_{S}^{eq}$ a rope can be used to tie two pieces of wood together, and in the case of $OAE_{C}^{eq}$ a rope would first be knitted out of strands of yarn and then used to tie. Both scenarios no longer accomplish the effect of tightening the screw, but accomplish the task goal of attaching the boards.

As can be seen from the examples, developing a robotic system that can solve the Macgyvering problem becomes progressively harder at each level as the reasoning requires a deeper understanding of the domain, desired task goal, the actions afforded by each object, and their resulting effects.

IV Framework for Level 1 Macgyvering

In this section, we present a computational framework for Level 1 Macgyvering. Fig. 3 presents an overview, with red lines denoting the tool substitution $O_{S}^{eq}$ pipeline, blue lines denoting the tool construction $O_{C}^{eq}$ pipeline, and purple denoting shared components.

Tool Substitution: The object substitution scenario follows the pipeline described by Abelha et al. [2], wherein the system is provided with a reference tool and a set of candidate tools. The reference and candidate tools are segmented into their constituent parts (hereafter, “tool components”), and the resulting components are fitted with SQs. Candidate tools are then scored and ranked based on the similarity of their components to the reference tool components. We further extend this work by incorporating a tool validation phase in which the robot physically tests that the substitution tool performs the necessary function; failure of this test causes lower ranked substitution candidates to be considered.

Tool Construction: In the tool construction scenario, the system is provided with a reference tool and a set of parts available for construction (hereafter, “candidate parts”). As before, the reference tool is segmented, then SQs are fitted to each reference tool component and candidate part. For the tool construction phase of the pipeline, we introduce a metric that takes into account the shape, size and possible configurations of the candidate parts and outputs a ranked list of possible tool constructions. The robot then uses the ranked list to construct candidate tools, which it validates on the target task.

Although only the tool construction pipeline serves as the main contribution of this paper, we present it alongside the tool substitution elements to highlight the synergy of our work with prior methods in the field. In the following section, we present the details of the tool construction methodology.

V Tool Construction $O_{C}^{eq}$

We formulate the tool construction problem as follows:

“Given a reference tool $R$, and a set $C$ of $n$ candidate parts, how can the robot reason about the geometric properties and attachment points of the available parts in order to construct a replacement tool that can be used to accomplish task goal $G$ using the same action and effect as the reference solution?”

We define attachment points as locations at which objects can be attached together. In general, attachments can result in fixed, revolute or prismatic connections [20] achieved through insertion, glueing, duct tape, etc. In this work, we apply our framework in two scenarios: i) when a predefined library exists that specifies the attachment points available for each candidate part, as in [21], and ii) when no attachment information is known.

Given $n$ candidate construction parts, the total space of configurations or permutations for the $n$ parts is ${}^{n}P_{2}+...+^{n}P_{n}$, assuming at least two and at most $n$ parts must be combined to construct the tool, resulting in a prohibitively large problem space that is combinatorial in the number of candidate parts. In this work, we make the simplifying assumption that the number of parts required to construct the tool is equal to the number of reference tool components111Most tools have two key components, grasp part and action part [22, 23], thus, given $m$ tool components, we obtain a state space of ${}^{n}P_{m}$. Below, we detail our tool construction approach.

V-A Segmentation, SQ Fitting and Part Score Computation

The tool construction pipeline begins with segmentation, which enables the system to identify the basic components of the reference tool and the candidate parts in the robot’s workspace. Our reference tool model is obtained from the ToolWeb dataset [23], and we use Triangulated Surface Mesh Segmentation222The implementation was provided by the CGAL library to segment the tool (e.g., into handle and head of a hammer). The resultant reference tool components are denoted as an ordered tuple $R=(r_{1},r_{2},...,r_{m})$. We use plane subtraction and Sample Consensus Segmentation (SAC)333The implementation was provided by the PCL library to identify the candidate parts available to the robot using RGB-D data from a camera mounted over the table. We denote the resulting candidate parts as $C=\{c_{1},c_{2},...,c_{n}\}$.

Given $R$ and $C$, we next fit SQ444the presented framework can also be extended to other representations, e.g., ESF and SHOT [3] models to each reference tool component and candidate part. Each SQ model has 13 parameters: 3 for scale in each dimension, 2 for shape variance, 3 for Euler angles, 2 for tapering parameters and 3 for the central point/mean. We use Levenberg-Marquardt optimization to find the best fit parameters for each object [23]. We then use Algorithm 1 for computing the part scores. As input, the algorithm is provided a list of importance weights $\lambda_{1-4}$, SQ parameters of parts and a list of tuples $T$ consisting of all possible candidate configurations generated through the permutation ${}^{n}P_{m}$ of candidate parts in $C$; ordering of parts within each tuple, determines pairwise matching between $c_{i}$ and $r_{i}$. The output of Algorithm 1 is a list of candidate builds $T^{*}$ sorted by a computed vector of error values $e_{const}\in E$, and a list of attachment points $Att$ to use for combining the parts. To compute $e_{const}\in E$, we consider the following four metrics:

• $\bullet$

per-component shape fit, based on absolute difference between the two shape SQ parameters (denoted by 2D vector $shape$) of candidate parts and corresponding reference components (line 4).

• $\bullet$

per-component size fit, based on of absolute difference between the 3 scaling parameters (denoted by 3D vector $scale$) of candidate parts and their corresponding reference tool components scaled to real-world size (line 5).

• $\bullet$

pairwise component proportionality fit, calculated based on the relative scale ratios (denoted by $rel$) between reference tool components and candidate parts (line 8) Example: hammer handle may be $3x$ longer than the head.

• $\bullet$

if attachment point information is known, an attachment score encoding the proximity of available attachment points to the target attachment locations (line 11, details in Section V-B) is computed for the different possible part configurations of $T_{i}$.

Each of the above metrics produces an error value indicating the magnitude of the deviation of the candidate part combinations from the reference tool. We then compute an aggregate error term $e_{const}$ as a weighted sum (line 12) of all four error terms using $\lambda$ importance weights555$\lambda$ terms were manually chosen in this work. The list of candidate part configurations are then sorted by their associated error value $e_{const}$, from lowest to highest.

V-B Attachments With and Without Known Points

The presented framework is able to construct tools both when a list of predefined attachment points is available, and when it is not (Algorithm 2). In both cases, the process begins by aligning the components of the candidate tool $T_{i}$ in a configuration consistent with $R$ (line 2); we use Principal Component Analysis (PCA) to orient part point clouds w.r.t. the reference tool, resulting in a set of alignments $T_{i}^{\prime}$. We approximate the intersections of the point clouds in each alignment by calculating the centroid of closest points between the point clouds (line 3). The resultant set of centroids, $P$, is the candidate list of attachments we want to make.

If no part attachment information is known, the set $P$ is used to guide the robot in constructing the tool. Specifically, the robot will attempt to attach parts at the locations specified by $P$, and then verify whether attachment was successful before proceeding (see Fig 4 and accompanying video666https://www.youtube.com/channel/UCxnm8iu1TS75YNXcAiI-nEw). This search process enables the robot to explore possible attachments until a successful part combination is found, however, leading to more tool construction attempts. Alternatively, if attachment information is provided, it is incorporated into $T_{i}$ ranking by computing an error score based on the Euclidean distance between points in $P$ and the closest attachment locations on each part $c_{j}$, in each alignment $t_{i}\in T_{i}^{\prime}$ (lines 5-9). The resulting error value, $e_{att}$, is combined with the other error metrics, described above, to rank candidate parts and configurations. If a part in $t_{i}$ is known to have no attachment points, $e^{T_{i}}_{att}=\infty$.

V-C Tool Validation with Manufacturing and Testing

Given the ranked list of potential tool constructions obtained above, the robot next constructs a tool by joining the components specified by the best-rated configuration using the attachment points (output of Algorithm 1). The robot then evaluates the constructed tool for its task suitability by applying the desired action on the tool. In this work, we assume that the robot is given the required action for each object, and that it can observe whether the tool succeeded. Alternatively, this information could be learned from demonstrations [24], including, if necessary, adapting the original action to fit the dimensions of the new tool [25, 26].

VI Experimental Validation

To validate our tool construction approach, we constructed three tools: a hammer, spoon and spatula. Each tool consisted of two components ($m=2$), and the robot was given $n=4$ candidate parts. Reference tool models were acquired from the ToolWeb dataset [23]. The manually-set weight parameters $\Lambda=\{1,1,5,5\}$ ($e_{shape},e_{scale},e_{ratio},e_{att}$) worked well across all three tasks. We use magnets as attachments for the parts, and seek to validate the performance of our approach both when attachment points are known and unknown.

Fig. 5 shows the robot’s workspace and a typical build-and-test cycle of tool building, demonstrated on the hammer. All three tasks, and their results, are summarized in Table III, including: the reference tool used, list of available candidate parts, the number of possible configurations of parts given the attachment points, the part combinations that were attempted and that succeeded (shaded, bold) for each build, along with their corresponding $e_{const}$ value, the number of attempts the robot required to build the tool, and the final constructed tool. Each tool design has a single working configuration, but tests different aspects of our approach. We first describe our results for the case of known attachments, and then unknown attachments. The avg. computation time for $e_{const}$ was only 15.44 seconds and we note other key insights in each case.

VI-A Tool Construction with Known Attachments

Hammer: Candidate parts for the hammer consisted of four wooden blocks of various dimensions, each with a single attachment point. The highest ranked candidate tool (lowest error) consisted of parts B and C, which most closely resemble a hammer shape. However, the resulting magnet connection was not strong enough to withstand the impact of the robot hitting a box during the tool validation phase and the tool broke. Subsequently, the robot attempted to combine the next ranked set of parts, B and D. The resulting tool passed testing because the curved shape of part D, while less closely resembling a real hammer, provided needed support during impact. This demonstrates the robot’s ability to validate the tool beyond only geometric fit.

Spoon: Candidate parts for the spoon consisted of two wooden handles and two scoops; one scoop had two attachment points and the other had zero. Using available attachment information, the robot correctly ignored the scoop with no magnets (which was a closer match by shape alone), and correctly chose to combine parts B and D to construct a spoon that successfully scooped beans out of a bowl. This demonstrates the robot’s ability to reason about 1) symmetric constructions (parts D+B ranked first and second due to symmetric attachments) and 2) absence of attachments.

Spatula: Candidate parts for the spatula consisted of three possible handles and one flat piece with two attachments. The robot first attempted to combine parts B and D, however, a successful connection failed due to polar orientation of the magnets. The robot recovered from this problem by successfully combining parts C and D, even though the resulting shape was less similar to a spatula than the original attempt. The resulting tool was successfully used to pick up a piece of plastic lettuce. This demonstrates the robot’s capability to 1) create tools that vastly deviate in terms of geometry but still accomplish the function, and 2) reason about multiple attachment configurations (part combinations D+C repeat for the different attachment configurations).

VI-B Tool Construction with Unknown Attachments

The results in Table III highlight several key differences in the construction process with and without known attachments. When the robot does not have knowledge of attachment points a priori, the construction process is guided by the candidate attachments in $P$ (Section V-B). As a result, the robot attempts multiple attachments per configuration. For example, the solution for the spoon was found using the fourth part combination (D+B) after 14 construction attempts. Thus, identifying a valid combination of parts using shape information alone is significantly more challenging than when attachment points are known. However, in all cases the robot successfully identifies a solution within the first 4 configurations (out of possible 12, 18 or 26), validating that the $e_{const}$ ranking is successful at guiding the search.

VII Conclusions and Future Work

In this work, we have contributed a novel computational framework for tool construction based on geometric reasoning, which we have demonstrated on the construction of three tools, both when part attachment information is known and unknown. In all cases, the robot efficiently constructed the tool, exploring only a small percentage of all possible part combinations. Also, we have introduced a formalization of tool Macgyvering, to guide future research in this area by defining varying levels of difficulty for this problem. Future work should address the following limitations of the current approach. Two key assumptions, the current limitation that the number of candidate parts must equal the number of tool components, and that $\Lambda$ parameters are hand-coded, should be relaxed. Further, as tool complexity is increased, metrics and heuristics should be explored that maintain the computational tractability of the search problem. Finally, the tool representation should be extended to include a broader range of properties, including material and density.

Acknowledgments

This work is supported in part by NSF IIS 1564080 and ONR N000141612835.