Object Placement on Cluttered Surfaces: A Nested Local Search Approach

TL;DR

This paper presents a novel nested local search method for planning collision-free object placements on cluttered surfaces, optimizing for minimal displacements without prior goal configuration knowledge.

Contribution

It introduces a multi-objective nested local search approach that efficiently computes object placements in cluttered environments without needing predefined goal states.

Findings

High computational efficiency demonstrated in experiments.

High success rate in finding feasible placements.

Solutions with good quality and minimal object displacements.

Abstract

For planning rearrangements of objects in a clutter, it is required to know the goal configuration of the objects. However, in real life scenarios, this information is not available most of the time. We introduce a novel method that computes a collision-free placement of objects on a cluttered surface, while minimizing the total number and amount of displacements of the existing moveable objects. Our method applies nested local searches that perform multi-objective optimizations guided by heuristics. Experimental evaluations demonstrate high computational efficiency and success rate of our method, as well as good quality of solutions.

Full text

I INTRODUCTION

For useful integration of robotic systems into everyday life, they must be capable of performing high-complexity real-life tasks efficiently. For instance, typical human environments, such as table tops, kitchen shelves, or office desks, are usually cluttered; and manipulating the environment to deal with such clutter is integral to performing everyday chores in social environments, whether that means rearranging objects upon a surface or across multiple surfaces.

Geometric rearrangement of multiple movable objects on a surface is a difficult problem, because it requires the manipulation of existing objects on the surface, as well as the placement of new objects to be put on the surface. To solve such a problem, in general, task planning is required to decide for the order of manipulation actions (e.g., when to pick, place, move objects), and feasibility checks are required to check the execution of each manipulation action against geometric/kinematic constraints (e.g., to avoid collisions). These requirements usually lead to hybrid planning solutions that combine high-level task planning and low-level feasibility checks.

To solve such a planning problem for the rearrangement of objects in a clutter, one needs to know the goal configuration (i.e., how the objects are arranged on the surface at the end of the plan). However, in real life scenarios, this information is not available most of the time. For that reason, it is essential to determine a geometrically feasible goal configuration of objects on the surface before planning for rearrangements.

With this motivation, we study the object placement problem: given a surface cluttered with (unmoveable) obstacles and a set of existing moveable objects on it, and a set of new objects to be placed on the surface, the goal is to find a collision-free placement of all objects on the surface while minimizing the total number and amount of displacements of the existing moveable objects.

We introduce an efficient algorithm based on nested local searches. The innermost search aims to minimize the total penetration depth of objects utilizing a potential field method over a physics-based engine, the intermediate local search aims to minimize the number of collisions by allowing re-placements of objects, and the outermost local search further tries to minimize the number of displacements of the movable objects on the surface with respect to their initial configurations. The intermediate local search relies on a grid-based heuristics to find more plausible object configurations, while the outermost local search relies on a heuristic that gradually relaxes the constraints imposed on object movements.

II RELATED WORK

Related work in robotics Rearrangement of multiple movable objects, a challenging problem that involves planning, manipulation and geometric reasoning, has received much attention in robotics. In particular, planning for geometric rearrangement with multiple movable objects and its variations, such as navigation among movable obstacles [1, 2], have been studied using various approaches. Since even a simplified variant the rearrangement problem with only one movable obstacle has been proved to be NP-hard [3, 4], most studies introduce several important restrictions to the problem, like monotonicity of plans [5, 6, 7, 8, 9], where each object can be moved at most once. Recent work have focused on generating non-monotonic plans [10, 11, 12, 13, 14, 15]. However, in most of these studies [5, 6, 7, 8, 11, 12, 13, 14], it is assumed that the goal configuration is known. Finding suitable arrangements for objects on a cluttered surface has received relatively less attention.

Cosgun et al. [9] propose an algorithm that searches for a suitable placement for a single object on a cluttered surface by discretizing the possible orientations of the object, convolving object pixels with the ones on the table, and identifying candidate regions for the object placement that result in minimal penetration with other objects. A placement is then produced by sampling these regions; however, this placement may not be collision-free. Then, they plan for a sequence of linear push actions to rearrange the clutter and clear space for the new object such that this placement becomes collision-free. Note that there are several limitations in this approach; multiple new objects are not considered, the surface and possible object orientations are discretized, and the final configuration is not necessarily collision-free.

Yu et al. [16] aim to find sensible placements for furniture by initially generating a random arrangement, then rearranging it to minimize a cost function that measures the difference between the current arrangement and several positive examples provided by the user. Kang et al. [15] also follow on this idea, and modify the algorithm so it becomes more suitable for robotic applications. Neither study considers heavily cluttered scenes or utilizes high resolution collision checks. In [15], the task is to rearrange objects currently available in the scene to achieve a more tidy arrangement; no new objects are added and there exists no constraints that force a certain set of objects to be on certain surfaces. Furthermore, in these studies, even the initial state is a feasible (collision-free) configuration, and the goal is to improve it in terms of a measure of tidiness.

Jiang et al. [17, 18, 19] extract object-to-object and object-to-human features from databases of 3D environments and learn semantic/geometric preferences for object surface pairs. Then, they discretize the surfaces’ point cloud into placing areas by random sampling and solve an maximum matching problem to assign each object’s pose to a suitable placing area. This approach only considers placements to a predetermined set of discrete configurations and does not address the more challenging continuous version of the problem.

In our previous work [10], we have proposed an object placement algorithm based on a local search guided with heuristics and random restarts. This work significantly extends our earlier study by introducing an innermost potential field, as well as two nested local searches wrapped around this basic search algorithm, to improve upon the efficiency and quality of solutions, as well as the success rate. Our results indicate orders of magnitude difference in terms of CPU time and success rate in cluttered scenarios.

Related work in other areas A closely related problem to object placement, studied in computer graphics and operations research, is the packing problem (also known as the knapsack problem), where the goal is to place as many objects as possible in a non-overlapping configuration within a given empty container. The packing problem is NP-hard [20]. It has been widely studied in 2D (cf. the survey [21]). It has been also studied in 3D under various conditions/restrictions [22, 23, 24] (e.g., packing a set of polyhedrons into a fixed size polyhedron without considering rotations [25], orthogonal packing of tetris-like items into rectangular bins [26, 27]).

However, the object placement problem is quite different from these 3D packing problems. First of all, since object placement problem is motivated by the geometric rearrangement of objects on a cluttered surface, the surface does not have to be empty and contains movable objects. The packing problem, on the other hand, assumes that the fixed size container is empty. Also the objective function for the object placement problem is different: the goal is to find a collision-free configuration of all objects, so as to minimize the total number and amount of displacements of the existing objects on the surface. The packing problem, on the other hand, aims to find a collision-free configuration of some objects, so as to maximize the coverage rate (i.e., the total volume of the objects packed in the container). Along these lines, the packing problem and the placement problem are different computational problems with different optimization goals. The methods to attack these problems are significantly different from each other and do not allow for direct comparisons.

III PROBLEM DEFINITION

The object placement problem is defined by

• •

a flat surface $S$ and its geometric model $g(S)$ that details its size and shape,

• •

the set $O_{O}$ of non-movable objects (obstacles) on the surface $S$ and the set $g(O_{O})$ of their geometric models,

• •

the set $O_{C}$ of movable objects (clutter) on the surface $S$ and the set $g(O_{C})$ of their geometric models,

• •

the set $O_{N}$ of new objects to be placed on the surface $S$ and the set $g(O_{N})$ of their geometric models,

• •

a set $W$ of continuous placement constraints on objects (e.g., a monitor may be forced to be in the corner of the table), and

• •

an initial collision-free configuration $C_{I}$ of all objects in $O_{C}\cup O_{O}$ on the surface $S$ relative to $g(S)$.

All objects are assumed to be rigid bodies.

Before we define a solution for an object placement problem, let us introduce the following definitions and notations for a configuration $C_{x}$ of all objects $O_{N}\cup O_{C}$ on the surface $S$ relative to $g(S)$.

• •

The number of displacements $\hbox{\it\#move\/}(C_{x})$ in $C_{x}$ is the number of objects in $O_{C}$ whose configurations are different in $C_{x}$ from their original configurations in $C_{I}$.

• •

The amount of displacement $\hbox{\it dist\/}_{o}(C_{x})$ of an object $o\in O_{C}$ in $C_{x}$ is the distance between the centroid of $o$ in $C_{I}$ and the centroid of $o$ in $C_{x}$. The total amount of displacements of objects in $O_{C}$ is $\sum_{o\in O_{C}}\hbox{\it dist\/}_{o}(C_{x})$.

• •

The amount of change $\hbox{\it arc\/}_{o}(C_{x})$ in the orientation of an object $o\in O_{C}$ in $C_{x}$ is the arc length traced by the most distal point on the object, due to the change of configuration of $o$ from $C_{I}$ to $C_{x}$. The total amount of change in orientations of objects in $O_{C}$ is $\sum_{o\in O_{C}}\hbox{\it arc\/}_{o}(C_{x})$.

• •

The total amount of change $\hbox{\it cost\/}_{d}(C_{x})$ in configurations of objects in $O_{C}$ with respect to $C_{I}$ is the sum of the total amount of displacements of objects in $O_{C}$ (i.e., $\sum_{o\in O_{C}}\hbox{\it dist\/}_{o}(C_{x})$) and the total amount of change in orientations of objects in $O_{C}$ (i.e., $\sum_{o\in O_{C}}\hbox{\it arc\/}_{o}(C_{x})$).

A solution for an object placement problem $\langle g(S)$, $O_{O}$, $g(O_{O})$, $O_{C}$, $g(O_{C})$, $O_{N}$, $g(O_{N})$, $C_{I}$, $W\rangle$ is a collision-free final configuration ${C}_{F}$ of all objects $O_{N}\cup O_{C}$ on the surface $S$ relative to $g(S)$, with a minimal value of $\langle\hbox{\it\#move\/},\hbox{\it cost\/}_{d}\rangle$ relative to lexicographic ordering (i.e., there does not exist a collision-free configuration $C_{x}$ such that $\hbox{\it\#move\/}(C_{x}){<}\hbox{\it\#move\/}(C_{F})$, or $\hbox{\it\#move\/}(C_{x}){=}\hbox{\it\#move\/}(C_{F})$ and $\hbox{\it cost\/}_{d}(C_{x}){<}\hbox{\it cost\/}_{d}(C_{F})$.

Figure 1 presents a sample problem instance: initially in Figure 1(a), a cylindrical obstacle (yellow), and four geometrically different movable objects (red) are placed on a surface (green); the goal is to find a collision-free configuration of these objects and some more new objects (blue), like in Figure 1(c).

IV METHOD

We propose a nested local search algorithm (Algorithm 1) to compute a solution for an object placement problem. Intuitively; the innermost search applies potential field method over a physics-based engine, with the goal of minimizing the total penetration depth of objects, the intermediate local search further aims to minimize the number of collisions by allowing re-placements of objects, and the outermost local search further tries to minimize the number and amount of displacements of movable objects with respect to their initial configurations.

IV-A Penetration Minimization

The innermost search relies on artificial potential fields [28] defined for each object (including the obstacles and boundaries of the table) and a dynamic simulator to calculate a configuration in static equilibrium. In particular, trajectories of objects under the action of interaction forces due to potential fields are simulated according to the governing equations of motion, that are, effectively, a solution to a dynamic optimization problem over the system’s Lagrangian. The algorithm starts with a configuration $C_{\hbox{\scriptsize\it current\/}}$ obtained from $C_{I}$ by randomly placing the new objects $O_{N}$ on the surface $S$, with zero velocities. It searches over configurations of all objects $O_{O}\cup O_{C}\cup O_{N}$ on the surface $S$, with the goal of minimizing the objective optimization function $\hbox{\it cost\/}_{p}$ defined as the sum of maximum penetration depth [29] of pairs of objects in collisions. The physics engine returns a configuration $C_{\hbox{\scriptsize\it next\/}}$ of all objects, minimizing $\hbox{\it cost\/}_{p}(C_{\hbox{\scriptsize\it next\/}})$.

For instance, consider the initial configuration $C_{I}$ of obstacles (yellow) and movable objects (red) shown in Figure 1(a). A configuration $C_{\hbox{\scriptsize\it current\/}}$ of objects obtained by randomly placing all the new objects (blue) on the surface can be seen in Figure 1(b). Note that there are many collisions in $C_{\hbox{\scriptsize\it current\/}}$, and objects penetrate with each other. By applying this innermost search, the configuration shown in Figure 1(c) may be generated, where the total penetration is zero, since there is no collisions.

The only action that the physics engine can perform (due to the use of the potential field method) is to push objects in collision outside of each other; so, for instance, it cannot swap locations of two objects. This may lead to local minima where the objects can no longer be pushed, and there may be still some collisions in $C_{\hbox{\scriptsize\it next\/}}$. This motivates us towards the intermediate local search algorithm that utilizes some heuristics to re-place all the objects in collision.

IV-B Collision Minimization

The intermediate local search algorithm (Algorithm 2) utilizes heuristically-guided re-placements to avoid local minima. Intuitively, for each object $o$ in collision, the heuristic suggests (i) dividing the surface $S$ into cells and (ii) re-initiating the placement of the object $o$ into the center one of the cells that is unoccupied (i.e., has no other object centroids); the heuristics is used to guide generation of new configurations. For (i), the heuristic imposes a grid on the surface $S$. If a grid cell has no object centroids in it, it is marked as free. If no free cell exists, a new refined grid is imposed on $S$ with smaller but more number cells. This refinement process continues until when there is at least one free cell in the imposed grid.

The intermediate local search algorithm also starts with a configuration $C_{\hbox{\scriptsize\it current\/}}$ of all objects $O_{O}\cup O_{C}\cup O_{N}$ on the surface $S$ obtained from $C_{I}$ by randomly placing the new objects $O_{N}$ on the surface $S$. It calls the innermost search algorithm described above to find a better configuration, but starting with the configurations obtained from $C_{\hbox{\scriptsize\it current\/}}$ as suggested by the heuristics. The goal is to minimize the objective optimization function $\hbox{\it cost\/}_{c}$ defined as a tuple $\langle\hbox{\it\#col\/},\hbox{\it cost\/}_{p}\rangle$, where #col is the total number of collisions and $\hbox{\it cost\/}_{p}$ is the total amount of penetration depth of pairs of objects in collisions. Here, lexicographic ordering is used to find the minimum of two tuples: $\langle a,b\rangle\prec\langle a^{\prime},b^{\prime}\rangle$ if either $(a\prec a^{\prime})$ or ($a=a^{\prime}$ and $b\prec b^{\prime}$). In this way, priority is given to #col and then to $\hbox{\it cost\/}_{p}$: among multiple configurations of all objects with the same minimum number of collisions, the configuration $C_{\hbox{\scriptsize\it next\/}}$ with the least cumulative penetration depth is returned.

An example is presented in Figure 2 to illustrate the usefulness of the heuristics in the intermediate local search algorithm. The search starts with a configuration $C_{\hbox{\scriptsize\it current\/}}$, where ${\hbox{\it\#col\/}(C_{\hbox{\scriptsize\it current\/}})=24}$ and ${\hbox{\it cost\/}_{p}(C_{\hbox{\scriptsize\it current\/}})=1.75}$. Note that the innermost search alone cannot find a better configuration with less value of $\hbox{\it cost\/}_{p}$: the physics engine gets stuck, as it can no longer push objects on the right half of the surface. First, the heuristic is utilized to re-place the cyan-colored object, which is in collision with some other object, in $C_{\hbox{\scriptsize\it current\/}}$. For that, a grid is imposed over the surface with two free cells, labelled A and B, that do not contain any object centroids. Then, two new configurations, $C_{\hbox{\scriptsize\it A\/}}$ and $C_{\hbox{\scriptsize\it B\/}}$, are obtained from $C_{\hbox{\scriptsize\it current\/}}$ by randomly re-placing the cyan-colored object in A and in B, respectively. Here, ${\hbox{\it cost\/}_{c}(C_{\hbox{\scriptsize\it A\/}})=\langle 19,1.43\rangle}$ and ${\hbox{\it cost\/}_{c}(C_{\hbox{\scriptsize\it B\/}})=\langle 20,1.37\rangle}$. At this point, the intermediate local search algorithm calls the innermost search algorithm for each of these two configurations.

The intermediate local search algorithm with heuristically-guided re-placements is useful for minimizing the number of collisions on a surface, but the objects in $O_{C}$ may end up being displaced and rotated too much with respect to their original configurations in $C_{I}$. This is undesirable from the perspective of rearrangement planning, because it will require more number of manipulation actions to rearrange such objects. This motivates us towards the outermost local search algorithm, which utilizes some constraints on the placements of objects to limit their movements.

IV-C Rearrangement Minimization

The outermost local search algorithm (Algorithm 1) utilizes placement constraints to minimize displacements. Intuitively, the amount of displacement of a movable object $o\in O_{C}$ is constrained to a ball whose centroid is the object’s centroid in the given initial configuration in $C_{I}$. Initially, $\hbox{\it radius\/}_{o}$ is set to [math]; if the outermost local search algorithm cannot find a better configuration under these constraints, then $\hbox{\it radius\/}_{o}$ is increased slightly.

The outermost local search algorithm starts with a configuration $C_{\hbox{\scriptsize\it current\/}}$ of all objects $O_{O}\cup O_{C}\cup O_{N}$ on the surface $S$ obtained from $C_{I}$ by randomly placing the new objects $O_{N}$ on the surface $S$. It calls the intermediate local search algorithm described above to find a better configuration, but with respect to the given set $V$ of placement constraints. The goal is to minimize the objective optimization function $\hbox{\it cost\/}_{r}$ defined as a triple $\langle\hbox{\it\#col\/},\hbox{\it\#move\/},\hbox{\it cost\/}_{d}\rangle$, where #col is the total number of collisions, #move is the total number of moves of objects in $O_{C}$ from their original places, and $\hbox{\it cost\/}_{d}$ is the total amount of change in configurations of objects in $O_{C}$ with respect to $C_{I}$. Here, $\hbox{\it cost\/}_{d}$ for a configuration $C_{x}$ is computed as the sum of the total amount of displacements of objects in $O_{C}$ (i.e., $\sum_{o\in O_{C}}\hbox{\it dist\/}_{o}(C_{X})$, where $\hbox{\it dist\/}_{o}(C_{X})$ is the distance between the centroid of $o$ in $C_{I}$ and the centroid of $o$ in $C_{x}$) and the total amount of change in orientations of objects in $O_{C}$ (i.e., $\sum_{o\in O_{C}}\hbox{\it arc\/}_{o}(C_{X})$, where $\hbox{\it arc\/}_{o}(C_{X})$ is the arc length due to the change of configuration of $o$ from $C_{I}$ to $C_{x}$). The outermost local search algorithm also uses lexicographic ordering to find the minimum of two triples. In this way, priority is given first to #col, then to #move, and then to $\hbox{\it cost\/}_{d}$.

An example is presented in Figure 3 to illustrate the usefulness of the placement constraints in the outermost local search algorithm. The search starts with a configuration $C_{\hbox{\scriptsize\it current\/}}$, where ${\hbox{\it\#col\/}(C_{\hbox{\scriptsize\it current\/}})=4}$ and $\hbox{\it\#move\/}(C_{\hbox{\scriptsize\it current\/}})=\hbox{\it cost\/}_{d}(C_{\hbox{\scriptsize\it current\/}})=0$. Initially, the radius of the placement balls for every object that is initially on the table is zero. With these placements constraints, the intermediate local search cannot find a better configuration to optimize $\hbox{\it cost\/}_{r}$. Then, for each object, the outermost local search algorithm relaxes its placement constraints by increasing the radius of its placement ball by a certain percentage of $max_{d}(o)$, where $max_{d}(o)$ is the distance from the centroid of the object to the furthest point of the surface, then calls the intermediate local search again. With such relaxed constraints, the intermediate local search algorithm returns better configurations with less costs. For example, when it is allowed to place the cuboid object within a small circle around its initial configuration, the intermediate local search algorithm returns a configuration $C_{\hbox{\scriptsize\it cuboid\/}}$ with ${\hbox{\it cost\/}_{r}(C_{\hbox{\scriptsize\it cuboid\/}})=\langle 2,1,2.82\rangle}$; whereas, when it is allowed to displace the cylindrical object by a small amount, it returns a configuration $C_{\hbox{\scriptsize\it cyl\/}}$ with ${\hbox{\it cost\/}_{r}(C_{\hbox{\scriptsize\it cyl\/}})=\langle 0,1,1.76\rangle}$ (Figure 3(a)). The outermost local search algorithm continues search from $C_{\hbox{\scriptsize\it cyl\/}}$, with even more relaxed placement constraints, but cannot find a better configuration with less cost; so the outermost search stops (Figure 3(b)).

V EXPERIMENTAL EVALUATION

We have conducted three sets of experiments to evaluate and compare performance of each search level. All simulations were executed on workstation with an Intel Xeon W-2155 CPU running at 3.30 GHz using a single thread and 32 GB RAM. All algorithms were implemented in Python, with Bullet [30] as the back-end physics engine. Each instance of each experiment was run 60 times to allow for averaging of the results. A timeout of 300 s per trial was imposed.

Experiment 1 started with the initial configuration depicted in Figure 1(a), where 4 objects and an obstacle were on a surface. The number of new objects to be added to the table was the control variable in this condition. The number of new objects was increased from 4 to 36, such that the total footprint area covered by all objects and the obstacle were gradually increased up to 95% of the total surface area.

Experiment 2 involved an obstacle and 4 new objects to be placed on the table. The number of movable objects that were initially on the table was the control variable in this condition. The number of movable objects on the table was increased from 4 to 36, such that the total footprint area covered by all objects and the obstacle were gradually increased up to 95% of the total surface area.

Experiment 3 involved 4 initial objects randomly placed on the surface in collision-free configurations and 4 new objects to be added to the surface. The number of obstacles on the table was the control variable in this condition. The number of obstacles was increased from 4 to 32, such that the total footprint area covered by all objects and the obstacles were gradually increased up to 95% of the total surface area.

Five algorithms were compared, where three of them correspond to the different levels of our nested local search:

• •

innermost search (Inner) based on potential fields,

• •

intermediate local search (Intermediate) wrapped around the potential field, and

• •

the outermost local search (Outer) wrapped around the intermediate local search.

Remaining two algorithms are taken as baselines to demonstrate effectiveness of our approach with respect to naive implementations. In particular,

• •

to highlight the benefits of using the innermost search, we have tested Random Sample approach, which randomly samples new configurations for objects with uniform distribution until a collision-free placement is found.

• •

To highlight the benefits of using our intermediate local search, we have tested Random Restart which is implemented as the innermost search Inner with random restarts.

The performance of the algorithms were compared based on three metrics:

• •

efficiency, measured by the average CPU time spend to calculate a solution,

• •

quality, measured by the average number of objects moved that were initially on the table, and

• •

success rate, measured by percentage of trials that converge to a collision-free final configuration before a time-out is reached.

Efficiency and quality metrics are computed for partial solutions of unsuccessful trials that return configurations with collision(s).

Figure 4 graphically summarizes the data collected from Experiments 1–3. In Figure 4, each row corresponds to an experiment, while columns present, CPU time, quality and success rate metrics, respectively.

We can observe the effect of increasing the number of new objects to be put on the table from the first row of Figure 4.

• •

In terms of average CPU time, Inner outperforms Random Sample, while Intermediate consistently outperforms all other algorithms, including Random Restart. While Outer is in general slower than Random Restart for problems with surface coverage less than 75%, Outer outperforms Random Restart in highly cluttered environments, since it relies on Intermediate when stuck at local minima.

• •

In terms of solution quality, Inner and Intermediate perform quite similarly and move more objects as the table gets more cluttered, while Outer performs significantly better than both. In particular, up to 30% surface coverage, Outer could find solutions that do not require any movements of the objects initially on the table. As the clutter increases, it becomes necessary to move some of these objects, but the number of moved objects stays significantly less than those in the solutions computed by Inner and Intermediate. It is important to note that the baseline algorithms, Random Sample and Random Restart, consistently result in low solution quality, even when compared to algorithms that do not distinguish between original and new objects, such as Intermediate.

• •

In terms of success rate, we note that Intermediate and Outer can solve almost all problems up to 85%, and are the only algorithms capable of solving problems more cluttered than 85%, although the success rate drops to about 40%. While Random Sample outperforms Inner for simple problems where the surface coverage is less than 30%, Inner can solve some problems of up to 75% surface coverage, while Random Sample cannot solve any problems that has more than 40% surface coverage.

We can observe the effect of increasing the number of original objects on the table from the second row of Figure 4. Note that these objects can be moved but are desired not to be relocated; hence, the quality of solutions becomes more emphasized in these experiments.

• •

In terms of average CPU time, once again, Inner outperforms the Random Sample, and Intermediate outperforms Random Restart. From these experiments, we can observe how increasing the number of original objects affects Outer, causing it to have a sharp increase in computation time at about 50% surface coverage.

• •

The trend observed from the first set of experiments regarding solution quality can be seen again here. As Outer is the only algorithm that attempts to improve solution quality, it consistently moves fewer original objects across all surface coverage levels.

• •

The success rate of Outer varies over surface coverage levels, but is always worse than Intermediate. In particular, given that Outer aims to solve a more constrained version of the problem solved by Intermediate, it is not surprising that it fails more often. In this experiment set, Intermediate is the only algorithm that is capable of solving a significant portion of problems with 95% surface coverage.

We can observe the effect of increasing the number of obstacles on the table from the third row of Figure 4. Note that since obstacles cannot be moved, these instances prove to be much harder than the previous experiments, as the percentage of surface coverage increases.

• •

In terms of average CPU time, Random Restart, Intermediate, and Outer perform similarly up to 65% clutter, after which Random Restart baseline falls behind in computation time. The computation time for Outer and Intermediate display great similarity until about 85%, after which no algorithm can solve even a single instance. Note that as the obstacles increase, the problem become highly constrained and objects initially located on the table are trapped; hence, object movements are much less in this experiment.

• •

In terms of solution quality, Random Sample and Random Restart baselines perform quite similarly and move almost all objects initially on the table, while Outer and Intermediate perform slightly better than both baselines. Inner performs the best in terms of solution quality, but this is only due to the constrained nature of the problem mentioned earlier.

• •

Inner and Intermediate display significantly different trends in terms of success rate. In particular Inner starts failing quite early and can only solve about 30% of instances at 55% clutter, while Intermediate and Outer consistently find over 80% of the solutions up to 85% clutter, after which all approaches fail.

The results of these experiments indicate the usefulness of all three nested local searches proposed by our method. Outer significantly improves solution quality by minimizing movements of objects on the table, Intermediate significantly improves success rate by allowing replacements when stuck in local optima, and Inner significantly improves CPU time over Random Sampling especially for cluttered scenarios.

VI Solutions to Benchmarks Instances

To demonstrate the ability of our local search approach to solve a large variety of placement problems, we have also tested it with several difficult benchmark scenarios. These benchmarks have been engineered to result in difficult instances, by introduction of non-convex objects, confined surfaces, and very specific configurations that result in feasible placements.

VI-A Confined Placement Benchmark

In the confined placement benchmark, 4 objects needs to be placed on a surface highly cluttered with randomly placed of obstacles and 4 movable objects, as shown in Figure 5(a). All objects and obstacles are selected as clones of the same 4 basic convex shapes. This benchmark is challenging as it requires the placement algorithms to be able to handle highly restricted spaces. Figure 5(b) and 5(c) present solutions computed by Intermediate and Outer, respectively. The solution computed by Outer does requires only 1 of the 4 movable objects initially on the surface to be relocated, while the solution computed by Intermediate relocates all of these objects.

VI-B Tight Placement Benchmark

The tight placement benchmark shown in Figure 6(a), where the goal is to find a placement for the blue box upon the green surface, has been introduced in [10]. This benchmark is challenging as it requires the objects on the surface to be rearranged to very specific configurations such that a solution can be computed. Furthermore, this benchmark includes non-convex objects with holes, which need to be utilized to compute a solution. Figure 6(b) depicts a random placement for this problem, demonstrating the need for rearrangement of objects on the surface. Figure 6(c) presents the solution computed by our algorithm.

VI-C Elongated Object Benchmark

In the elongated object benchmark, the length of slender objects are designed to pose a challenge, as shown in Figure 7(a). In particular, one of the objects is selected to have a length greater than the width of the square surface, while 11 other slender objects are set to have a length that is equal to the half the width of the square surface. This benchmark is difficult, as the placement of the longest object introduces unusable space on the square surface, rendering convergence to a solution significantly more difficult. Figure 7(b) depicts a random placement for this problem, demonstrating the unusable area introduced by the long object. Figure 7(c) presents the solution computed by our algorithm.

VI-D L-Shape Placement Benchmarks

L-shape benchmarks are commonly utilized to demonstrate capabilities of packing algorithms [31]. L-shape placement benchmarks adapt these to placement problems with 2D and 3D variations, as shown in Figures 8 and 9, respectively. In L-shape placement benchmarks, the objects are required to be placed on a surface with no overhangs, and the total contact area of the L-shapes to be placed are equal to the area of the surface. In the 2D version, the L-shapes are configured such the objects have uniform height along the z-axis and are non-convex on parallel to the surface, while in the 3D version, the L-shapes are configured such the objects are uniform parallel to the surface and are non-convex along the z-axis. Figures 8(b) and 9(b) present random placements for 2D and 3D benchmarks, demonstrating the difficulty of finding a collision free solution with no overhangs from the table. Figures 8(c) and 9(c) present the solutions computed by our algorithm to the 2D and 3D benchmarks, respectively.

VII CONCLUSION

Given a surface cluttered with (unmovable) obstacles and movable objects, and a new set of objects, the object placement problem asks for a collision-free placement of all the objects on the surface. We have introduced a novel algorithm to solve this problem, utilizing nested local search algorithms with multi-objective optimizations: the innermost search tries to minimize the total penetration depths of objects, the intermediate local search further tries to minimize the number of collisions, and the outermost local search further tries to minimize the changes in object poses. Each level of the search is guided by heuristics: the innermost search utilizes a potential field method over a physics-based engine, the intermediate local search gradually utilizes re-placements of objects to avoid local minima, and the outermost local search gradually relaxes the constraints that specify how far objects can be displaced. In that sense, our method introduces novel mathematical search models with nested multi-objective optimizations, and algorithms that further utilize heuristics to avoid local minima. On the other hand, due to local searches, our algorithm is more about local optimization and is likely not to reach the global optima.

Comprehensive experimental evaluations demonstrate high computational efficiency and success rate of our method, as well as good quality of solutions. Furthermore, difficult benchmark scenarios that include non-convex objects, confined surfaces, and very specific configurations that result in feasible placements, can also be solved with our method.