Locating Transparent Objects to Millimetre Accuracy

TL;DR

This paper introduces a method for precisely locating transparent objects like glass using a laser range finder and a physical understanding of laser-glass interactions, achieving millimetre accuracy.

Contribution

The paper presents a novel approach combining physical insights and ICP registration to accurately locate transparent objects with a simple LRF.

Findings

Achieves millimetre accuracy in locating glass objects

Effective for non-planar, transparent surfaces

Validated experimentally on car glass

Abstract

Transparent surfaces, such as glass, transmit most of the visible light that falls on them, making accurate pose estimation challenging. We propose a method to locate glass objects to millimetre accuracy using a simple Laser Range Finder (LRF) attached to the robot end-effector. The method, derived from a physical understanding of laser-glass interactions, consists of (i) sampling points on the glass border by looking at the glass surface from an angle of approximately 45 degrees, and (ii) performing Iterative Closest Point registration on the sampled points. We verify experimentally that the proposed method can locate a transparent, non-planar, side car glass to millimetre accuracy.

Figures

Tables

Table 1. TABLE I : LRF properties

Property	Value
Angle (from normal)	2
Exposure time	5 ms
Selection method	highest intensity
Distance from glass to opaque surface	57 cm

Table 2. TABLE II : Point localisation error

	Opaque object	Glass
Mean error ($\pm$ std) (mm)	1.01 ($\pm$ 0.14)	0.94 ($\pm$ 0.28)
Max error (mm)	1.27	1.35

Table 3. TABLE III : Average operation time

Stage	Average time (s)
Scanning	14.845
Pose estimation	0.023
Path planning	0.782
Execution	26.884

Table 4. TABLE IV : Full pipeline localisation error

	Localisation error
Mean error ($\pm$ std) (mm)	0.88 ($\pm$ 0.39)
Max error (mm)	1.65

Equations

$$ϵ:=||FK({\mathbf{q}}_{\mathrm{actual}}){}^{\mathrm{flange}}{\mathbf{T}}_{\mathrm{tooltip}}-FK({\mathbf{q}}_{\mathrm{ref}}){}^{\mathrm{flange}}{\mathbf{T}}_{\mathrm{tooltip}}.||$$

$$ϵ:=||FK({\mathbf{q}}_{\mathrm{touch}}){}^{\mathrm{flange}}{\mathbf{T}}_{\mathrm{tooltip}}-FK({\mathbf{q}}_{\mathrm{ref}}){}^{\mathrm{flange}}{\mathbf{T}}_{\mathrm{tooltip}}.||$$

Full text

I Introduction

Transparent surfaces, such as glass, transmit most of the visible light that falls on them. For a common plate of glass, only about 4% of the incident light are reflected from each surface while the other 92% are transmitted across. This non-opaque characteristic prevents common vision sensors from locating transparent objects with high accuracy. Given the prevalence of transparent objects in industrial and home environments, such a limitation hinders the application of many vision-based robotics systems.

Most robotics research into the detection/recognition of transparent objects is only concerned with coarse localisation – typically, centimetre-accuracy in indoor navigation or tens-of-centimetre-accuracy in outdoor navigation. By contrast, industrial tasks (transport, assembly, glue dispensing on car glasses, etc.) require a much higher level of accuracy, typically of several millimetres or submillimetre. Furthermore, most of the existing methods in transparent objects shape and pose estimation require elaborate apparatus setup (e.g. specific and fixed configurations of multiple sensors).

Here, we propose a method to locate glass objects to millimetre accuracy using a simple LRF attached to the robot end-effector. The proposed method can locate both planar and non-planar objects for subsequent manipulation. Briefly, the proposed localisation and manipulation pipeline is as follows:

Scanning: Sample $n$ points along the object border by looking at the surface with the LRF from an angle of approximately 45 degrees. We assume in this paper that a coarse localisation has been performed beforehand, using existing methods[1][2]; 2. 2.

Registration: Perform Iterative Closest Point (ICP) registration of the $n$ points against an available 3D model; 3. 3.

Manipulation: Based on the object pose estimate, compute the robot trajectory to perform the manipulation (here: following the object contour mimicking a glue dispensing task).

The remainder of the paper is organized as follows. In Section II, we discuss several works done in localisation of transparent objects. In Section III, we survey the background knowledge on laser-glass interactions. In Section IV, we present our glass localisation method. Sections V and VI report experimental results on respectively point and full-object localisation. Finally, in Section VII, we conclude and sketch directions for future work.

II Previous work on localisation of transparent objects

Most research into transparent object recognition does not require the estimated object to be manipulated upon. As such, the estimated models are often simplified and lack the necessary detail for accurate manipulation. Furthermore, most of the methods require elaborate apparatus setup.

In [1], Hata used a structured light setup to project stripe patterns into transparent object with one flat side. The refracted pattern is then inspected and extracted using imaging sensor. Genetic Algorithm is then iteratively employed to construct a 3D model until the calculated error is within a threshold. The author achieved shape accuracy below 20% after running 50 generations for six hours.

In [2], the authors tracked features from different frames of a moving camera to estimate shapes and poses of transparent objects. The shape reconstruction assumes parametric form of the transparent object. They showed that the problem can be highly non-linear even for object with simple shape. Achieved shape parameters error range from submillimetre to tens-of-centimetre while there was no data on real experimental pose parameters.

In [3], Narita proposed a non-contact optical method for measuring the 3D shape of transparent objects and their refractive indices using LRF. The method relies on observing and triangulating the two strongest reflected light rays. The author could measure thickness of two transparent object of simple shapes to submillimetre error, obtained through non-trivial arrangement of multiple sensors.

In [4], Kutulakos calculated depth maps of opaque and transparent specular objects by reconstructing light paths from a light source. The number of viewpoints required depends on the number of times the light get redirected. An experiment to reconstruct a diamond-shaped glass object with eight faces and from five viewpoints was done. The average RMS distance from the obtained points to the estimated plane is 1.33 mm.

In the robotics field, transparent object detection is of special interest for mobile robot localisation especially to avoid collision. Developments in non-flat object recognition through robotic arm haptic-based approach are of interest as well, which can be generalized to transparent objects. These approaches hold promise with their non-reliance on external vision sensors, which tend to fail transparent objects.

In [5], Foster tackled the problem of mobile robot localisation in an environment with many glass obstacles using a LIDAR sensor. The author looked into the behaviour and working of LIDAR sensors to recognise the visibility of glass at certain view angles and conditions. These view angles are later experimentally identified and used in modifying the standard occupancy grid algorithm. The algorithm looks for evidence that a transparent object occupies a cell but does not estimate the pose of occupying object.

In [6], Allen employed Utah-MIT hand fitted with tactile sensor pads that was attached to PUMA 560 manipulator. Borrowing previous research on human haptic system [7], they extended human hand movement strategies in discovering 3D objects attributes to robotic domain. The idea was to generate a rough initial shape estimate using Grasping by Containment before performing additional exploratory procedures (EPs) to refine the structure.

In [8], Ibrayev developed an invariant-based method to recover surface geometry with few data collected from touch sensors. Firstly, curve class recognition is done by calculating differential invariants for few classes of quadratic curves and special cubic curves. From there, actual curve is estimated based on the shape parameterisation as well as contact locations on the curve. The author claimed an average relative error of 1% depending on the curvature and its derivative estimation performance.

III Laser-glass interactions

III-A Triangulation-based laser scanners

The triangulation-based laser scanner works by projecting a laser line beam onto the object surface to be determined. The reflected light is then subsequently captured by the scanner’s receiver. In our application, the scanner projects laser line onto the object at right angle while the receiver is pointed towards the reflected line from an angle as seen in Figure 2 (left). In the sensor’s receiver, the reflected light will fall on an array of Charge-Coupled Device (CCD), from which the object position is calculated.

Most of the light that falls onto the transparent material either gets reflected or transmitted. The reflected light consists of two different types: specular and diffuse, as shown in Figure 2 (right). In the former, the incident and reflected rays’ angles are equal but located on opposite sides of the reflected surface normal. In the latter, the incident ray is scattered to reflected rays that cover many angles. Meanwhile the transmitted ray might be transmitted again through the rear surface or get refracted multiple times internally before being transmitted out from either surfaces.

III-B Laser reflection from glass surfaces

The laser scanner’s ability to register points from glass surfaces depends on several main factors, two of which are: laser reflection type and presence of opaque object in the background.

Specular reflections that reach the receiver sensor tend to saturate the CCD, causing blooming effect [9] as shown in Figure 3. The saturated pixels will lead to inaccurate reading and hence false points being registered. Avoiding specular reflections can be challenging especially when dealing with non-planar glass with varying surface normal angles.

On the other hand, the scanner’s receiver relies on diffuse reflection of the laser line to capture the surface contour. However, the diffuse reflections from the glass surface tend to be very low. As previously mentioned, only about 8% of the laser light get reflected back from both surfaces. Only part of this is reflected diffusely due to glass surface imperfection or small particles. Depending on the laser scanner, some setting adjustment on exposure time or minimum intensity threshold might be required to pick up these low intensity points.

Presence of opaque surfaces in the background can also cause the scanner to register false reflections. This problem is sometimes referred to as the Mixed Pixels problem in literature [10]. In some cases involving LIDARs, the range measurement with higher intensity has a higher chance of being selected [5]. The author argued that the intensity of return from glass can overpower the opaque background object only for certain critical angle near the glass normal for a constant distance between the two objects. An example given was that when a diffuse wall was placed 1 m behind the glass, the glass will only be visible from a small 0.25 degree angle range.

In Figure 4, we show the obtained reading from the glass surface taking the above factors into account. Figure 4(a) shows the sparse 3D points constructed by moving the robotic arm to scan along a flat glass surface marked in Figure 4(b). The LRF scanning properties used are listed in Table I.

However, a closer look into the obtained 3D points in Figure 4 reveal irregularities along both surfaces of the glass as shown in Figure 5(a). The point distribution is characterised by distinct inclined spikes which mask the true surface flatness. Outlier points are also registered along the thickness of the glass at different parts of the glass. In Figure 5(b), we show the noisy reading obtained when specular reflection is involved. The glass panel spans the horizontal middle section of the image, with irregular points distributed above and below it. All of these observations point towards the unpracticality of the scanned output.

In section IV, we propose an alternate method to realize our glue-dispensing robotic system that does not rely on such irregular output.

III-C Laser reflection from glass edges

While the transparent surface of the car glass window gives mostly unreliable readings, the edge offers a better compromise. Most glass panels, such as the car window, have higher surface irregularities along the border that can be attributed to applied glass edge finishing which is often required for aesthetics and safety reasons. This physical property allows lasers that fall on the less transparent glass edge to give out more diffuse reflection. This is desirable as it provides cleaner and hence more accurate range measurement.

There is, however, the uncertainty whether laser line projection could return points from the edge line reliably. To understand this, we devised an experiment to assess the distribution of laser line on the transparent-opaque surface intersection as seen in the bottom of Figure 6:

Tape is laid along the border and onto the glass surface. The laser reading obtained this way reflects the true contour of the glass. 2. 2.

A border cover object is placed on the glass surface that coincides with the edge line. Another laser reading is then obtained. 3. 3.

After removing the tape and cover object, another laser reading is obtained. 4. 4.

Repeat the experiment for various scan angles.

In both Figures 7(a) and 7(b), the laser reading from Step 1 and 2 are traced by the blue crosses and green squares respectively. The first intersection from the left between the blue and green plots corresponds to the point on the edge line and the laser line readings obtained from Step 3 are visualised as red plots. It was found that the selection of scanning angle affects the degree at which the laser points coincide with the edge line. In Figure 7(b), laser points from a 40 degrees scanning angle above the horizontal plane display closer proximity to the edge line compared to laser points from a 10 degrees scanning angle in Figure 7(a).

IV Glass pose estimation pipeline

IV-A Pipeline

Based on the observations of the previous section, we propose the following pipeline:

Scanning: Sample $n$ points along the object border by looking at the surface with the LRF from an angle of approximately 45 degrees. We assume in this paper that a coarse localisation has been performed beforehand, using existing methods[1][2]; 2. 2.

Registration: Perform Iterative Closest Point (ICP) registration between the $n$ points against an available 3D model; 3. 3.

Manipulation: Based on the object pose estimate, compute the robot trajectory to perform the manipulation (here: following the object contour mimicking a glue dispensing task).

IV-B Scanning

We position the laser scanner at $n$ different poses along the glass border with the laser line intersecting the edge. At each pose, the scanner receives a set of points from the laser line being reflected off the glass edge. We then proceed to filter these collinear points to obtain a single point which we call the Border Identifier Point (BIP). The BIP filtering method should ensure consistent point location among the collinear points for all the $n$ poses. For our application, we ensure that the BIP selected is the highest point among the collinear points above the ground plane. This point corresponds to the point $\boldsymbol{p}_{e}$ on the intersection between the transparent glass surface and diffuse glass edge (refer to the illustration on top of Figure 6). Hence, in the end we obtain set $S$ containing $n$ BIPs from $n$ scanner configurations.

IV-C Registration

Our proposed method requires a 3D model of the glass object to be available. If unavailable, pre-measures can be employed to augment the 3D model construction. For instance, coating materials such as talcum powder can be applied on the surface to obtain accurate scan measurement. Construction of the 3D model need only be done once and hence will not affect subsequent glue-dispensing operation time.

ICP registration [11] is then performed to obtain the estimated pose of the car window. The ICP algorithm works in principle by matching two cloud points, in this case the cloud points formed from $S$ against the constructed 3D model, $M$. Through minimizing the distance distribution, the algorithm returns a transformation that aligns the two models as seen in Figure 8.

V Experiment 1: point localisation

The experiment can be briefly summarised as follows: The robot aligns the laser line with the point-of-interest before moving the tooltip to touch it. The resulting distance from the point-of-interest to the final tooltip position is then recorded.

We perform this experiment on two different object profiles for comparison: Firstly on glass edge and later on an opaque object’s edge with diffuse reflection. Through comparison of the two results, we can obtain the relative accuracy of scanning on glass edge against opaque object edge as ground truth.

The laser scanner used is a 2D LRF, Microepsilon Scancontrol 2600-100, with 658 nm red semiconductor laser as light source. The LRF transformation parameters relative to the robot’s flange is calibrated based on method described in SCALAR [13].

V-A Method

The experiment steps are as follows:

Position the laser beam line to intersect the object edge at $n$ multiple locations spanning the perimeter and calculate the BIP point ${}^{B}\boldsymbol{p}_{\mathrm{BIP}}$ at each location, where $B$ represents the robot’s base coordinate frame. 2. 2.

At each location, mark the point on the object under the laser beam line which is supposed to coincide with the respective ${}^{B}\boldsymbol{p}_{\mathrm{BIP}}$ and move the tooltip to touch it while keeping a predefined orientation of the tooltip, $\boldsymbol{R}_{\mathrm{approach}}$. Record the joint states when the tooltip touches the desired point as $\boldsymbol{q}_{\mathrm{ref}}$. 3. 3.

Program the robot to move towards each ${}^{B}\boldsymbol{p}_{\mathrm{BIP}}$ with $\boldsymbol{R}_{\mathrm{approach}}$ as approach orientation. Record the resulting joints states as $\boldsymbol{q}_{\mathrm{actual}}$.

Let the Forward Kinematics mapping from the configuration-space to task-space be $FK:\boldsymbol{q}\mapsto\boldsymbol{p}$ and the tooltip transformation relative to the robot’s flange be ${}^{\mathrm{flange}}\boldsymbol{T}_{\mathrm{tooltip}}$.

The localisation error can be defined as follows

[TABLE]

One can then define the mean and maximum errors across all tested points.

V-B Results and discussion

The table below compares the performance result for the two object profiles:

The two experimental results show minimal discrepancy suggesting similar performance in point localisation. Such an outcome provides validation that scanning on glass edge offers comparable reliability to scanning on opaque objects. Henceforth, we can make use of glass edge measurement reading as input to glass pose estimation.

VI Experiment 2: full pipeline

To validate the accuracy of the full task pipeline, an experiment is devised to measure the accuracy of the transparent object pose estimation. For visual guidance, we refer the reader to Figure 9.

VI-A Method

The following validation steps are performed following the steps in Section V:

Perform ICP on the previously obtained $n$ ${}^{B}\boldsymbol{p}_{\mathrm{BIP}}$ points to obtain ${}^{M}\boldsymbol{T}_{M^{\prime}}$ transformation matrix that transform $M$ to the estimated pose $M^{\prime}$ in the workspace. 2. 2.

For every $FK(\boldsymbol{q}_{\mathrm{ref}})^{\mathrm{flange}}\boldsymbol{T}_{\mathrm{tooltip}}$, we calculate the nearest point in $M^{\prime}$ as ${}^{B}\boldsymbol{p}_{\mathrm{near}}$. 3. 3.

Program the robot to move towards ${}^{B}\boldsymbol{p}_{\mathrm{near}}$ with $\boldsymbol{R}_{\mathrm{approach}}$ as approach orientation. Record the resulting joint states as $\boldsymbol{q}_{\mathrm{touch}}$.

The localisation error is defined as

[TABLE]

One can then define the mean and maximum errors across all discretised points along the glass border.

VI-B Results and discussion

In the following Table III, we list down the average operation time for each stage making up the full glue-dispensing operation with $n$=4:

While scanning and execution stages contribute significantly to the total operation time, the values presented in the table do not reflect the best time achievable as they are highly dependent on the robot’s velocity. In our experiment, we limit the velocity and acceleration of the robot for safety reason.

The table below presents the pose estimation result based on the validation metric mentioned previously with $n$=12:

VII Conclusion

In this paper we have looked into the feasibility of accurately locating transparent object using a 2D LRF with the ultimate goal of having a 6 DOF robot manipulating on it. Based on a physical understanding of laser-glass interactions, we introduced a method that combines 6 DOF robotic arm with 2D LRF to estimate a transparent object pose in the workspace by looking at the object edge. The method was tested in an industrial scenario which emphasises the need for fast and accurate online operation. Experiments showed that our method could achieve errors in the order of the millimetre, making it suitable for many industrial scenarios, such as manipulation or glue dispensing.

Future work will include integration with upstream coarse localisation. The accuracy can also be further improved by performing a more complete calibration of laser-in-hand robotic system[13].

Acknowledgment

This work was supported in part by NTUitive Gap Fund NGF-2016-01-028 and SMART Innovation Grant NG000074-ENG.