SYNOSIS: Image Synthesis Pipeline for Machine Vision in Metal Surface Inspection

TL;DR

This paper introduces a new pipeline for generating realistic synthetic images of metal surfaces to improve machine vision inspection systems.

Contribution

The novel contribution is a synthetic data generation pipeline with precise control over texture parameters for metal surface inspection.

Findings

The pipeline generates physically realistic textures with interpretable parameters.

The synthetic dataset includes sandblasting, parallel, and spiral milling textures.

Image similarity metrics between real and synthetic data correlate with downstream detection performance.

Abstract

The use of machine learning methods for the development of robust and flexible visual inspection systems has shown promising results. However, their performance is highly dependent on the large amount and diversity of training data, which is difficult to obtain in practice. Recent developments in synthetic dataset generation have seen increasing success in overcoming these problems. However, the prevailing work revolves around the usage of generative models, which suffer from data shortages, hallucinations, and provide limited support for unobserved edge-cases. In this work, we present the first synthetic data generation pipeline that is capable of generating large datasets of physically realistic textures exhibiting sophisticated structured patterns. Our framework is based on procedural texture modelling with interpretable parameters, uniquely allowing us to guarantee precise control over the texture parameters as we generate a high variety of observed and unobserved texture instances. We publish the dual dataset used in this paper, presenting models of sandblasting, parallel, and spiral milling textures, which are commonly present on manufactured metal products. To evaluate the dataset quality, we go beyond final model performance comparison by measuring different image similarities between the real and synthetic domains. This uncovered a trend, indicating these metrics could be used to predict downstream detection performance, which can strongly impact future developments of synthetic data.

Figures

Full text

1. Introduction

Automated machine vision has seen wide adoption in industrial applications to automate repetitive processes that rely on visual information. Visual quality inspection is a popular choice due to its versatility across different geometries and materials, cost-effectiveness, and inspection speed. It can be integrated in different stages of the manufacturing process, ensuring final quality, while minimizing the time and resource costs of inspecting an object. While manual visual inspection involving humans is a popular and often cost-effective choice for low-volume productions, an automated solution is more fitting for various scenarios, such as rapid production lines, continuous manufacturing, and extreme working environments (temperature, noise, chemicals, etc.). Although, machine vision systems are a flexible approach in terms of their variety of possible applications, they are highly specialized to the task at hand, designed using scarce expert knowledge and face adaptability issues under unforeseen inspection requirements or production environment changes.

Automated visual inspection systems (further referred to as inspection systems) are currently developed in a manner that requires a high level of rigidness to ensure efficient and reliable performance [1]. The most rigid component is the machine vision algorithm, which is commonly implemented using traditional techniques. While they produce interpretable results and can offer verification, they are specialized for a particular use case and are sensitive to changes in the environment. This, in turn, reduces the customization capabilities of production lines, slowing down innovation in manufacturing. To overcome this, the integrators of inspection systems are turning towards machine learning (ML) techniques for machine vision, which demonstrated higher flexibility to changes. However, their reliability and flexibility over longer periods of time is yet to be proven, making the industry skeptical towards their integration into production lines. Additionally, machine learning requires a significant amount of training data, which is a major challenge since the industry aims at reducing the frequency of defective parts, making such data scarce. The inspection tasks are typically very specific and, since there are little to no publicly available datasets [2], the task becomes time-consuming due to data collection.

To overcome these issues, practitioners are turning to synthetic data sources as a supplement to data shortages. In academia, the usage of synthetic data for ML is rapidly gaining popularity [3]. In industrial inspection systems, isolated studies have been performed providing evidence of its benefit. However, to date, there is a lack of commonly agreed upon best practices or a more comprehensive evaluation of the different steps that are required to generate a photorealistic synthetic dataset for visual inspection. Industrial inspection is facing scarcity of the datasets that are needed for benchmarking recognition models on surfaces with prominent microstructures, as well as dual datasets containing matching real and synthetic images for the development of domain alignment methods. To the best of the authors’ knowledge, no pipeline has been presented that can generate such data with a high level of realism and variety.

As such, we introduce the SYNOSIS (synthetic, optically realistic images for ML-based inspection systems) pipeline (Figure 1). It is designed to fill the needs for generating realistic and controllable synthetic data that are specialized to a specific need, and it provides a backbone for the extension of support for other textured surfaces (e.g., casting, grinding, and additive manufacturing). Our contributions are as follows:

• This is a novel pipeline for designing a realistic, interpretable and controllable data generator, for the low-data regime of uncoated textured metal surfaces (Figure 2).
• The definition of the geometry scale decomposition is to include the notion of a surface’s microstructure in the synthetic object modeling nomenclature.
• The definition of dataset quality decomposition is to provide a more interpretable description of the domain gap and its source.
• Publication of a dual (real and synthetic) dataset that provides data for common processed metal surfaces, facilitates domain gap analysis, and serves the wider research community with a novel pattern recognition challenge.

Paper Structure

The content is grouped into eight sections and contains bolded keywords to make the reading easier. The generation process that is needed to obtain synthetic images for surface inspection is covered in Section 2. There, we provide an overview of the related work, introduce the inspection-relevant decomposition of scales, and discuss the most important elements of the generation process and their relation to different scales. Section 3 and Section 4 introduce the methods chosen to produce test object surfaces in a controlled manner to measure surface topography. Section 5 and Section 6 focus on texture and defect modeling as the most relevant aspects that need to be controlled when creating synthetic data for surface inspection. Section 5 discusses the different approaches that can be taken to represent surface texture and their benefits or shortcomings. Further, the section introduces the stochastic geometry modeling approaches used for the specific surfaces found in our specific use-case. Section 6 discusses the different approaches used to defect generation in the related work, and provides an overview of the defect modeling workflow that is used for our specific use-case. Section 7 introduces how both the real and synthetic data were obtained. For synthetic data, it explains how the synthetic image generation process (introduced in Section 2) was used and with what parameters to create the synthetic dataset. Finally, Section 8 provides a discussion of the different means in which can be used to evaluate the quality of the created synthetic dataset, as well as the results when applied to our use-case.

2. Synthetic Dataset Generation

2.1. Related Work

The goal of synthetic image data generation is to generate image datasets where the image characteristics are ideally the same as those of images acquired in reality. The characteristics include both appearance and content distribution. When there is a discrepancy between the synthetic and real images in terms of their characteristics, it is referred to as the domain gap.

An early, straightforward, approach to extending training datasets was to add defect textures to the images of defect-free surfaces [4]. The defects are 2D image patches that do not correctly represent light interaction and have an inconsistent appearance with respect to the remainder of the surface. Due to the nature of the approach, it is very difficult to enhance it in a way that would minimize the resulting domain gap. Therefore, further synthetic data efforts were aimed at generative methods, based on predominantly deep learning, and rule-based approaches, using computer graphics simulation based on a well-defined set of rules that describe the imaging context [5].

The high degree of modeling complexity when generating photorealistic surface features caused many of the authors to adopt the generative approach, focusing on defect generation. Jain et al. [6] compare the performance of three different GAN architectures for generating defects in hot-rolled steel strips. Defect-GAN [7] was introduced for automated surface defect synthesis by mimicking the defacement and restoration process and capturing stochastic variation of defects. Defect-Transfer GAN [8] is trained over multiple products with similar defects in order to enable the introduction of semantically new information and, therefore, make up for the limited availability of all possible object attributes in the real dataset. Wei et al. [9] generated defect patches and blended them into a clean image, thus they were able to obtain approximate segmentation masks. These approaches are more suitable for generating large defects that visibly differ from the surrounding surface and do not suffer if an image is resized to a lower resolution for inference performance reasons. As such, they can be expected to struggle when confronted with a surface that has a texture pattern whose appearance has prominent local variations, as is the case for milled surfaces. Schmedemann [10] explored a hybrid approach by applying style-transfer on images generated by a rule-based approach to obtain more control and reduce the domain gap. It has shown that this approach can lead to a loss of features, making it unreliable without a human operator review of the generated images.

Generative deep learning approaches may result in images that look realistic but that also incorporate detail hallucinations or global structure errors, which can be difficult to detect by human eye [11,12,13]. These cause misalignments of the detector models and risk increasing the number of false negatives. Also, these methods offer limited reasoning over the parameters that generate a particular scene, severely limiting the precision control over them. Hence, it is difficult to control the dataset content and the distribution of features within, severely limiting the generation of the edge cases that are crucial for the reliable operation of inspection systems. In addition and in part due to data shortage, to date, no generative models capable of generating data for visual inspection exist up to the precise specification of the end product.

Rule-based image synthesis relies on computer graphics for modeling and rendering virtual environments, which have recently gained popularity beyond the entertainment industry (e.g., movies or gaming). It represents a controllable, versatile, and reliable approach for generating arbitrary amounts of data with customized features and variations. As such, it has proven useful across various machine vision tasks, as discussed by Dahmen et al. [14]. Synthetic data are currently predominantly used in human-oriented domains, autonomous driving, and robotics; see [15].

Human-oriented synthetic data are used in tasks that include body tracking and pose estimation [16,17,18] or face recognition [19]. Synthetic data for autonomous driving rely on virtual environments of large-scale urban and traffic scenes [20] for recognition tasks [21] and object detection [22]. The robotics domain requires virtual environments [23,24] for recognition tasks, such as semantic segmentation [25] and object detection [26]. In all three domains, the focus is on large scale geometric features while simulation of small scale details and textures is unnecessary.

Surface inspection belongs to industry automation, which is closely linked to robotics but includes tasks beyond those needed for movement, localization, and mapping. In particular, building virtual environments for inspection planning and the development of defect recognition algorithms require a high level of realism and small-scale surface details. This makes this domain highly different from the domains discussed above.

A general-purpose dataset generator was introduced by Greff et al. [27]. It promises realism (using physical-based ray tracing), scalability, and reproducibility, which are also covered by our pipeline. However, being a general framework, it requires a user to define a 3D scene for the particular task by using scripting. Assets for the 3D scene must be either created by the user or imported from general-purpose asset libraries. As such, data generation for specialized problems, such as quality assurance, is challenging due to unavailable assets, assets with unsatisfactory features, or limited modeling capabilities of a user.

Moonen et al. [28] presented a tool kit for synthetic image data generation in manufacturing. Synthesizing datasets for industrial inspection using a rule-based approach was further discussed in [29,30,31,32,33]. Synthetic data were further used for various quality assurance tasks, such as inspections of industrial components [34], metal surface inspections [35], industrial visual inspections [36], scaffolding quality inspections [37], or viewpoint estimation [38]. Raymond et al. [39] introduced models of scratched metal surfaces where they stacked layers containing different scratch distributions. However, their model does not represent curving scratches nor dents or scratches with varying geometrical depths. None of the mentioned works provide the level of control introduced by Bosnar et al. [40], with parameters designed to separately control image acquisition contexts, texture appearances, or defect characteristics.

The work presented in this paper builds on the virtual inspection planning research done by Gospodnetić [41] and Bosnar et al. [40,42,43]. Bosnar et al. [40] identified the core computer graphics components needed to perform image synthesis for surface inspection, and they stressed the importance of texture and defect parameterization to minimize the need for an artistic approach. However, it focused mainly on the concepts and not challenges that arise when applying it to a realistic scenario. We extend the aforementioned pipeline and apply it to a realistic scenario. Procedural textures and defects for generating synthetic data for defect recognition in visual surface inspection were introduced in [42,43]. Their defect modeling approach was also adopted by Schmedemann et al. [44] for the rule-based generated images before application of style transfer. Fulir et al. [45] also built on top of [40,42,43] to produce a synthetic dataset for defect segmentation on metal surfaces; however, they mainly discussed ML results.

2.2. Decomposition of Scales

A 3D scene contains 3D objects that are traditionally decoupled into geometry and material. The geometry specifies the size, position, and shape of an object (high-level appearance), while the material influences how the light interacts with its surface (detailed appearance). Alternatively, a 3D object can be considered at three different scales as follows: macro-, meso-, and microscale. Macroscale refers to object geometry (e.g., represented by a mesh) and large geometrical features such as geometrical defects. Mesoscale refers to the surface structure on a much smaller scale than the shape of the object but larger than the wavelength of light (i.e., surface texture), as discussed by Dorsey et al. [46] (ch. 2). Microscale refers to the surface structure and properties that are not separately distinguishable by the imaging sensor but are contributing to light reflection.

It is difficult to make a strict distinction between the scales using units because the scales are relative to the imaging sensor resolution and viewing distance. Therefore, we propose the following guidelines: The macroscale should be attributed to geometrical shapes that take larger portions of the image. The microscale can be determined using the Nyquist–Shannon sampling theorem [47], stating that the sampling rate must be at least twice the bandwidth of the signal. In other words, every feature that cannot be sampled by more than one pixel falls under the microscale class. Finally, the mesoscale constitutes features that occupy a smaller but visible portion of the image, which are measured in a handful of pixels.

To give some examples, assume that we were to observe an outdoor scene of a park containing people and trees. Then, the shapes of the people and trees would make the macroscale, whereas the eyes or leaves would already fall under the mesoscale, and the skin or the leaf veins would be considered microscale. In the case of visual surface inspection, we are looking at specific parts of an object with a high resolution. Therefore, the inspected object is in the macroscale, its defects and surface texture caused by the manufacturing process are in the mesoscale, while roughness and smallest surface features are part of the microscale.

2.3. Rule-Based Image Synthesis

Rule-based image synthesis (Figure 3) can be divided into two main steps as follows: 3D scene modeling (i.e., modeling of the virtual environment) and rendering procedures, as discussed by Greenberg et al. [48] and Tsirikoglou [15]. During rendering, the 3D scene is used to generate 2D images. The crucial element for achieving realism of those images is physical-based light transport, using path-tracing (Kajiya [49], Pharr et al. [50]). In this work, we used the Appleseed [51] rendering engine, which implements these principles.

2.4. 3D Scene Modeling

The 3D scene represents the real-world context within which the images are acquired. As such, the scene to be simulated must faithfully represent the environment. For surface inspection, this includes 3D objects, lights, and cameras. Note that we do not necessary include environment illumination since inspection systems often have a strictly controlled acquisition environment and no uncontrolled illumination. While all the components contribute to the image’s realism, in this work, we focus on the realism of a 3D object, influenced by the geometry, material, and surface topographies (i.e., texture).

The geometry of the 3D models representing the inspected object (macroscale) must correspond to the geometry of the real inspected product. In this work, we use the 3D models of the test bodies (Section 3) and create defected product instances by augmenting defects (mesoscale) directly into the product’s geometry [43]. This is done because the defects are larger than the surface texture, and their complex shape causes intricate light scattering that can not be captured by normal maps alone, as described in Section 6.

Material properties describe the light–surface interaction and are modelled using the bidirectional reflectance distribution function (BRDF), as discussed by Dorsey et al. [46]. For this work, rough metal surface BRDF is used (Walter et al. [52], Kulla et al. [53], and Turquin et al. [54]). The BRDF is evaluated in every surface point based on the local surface normal. Unrealistically smooth surfaces are avoided through variations in the BRDF parameters and normals using texture.

Texture is a term for which the exact definition differs across domains. Here, we refer to the computer graphics concept that defines it as a procedure to introduce the surface topography by influencing (perturbing) the surface normal and BRDF parameters for every surface point at the rendering time (Mikkelsen et al. [55]). As such, the texture describes the mesoscale surface patterns, which would otherwise be very inefficient if they were represented by the geometry. Section 5 offers a more complete overview of the texture modeling approaches. Traditionally, texture modeling is aimed to be used by artists to approximately recreate a photorealistic visual appearance and enable their creative expression (Guerrero et al. [56], Adobe Substance [57], Ebert et al. [58]). Therefore, the surface models are based more on artist-oriented parameters rather than real-world parameters, making them visually appealing but not necessarily correct. Dong et al. [59] approached realism by fitting a Gaussian random field to surface measurements and using it as a texture. It gave impressive results for a single instance of measurements, but offered no further control over the pattern’s properties.

In contrast to [59], we recreated the surface measurements by using mathematical models that can be controlled based on real-world parameters (see Section 5). The textures are generated as 2D images, encoding the normal variations, which are then mapped onto the surface of the object, as explained in Section 2.5. Given the high level of details produced by the models, we kept the BRDF roughness parameter fixed to constant small values (e.g., within a range (0.0–0.1)).

Light in the 3D scene represents the illumination devices used in the surface inspection environment. We used a model of the ring light geometry used in the real inspection system, which has a torus-like shape (Figure 3) with diffuse emission distribution describing the light emission in each point of the shape. As in the real setup, the camera was placed in the center of the ring, with both objects having the same orientation.

The camera used a pinhole camera model, defined by its resolution, pixel size, focal length, position, and orientation parameters. Simulated camera parameters are set to resemble the real camera parameters used in the surface inspection environment. The pinhole model does not account for the depth of field or optical aberrations, such as distortions. For the purpose of this work, this is sufficient since the inspected surface is expected to be in focus, and effects, such as image distortion or global blurring, can be introduced in post-processing or during training in the form of image augmentations.

2.5. Texture Mapping

The textures used in this work are represented as normal map images, where each pixel encodes a normal vector. While [60] suggests that the implicit procedural generation is ideal, it requires an additional level of restrictions and complexity when developing the new models. In particular, it is the limited spatial information that makes describing spatially varying patterns with regular global structures challenging. In this work, the milling patterns (Figure 2) pose a great challenge, and the sequential generation of rings on-the-fly would be intractable for implicit procedural generation. While the implicit implementation is possible, it is outside the scope of this work in order to allow us to focus on physically realistic texture modeling.

In order to use the texture images together with a BRDF evaluation during rendering, it is required to map them onto the 3D object. Typically, it is either done by using projection directly [50] (e.g., planar, cube, or spherical) or by unwrapping the object [61] (i.e., mapping the object faces to a plane). Applying the texture projection directly onto the whole object will cause artifacts, such as texture stretching, for complex geometries. This is not acceptable when generating synthetic training data. Therefore, in this work, we first separated the geometry into main planar surfaces, and then we used planar projection separately per surface. To avoid the occurrence of texture tiling (visible repetition in pattern), the texture images were generated large enough to cover the entire target surface.

2.6. Rendering

The 3D scene description was used to render the synthetic images from all given camera positions. The main rendering parameters influencing the quality of the synthesized images were the number of samples per pixel (SPP) and the number of light ray bounces.

In surface inspections, each pixel covers rich and complex surface structures. A hHigher SPP ensures that the computed pixel color is a good approximation of the mean light response value of the covered area. As such, increasing the number of SPP will reduce the noise and aliasing in the rendered image.

In addition to texture, our 3D objects contained defects in the form of small, complex, geometrical features. Increasing the number of light bounces (i.e., the times that the ray is allowed to bounce off the intersection point) enables us to represent complex light phenomena, such as shadowing, masking, or interreflections. These phenomena can not be correctly represented using normal maps alone.

2.7. Defect Annotations Generation

The object geometry used in the 3D scene is defected using methods based on work by Bosnar et al. [43]. Each imprinted defect geometry has a corresponding geometrical defect mask; see Section 6. Black (non-emissive) material is applied on the 3D object, emissive material is set to the geometrical masks, and all light sources are disabled. Finally, the annotations are generated using the same camera parameters as for the corresponding photorealistic image, which ensures that the pixel-precise labels of the defect areas are visible from the camera position and not occluded by the 3D object. This approach was introduced by Bosnar et al. [43] and will always generate defect annotations, regardless of their light responses, as can be seen in Figure 4.

3. Test Body Design

Injection of the domain knowledge is crucial for developing photorealistic models. Therefore, we used custom designed and manufactured test objects, giving us complete insight into and control over the processing chain. This allowed us to focus on the image synthesis, surface modeling, dataset generation, and training challenges.

In total, three identical sets of test objects were fabricated, each containing 10 polyhedral test bodies. When designing the test bodies, the aim was to generate bodies with multiple faces that showed different realistic surface texture patterns.

The base bodies (type B and type M) are polyhedrons that are made from aluminum; see Figure 5a. Each polyhedron has a size of 5 cm × 5 cm × 5 cm with the planar faces A, B, and C having approximate sizes of 2 cm × 2 cm, which are measured and modeled subsequently. The remaining faces are left “unfinished” (rough milling) and are not further considered. The difference between the type B and type M is the mirrored arrangement of the faces of the base’s body.

First, the surfaces were milled using different processing parameters (Figure 5c). A rotating cutter with a specific head diameter was used to remove material from the surfaces by performing many separate small cuts along the tool path with a defined feed rate. We considered face-milling, which means that the tool moves perpendicularly to the object’s surface. The distance between the neighboring tool paths depends on the radial depth of the cut and influences the resulting tool marks; see Figure 5b. Furthermore, parallel linear and spiral tool paths were used. Artifacts in the form of exit lines can occur with spiral milling.

After milling, some of the faces were sandblasted—a process in which a hand-held nozzle of a sandblasting tool blasts the surface with a mixture of sand and air at a high pressure. This abrasive method can smoothen or roughen the surface. The resulting surface topography depends, thus, on the pressure of the air jet, the grain size distribution, and the shape and material of the sand. The parameter values of both processing techniques (see Table 1) were chosen so as to obtain the realistic patterns that were expected to emerge from industrial processing.

Objects in set 1 contain no surface defects, while sets 2 and 3 contain more than 200 scratches and localized point defects, created precisely on the faces. For this purpose, a custom “indenter” and a scratch test tool were used, where the tips can be loaded with different masses. For the tips, we used a diamond tip, a steel needle, and a screw. The loads of the tips were 500 g, 1000 g, and 1500 g. Using the tools, the scratches (straight and free-form lines) and digs (dents) were realized on the surfaces as typical surface defects in accordance with ISO8785 [62]. The generated defect sizes were in the range of sub-millimeters to millimeters. See Figure 6 for an example of the defect specifications and Figure 7 for the examples of the imaged test bodies’ appearances.

4. Material Measurements

In order to characterize the machined surfaces, optical 3D and topography measurements were performed by means of focus-variation microscopy (Bruker alicona InfiniteFocus G4, Bruker, Billerica, MA, USA). In the high precision machining industry, measurement systems based on the focus-variation technique are standard inspection tools to characterize 3D mechanical parts. The advantage is the high precision and contact-free measurement of roughness, surface structure, micro-geometry, and form by using one optical sensor. The measurement system is based on a precise optical lens system with shallow depth sharpness. By changing the working distance between the measured surface and the microscope lens, different depths of the surface come into focus and are projected sharply onto the sensor. The proprietary software analyzes the distance for each point and measures the sharpness, which is used for the calculation of the surface depth profile (topography).

Depending on the magnification of the optical lens system, this process allows a vertical resolution down to 10 nm (a magnification equivalent to 100×, which is a physical magnification limit). For the purpose of this work, an optical lens system with a magnification of 5× and a nominal vertical resolution of 410 nm was used. The vertical resolution was limited during data acquisition by the measurement software. The defect-free samples of objects in test set 1 were measured with different lateral resolutions, depending on the size of the scanned surface area, as it is shown in Figure 8. The topography measurements are converted into ASCII xyz-files in order to create readable files used for surface modeling. This results in height images, where each pixel is assigned a height value.

Depending on the process parameters, the tool marks created by the milling process form a unique pattern that can be observed by the naked eye (Figure 5b) and influences the subsequent surface light response, playing a key role in surface inspection. Measurements of the surfaces that were processed using different parameter settings are given in Figure 9.

When sandblasting, the previous milling does not affect the final surface pattern; see Figure 8b. The process results in a homogeneous surface texture without long scale spatial relations.

In order to characterize the generated defects, we used an optical lens system with a magnification of 20× and manually adjusted the vertical resolution during data acquisition through the software. The depth quality filter value was set between 2e-6 and 8e-6, depending on the defect type and size (e.g., Figure 10). The shape of the dent resembles a dig with a peak and a valley of about 150 µm in height and depth, with respect to the undamaged surface level.

5. Texture Modeling

Due to the very different surface topographies (Figure 8), separate models are required for sandblasted and milled surfaces. Models are selected based on the corresponding topography measurements $[eqn]$ , and used to simulate the mesoscale of the object (texture). For completeness, in this section, we provide an overview and description for the process of modeling a procedural texture for the aforementioned surface microstructures, relying on the detailed process description presented by Jeziorski et. al. [63].

Measurements of the test body surfaces are provided as 2D height images $[eqn]$ of size $[eqn]$ for $[eqn]$ and pixel spacing $[eqn]$ . The output of the models is a texture image $[eqn]$ of arbitrary size $[eqn]$ and pixel spacing $[eqn]$ coarser or equal to the input’s pixel spacing. For the purpose of this work, we use $[eqn]$ μm and $[eqn]$ so that a squared imaged region with an edge length $[eqn]$ is provided. The chosen pixel spacing is large enough to generate images of appropriate size for which the rendering process can still be done in acceptable time. On the other hand, pixel spacing is chosen small enough so that no visible discretization artifacts occur. The resulting texture image can completely cover the test body surfaces during the rendering process, even when the texture image is rotated. This way, image tiling is avoided, as it may produce visible edges between the tiles.

5.1. Related Work

In computer graphics, the textures representing surface topographies can be roughly separated into procedural and data-based, see Figure 11. In this work we used explicit procedural for the milling texture and an exemplar-based approach for the sandblasted texture.

Procedural texture modeling is based on algorithms that completely describe the surface patterns without the need for additional data (e.g., image data). The main concept is to combine regular patterns (e.g., sine functions, simple shapes such as lines and circles, as well as other mathematical functions) discussed by Vivo et al. [64] with irregular patterns (e.g., noise functions), as discussed by Dong et al. [65]. Since procedural texture models are algorithmic descriptions of a texture, they offer parameterization and, thus, immense controllability, which is recognized by the computer graphics community for content creation [66]. However, developing procedural textures is often done by artists who base their work on experience and expression rather than encoding physically correct parameters.

Deguy [66] separates procedural models into implicit and explicit. An explicit procedural texture model generates a texture image that is first mapped onto the object’s surface and then used during rendering. Contrary, an implicit procedural texture model is directly evaluated during the rendering procedure, thus generating texture in render-time and allowing render-time re-parametrization. While explicit procedural models are easier to develop for complex texture patterns, implicit procedural texture models can cover arbitrarily large surfaces without repetitions or seams and are not restricted by the resolution. Bosnar et al. [42] developed implicit procedural texture models for circular, parallel, and radial textures. Those patterns resemble processed surfaces but without taking an explicit machining method into account or basing them on real surface topography.

Data-based texture modeling relies on sensor data and measurements as input, as discussed by Tsirikoglou et al. [15]. For example, the input texture image may be obtained using photogrammetry [67] or other scanning techniques [68].

The advantage of data-driven texture modeling is that it can synthesize approximations of real-world surfaces for which there are no well-defined mathematical descriptions. However, those methods offer little control over the generated content since they depend entirely on the input data. Data-based texture modeling can be roughly separated into generative AI and exemplar-based approaches. Generative AI texture synthesis is often based on GANs, as discussed by Jetchev et al. [69], Bergmann et al. [70], and Zeltner et al. [71]. While the results of these models often have remarkable appearance to a person, it is not possible to ensure that the results are physically correct, and it is very difficult to introduce controllable parameters for different types of textures. Exemplar-based texture synthesis uses algorithms to generate new texture images similar to a given input image. While stationary textures can be well reproduced, problems occur for textures with complex, large structures, e.g., milling rings.

Furthermore, since the output image relies entirely on the input image, it leaves no possibility of influencing the pattern itself. Raad et al. [72] gave an overview of the common exemplar-based texture synthesis methods. They distinguished between statistical and patch-rearrangement methods.

Statistical exemplar-based methods perform two steps. First, model-dependent statistics of the input image are computed. Second, a randomly initialized image is adjusted such that it fits the observed statistics. The asymptotic discrete spot noise (ADSN) and the random phase noise (RPN) are both parameter-free and non-iterative statistical methods [73]. A texture image simulated by the ADSN maintains the mean of the input image but has minor differences when comparing the sample auto-correlation since its Fourier modulus is the Fourier modulus of the input image multiplied by Rayleigh noise. The RPN creates texture images that have the same Fourier modulus as the input image and, thus, the same auto-correlation but with a random Fourier phase. Heeger and Bergen [74,75] introduced a procedure for texture modeling that uses image pyramids. A random image is adjusted iteratively by matching the histograms of all images in its image pyramid to the corresponding images in the image pyramid of the reference image. Portilla and Simoncelli [76,77] improved this method by using statistics such as cross-correlations between pyramid levels instead of histogram matching. In contrast to the exemplar-based methods that generate completely new texture images, the patch-rearrangement methods quilt patches taken from the input image. With the method of Efros and Freeman [78,79], the output image grows successively by adding certain patches one after the other in a raster-scan order. It is an extension of the method of Efros and Leung [80], which generates a new texture image pixel by pixel.

Guehl et al. [81] introduced a hybrid approach combining procedural and data-based texture synthesis. The visual structure of the generated image texture is based on a procedural parametric model, while texture details are synthesized using a data-driven approach. Another hybrid approach is presented by Hu et al. [82], in which the texture of the input image, which contains the BRDF parameters, is decomposed and proceduralized.

Modeling of metallic surfaces in computer graphics has received a lot of attention. Particularly interesting methods are the ones describing highly specular surfaces with high-frequency details (glints), as discussed by Zhu et al. [83] and Chermain [84]. Glint effects appear due to complex and unstructured small-scale, high-frequency geometries, such as tiny bumps, dents, and scratches. The problem with these approaches is that they do not model the actual properties of surfaces to resemble any particular and standardized surface. Rather, they are modeled to resemble the surface finishing appearance in an artistic way that is difficult to compare to any particular real-world counterpart.

Milling belongs to the machining processes resulting in deterministic and describable surface patterns [85,86]. Therefore, it is extremely useful to incorporate existing domain knowledge into texture models for milled surfaces. To estimate the surface quality of milled surfaces and, thus, determine the required quality, Felho et al. [87,88] and Kundrak et al. [89] developed models for parallel face-milled surfaces. Hadad et al. [90] developed a more detailed model that takes other typical milling parameters into account, e.g., tilting of the tool to avoid re-cutting. All models create milled surfaces using CAD software, and the patterns are, thus, geometrically imprinted into the surface. Oranli et al. [91] provided a similar approach to simulate the physical process of sandblasting and the resulting surface deformation in high precision using the Abaqus/Explicit 2019 software. Using such an approach in data synthesis would result in unfeasibly long computation times. Furthermore, the surfaces generated in this way look too perfect, as possible irregularities, e.g., slight deviations in the cutting geometry due to irregular material and machine behavior, are not included in the models.

5.2. Sandblasted Surface

The surface topography shows a different range of height values and degrees of roughness, depending on the pressure of the air jet (Figure 8b). The resulting surface topography is homogeneous, resembling a stationary Gaussian random field. However, fitting parametric Gaussian random fields did not provide satisfactory texture images since the structure sizes were not reproduced correctly. In addition, due to the small number of process parameters, it is difficult to develop a model that depends exclusively on them. Therefore, we used a combination of exemplar-based texture synthesis methods that only receive the measurements as input. We selected measurements with sizes between $[eqn]$ and $[eqn]$ and pixel spacing $[eqn]$ µm that correspond to an area of about $[eqn]$ .

To generate texture images for sandblasted surfaces, the asymptotic discrete spot noise (ADSN) introduced by Galerne et al. [73] was used. The size and pixel spacing of the input image were maintained. To obtain a new texture image by simulating the ADSN, a sample from the multivariate normal distribution $[eqn]$ is taken, with $[eqn]$ being the sample mean and $[eqn]$ the sample covariance matrix of the input image.

Next, the desired size $[eqn]$ and pixel spacing $[eqn]$ of the texture image have to be met. The pixel spacing can be adjusted by downsampling the input image using nearest neighbor interpolation. ADSN can generate texture images larger than the input image while maintaining the same pixel spacing [73]. Therefore, the normalized input image $[eqn]$ is enlarged using padding with mean value $[eqn]$ before applying the ADSN. However, periodical repetitions are visible in the resulting texture. To avoid this, multiple texture image patches are generated and stitched using EF-stitching [78,79]. To stitch two neighboring patches within an overlap region, both are cut apart along an intersection edge first and put together afterward. The intersection edge is determined by finding the path within the overlap region where both images are most similar. Depending on the alignment of two neighboring patches, we distinguished between vertical, horizontal, and L-shaped stitching (Figure 12); see [63] for details.

5.3. Milled Surface

The model generating textures of milled surfaces is given as a function in a continuous domain $[eqn]$ . Thus, it is procedural but implemented explicitly for the sake of simplicity. The discrete texture image is obtained by evaluating the function at the image grid points $[eqn]$ for $[eqn]$ and $[eqn]$ . The milling pattern appearance depends on a number of parameters associated with the production parameters. The most prominent structures are the cycloidal edge paths caused by the rotation of the milling head. They are approximated by the $[eqn]$ rings $[eqn]$ (Figure 13a). The model is further divided into the following sub-models for the:

1. Tool path providing the arrangement and order of the rings by defining their center points $[eqn]$ ;
2. Appearance of an individual ring $[eqn]$ (Figure 13b);
3. Interaction between neighboring rings (Figure 13b).

For $[eqn]$ . The heatmap scale is omitted, as the method is agnostic to the vertical scale.

The first sub-model generates the ring center points $[eqn]$ along the tool path. Within this study, either parallel or spiral milling (Figure 14) is considered. First, the respective tool path has to be modeled, together with the distance $[eqn]$ between two neighboring sub paths. The distance is controlled by the amount of ring overlap $[eqn]$ and the diameter $[eqn]$ of the rings, as $[eqn]$ . The diameter of the milling head is crucial for defining the ring diameter, and the ring overlap depends on the radial cutting depth $[eqn]$ by $[eqn]$ . The radial cutting depth defines the distance that the tool steps over in relation to the ring diameter. As the blades do not reach the edge of the tool, $[eqn]$ is reduced here by taking $[eqn]$ with $[eqn]$ . The ring center points $[eqn]$ are then modeled by random displacements of points $[eqn]$ that are computed along the tool path. That is, we sample from a two-dimensional normal distribution $[eqn]$ . The location of $[eqn]$ is specified by the distance $[eqn]$ between two center points along the tool path. The distance $[eqn]$ depends on the tool’s feed rate and rotational speed. It is important to keep the chronological order of the rings in order to keep the tool path visible. Finally, while being very precise, the milling process is not perfect, and every once in a while, there is an occurrence of a ring that is more visible than the others. To introduce such irregularities in visibility of rings, the order of a certain amount $[eqn]$ of rings is changed.

The second sub-model controls the appearance of an individual ring $[eqn]$ , produced by a single rotation of the milling head cutting edge. Therefore, the width of the cutting edge is crucial for the indentation width $[eqn]$ . To obtain the height values, the shape of the indentation is modeled by the cosine function $[eqn]$ for $[eqn]$ scaled to the width of the ring. Material depositions are modeled by accumulations beside the indentation in the form of the rings having positive values. The width for the inner and outer rings are $[eqn]$ and $[eqn]$ , and the height component is, again, modeled using the cosine function. All functions are combined to form the shape $[eqn]$ of ring $[eqn]$ (Figure 15).

Thus far, perpendicularity between the milling head and the surface has been assumed. However, the milling head is often tilted forwards in direction of the tool movement to prevent re-cutting the surface. This direction is given by the angle $[eqn]$ in the x-y-plane between the tool path and the x-axis at point $[eqn]$ . Tilting introduces a slope within the rings defined by the minimal $[eqn]$ and maximal $[eqn]$ indentations and inner and outer accumulations at the ring’s front, respectively, back for $[eqn]$ (Figure 15). Since the tool’s presence prevents pushing material to the inside, $[eqn]$ and $[eqn]$ are chosen small. Tilting $[eqn]$ is then applied by multiplying with the shape $[eqn]$ .

Finally, noise $[eqn]$ is added, simulating the irregularities caused by resistances in the material or vibrations during the process. It is modeled by a combination of $[eqn]$ sine curves with varying frequencies $[eqn]$ and random shifts $[eqn]$ for $[eqn]$ . Here, $[eqn]$ denotes a Poisson and $[eqn]$ a uniform distribution. Thus, $[eqn]$ defines the sub-model for the appearance of one individual ring (Figure 15).

The third sub-model deals with the interaction of the neighboring rings. First, the rings are successively added into the model, according to their order by using a recursive function $[eqn]$ for $[eqn]$ and $[eqn]$ (Figure 16). Since every part of the surface is processed several times, all associated rings must be taken into account, with most of the weight on the temporally last ones. This is achieved by a convex combination. Weighting parameters are chosen as the linear function $[eqn]$ , which gives more weight to the front part of the ring than to the back. It is zero for points outside the ring. Values at the front and back of the rings are given by $[eqn]$ and $[eqn]$ .

We refer to [63] for more details on the sub-models. A summary of the model parameters is given in Table 2. Some parameters are known explicitly (Table 1), while the others are estimated by visual comparison using high resolution topography measurements (Figure 8a). Simulations of the model using distinct parameter configurations are shown in Figure 17. Height values of the preliminary texture image $[eqn]$ are adapted so that their distribution resembles that of the corresponding measurement $[eqn]$ . Mean values $[eqn]$ and sample variances $[eqn]$ , $[eqn]$ are fit by setting $[eqn]$ .

6. Defect Modeling

In this work, we used the approach of Bosnar et al. [43] to generate multiple instances of defected object geometry, overview of which is provided in this chapter.

The defected object geometry is created through procedural geometry modeling of dents and scratches, thus ensuring correct light scattering. The three main steps are (1) generation of defect positions, (2) generation of 3D geometries representing the defecting tools, and (3) imprinting the defecting tools into the surface and creating geometrical masks, which are used for generating defect annotations. Defect positions are determined by sampling points from an arbitrary distribution on the inspected product geometry. In this work, we used normal distribution. The defects are embedded into the geometry using defecting tools, i.e., negatives of dents (denting tools) and scratches (scratch tools). Denting tools are typically spherical and parameterized for controlling their scale, rotation, and shape. Scratch tools are created by performing a random walk on the product geometry, resulting in a path that is further converted into a solid geometry. The parameters are the thickness and curviness, which control the size and shape of the scratches.

The defecting tools are imprinted into the inspected product mesh by using a Boolean difference operation. Geometrical masks, used for annotation generation (see Section 2.7), are then created by using the inspected product mesh and the defecting tool. First, an intersection between the product mesh and the defecting tool mesh is computed. The resulting mesh represents a solid filling of the defect. The solid filling might be useful when the annotations must label missing areas of the object. In this work, we labeled only the visible defected surface and, therefore, performed an additional step that results in a mask shell. The defecting tool was first scaled with a factor in the range of [0.9, 1.0] and then it was intersected with the solid from the previous step.

The defect parameters can be chosen to match the shape, size, and positions of the real defects observed in the measured defect topographies. However, more importantly, the parameter range can also be extended to produce defects beyond those observed in the available samples, i.e., rare occurrences or edge-cases. This enables the creation of an arbitrary number of defected 3D objects containing a wide variety of possible defects (see Figure 18) for which pixel-precise annotations are generated.

7. Dual Dataset

Publicly available datasets for inspection of metal objects mostly focus on defect recognition over flat surfaces and single-view inspection setups [92]. With it, they reduce the expected appearance diversity of defects and surfaces in the image, making the defect recognition task easier. However, such a restriction is not possible for an inspection of more complex objects where multi-view imaging is required and the visibility of the defects varies between different views. The aforementioned datasets contain objects, such as hot-rolled steel [93,94,95,96], slightly curved metal pads [97,98], rails [99], and pipes [100]. Datasets of smaller metal objects with complex geometry are also present; however, they are also under single-view inspection setups [101,102]. Multi-view inspection setups inspect the same surface multiple times while varying, at least, the direction of the incoming light source or the direction of the camera so as to reduce the chance of missing defects due to low visibility. Examples in recent datasets include single-axis rotation [103], multiple illumination directions [104], and multiple view directions [45].

Synthetic datasets for inspection exist in the form of generic stochastic textures [105], outdoor maintenance scenes [30], multi-object pose estimation task [32], industrial part recognition [106], and surface defect recognition [45].

We present a dual dataset consisting of a real and synthetic part, following the multi-view inspection setup described in [45]. Our dataset focuses on recognition difficulties arising from complex reflectance of different surface finish patterns that may cause defect curtaining.

7.1. Real Dataset

The real dataset was acquired using a setup consisting of a robot manipulator, a matrix grayscale camera, a diffuse ring light, and an acquisition table. A thick black curtain removed any influence of the environment illumination or reflection from within the acquisition box. The manipulator was used to position the camera and illumination onto predefined viewpoints. The acquisition table was a flat surface covered in diffuse black velvet, and the acquired object was manually placed on top. The acquisition viewpoints were defined using V-POI [107] as an inspection plan, relative to the 3D model of the acquired object. Our acquisition plan consisted of viewpoints focused on the center of each inspected object face (faces A, B, and C), at angles 0°, 10° and 20° from the respective surface normal, as visualized in Figure 19. The viewpoints were at a distance of 140 mm from the object’s surface, in optical focus, and acquired in the size $[eqn]$ .

To minimize the placement error between the expected and actual position of the object, the operator must manually perform object positioning. This routine uses a predefined reference viewpoint—in our case, the viewpoint was perpendicular to the C-face—and a synthetic image of the object was taken from it. The physical camera was placed into the position defined by the reference viewpoint, and the reference synthetic image was overlaid. The object was then aligned manually to make the observed image match the reference image. For this procedure to be appropriate, either the synthetic image must contain image distortion introduced by the lens or the real image must be undistorted.

The real dataset consists of images of the test bodies defined in Section 3, acquired using the setup above and manually annotated using LabelMe [108]. The images were resized to a size of $[eqn]$ so as to reduce the downstream costs while maintaining meaningful defect appearance clarity. The dataset contained 30 objects, 20 of which were defective, and with nine viewpoints per object. Overall this amounts to 270 annotated images, 180 of which display defects. The dataset was split into train-val-test sets. The test set contained 10 non-defective objects and 10 defective objects. The train and validation sets contained the remaining 10 defective objects, where only one viewpoint per object was used for the validation set.

7.2. Synthetic Dataset

The synthetic dataset generation environment described in Section 2 was developed to match the physical acquisition setup. The dataset generator is an automated procedure which uses object, texture, and defect specifications for generating annotated synthetic data in multiple stages. The first stage of the dataset generator is geometric defecting, as described in Section 6, which generates and applies defects to the original object geometry. We generated 30 defective synthetic object instances through stochastic sampling of the defect locations and specifications defined in Table 3. The defect locations were sampled independently over the inspected surfaces using uniform barycentric face sampling. Finally, we grouped our generated defects into the classes ’dent’ and ’scratch’.

The second stage produced texture images of the object surface using the methods introduced in Section 5. For each generated texture instance, the texture parameters were sampled at random from the values listed in Table 4, thus producing controlled surface appearance variations. The ranges were chosen empirically by observing differences in the physical samples to model plausible variations in the textures, such as milling at different translational speeds or amplitudes of parallel and orthogonal tool vibrations. For each of the three surface finishing methods, we generated 5 different texture instances.

The third stage combined the pre-generated synthetic objects and textures to produce annotated inspection images, as described in Section 2. A set of synthetic image-mask pairs were rendered for every synthetic object, following the same pre-defined inspection plan. Before rendering, each object surface was assigned one of the pre-generated textures at random. Additionally, each time a texture was assigned, it was randomly rotated by an angle in the range $[eqn]$ and translated by 5, 3, or 1 pixels for faces A, B, and C, respectively. To model different surface oxidation levels, for each synthetic object, we uniformly sampled the material’s roughness parameter from the range $[eqn]$ . To produce a balanced dataset, we generated the same amount of image-mask pairs by re-using the correct object geometry instead of the defective ones.

Overall, the synthetic dataset consists of 30 correct and 30 defective object instances for each of the 10 physical objects. To remove the bias arising from the location and choice of defects when comparing the results between textures, we re-used the same 30 defective object geometries across all the physical objects. Each object was inspected using the same 9 viewpoints used in the pre-defined inspection plan, totaling in 5400 images, 2700 of which contained defects. The images were rendered using 256 SPP and 8 light bounces at a resolution of $[eqn]$ , with every object instance taking on average of 25 min to render with labels. The train-val-test set was produced, ensuring label balance in a ratio of 70:10:20 by splitting it along the dimension of object instances.

A selection of images from our dataset is shown in Figure 20.

8. Pipeline Evaluation

To evaluate the quality of the pipeline, we have to evaluate the similarity of the data tht it produces to the real data acquired through inspection. Various metrics were proposed in recent works to estimate the quality of a dataset [109] through general dataset factors, such as annotation consistency, or class balance. However, these metrics usually provide limited insight into the data coverage of the dataset and ignore domain- and task-specific factors that could describe biases taken when collecting the dataset. Another approach for comparing the quality of synthetic datasets to the real ones is by measuring their utility for the given task through the model performance changes when using the synthetic data [15,44,45,106,110,111,112]. This, however, includes the inductive biases introduced by the model and training procedure selection, making it more difficult to estimate the quality of the dataset itself. We, thus, performed a more direct comparison of the datasets through image and data distribution similarities. When modeling a synthetic dataset, assumptions about the real world are made to increase the feasibility and economy of generating such datasets over the collection of more real data. To organize our evaluations, we proposed a distinction between a priori quality, evaluating the accuracy of the assumptions made before the data generation, and a posteriori quality, evaluating the accuracy of the generated synthetic data. To the best of the authors’ knowledge, this is a first evaluation of synthetic data quality that goes beyond the comparison of recognition model performances.

8.1. A Priori Quality

The a priori quality of a synthetic dataset quantifies the realism and precision of the environmental models used to generate the images compared to the target domain. It is reported through assumptions used to define parameters for the data generation and can be described prior to generating the final synthetic dataset. In our application, assumptions made to generate the dataset are tightly coupled to the physical processes used to manufacture and inspect the object. Using the same CAD model as used in manufacturing of the object, we guarantee the produced images will represent the correct geometry and restrict the data to the object of interest (Section 2.5). Using textures modeled to match the real surface measurements, we restricted the recognition task to the nominal patterns present in the real objects (Section 5). Where possible, the texture parameters were directly derived from the processing parameters. All parameters were sampled around the nominal values to increase model robustness towards variations in the surface texture (Section 7.2). Defect randomization contains ranges of parameters such as the expected defect dimensions and shapes that cover the defects that are present in the real objects, as well as the defects outside the observed range, for better generalization (Section 7.2). By using physics-based light simulation, we ensured that the images were produced realistically, holding the assumptions used for the material models and the rendering processes defined in Section 2.6. The simulated acquisition hardware (camera and light) uses the real acquisition hardware parameters to resemble the real inspection environment.

8.2. A Posteriori Quality

The a posteriori quality of a synthetic dataset describes the similarity of the generated data to the data collected in the target environment. This type of quality estimation relies on the established concepts of domain shifts actively used for methods of domain adaptation [113]. We categorized these qualities into domain similarity and task similarity.

8.2.1. Domain Similarity

Domain similarity measures the similarity between the distributions of real and synthetic images to quantify the covariate shift. In our case, we measured the distance between the generated images and the collected real images acquired using different types of metrics.

Value-wise metrics compare the global distributions of intensities between images. Distributions of image intensities can be summarized using histograms and can be compared using a suitable metric [114].

Pattern-wise similarity compares the local patterns between the images. This type of comparison is affected by the scene structures, e.g., patterns of the surface finish, the light source shape, and the material reflectance function. Handcrafted image descriptors can be used to measure the similarities of specific aspects of patterns in images, such as pixel-wise closeness and alignment (e.g., the mean absolute error, root mean squared error, and cosine similarity) or statistical local similarity (e.g., SSIM [115], peak signal-to-noise ratio, Adapted Rand [116], and GLCM [117]). Learned feature extractors alleviate the need to manually select specific features in images and instead rely on a very diverse dataset and deep learning to train a general purpose feature space for pattern and shape comparisons (e.g., LPIPS [118] or FID [119]). However, due to inherent differences between natural (e.g., dogs, traffic, people, etc.) and technical images (e.g., surface inspection, manufacturing line, product appearance, etc.), the generality is still bound to the imaging context. To expand that context, one would need a large amount of representative technical images, which is not publicly available.

We evaluated the following metrics: Wasserstein distance between image histograms (HistWD), the mean absolute error (MAE), SSIM, and LPIPS, using ImageNet pretrained AlexNet [120]. We limited our study to a single representative metric from each aforementioned subgroup in regard to their popularity in image quality estimation, interpretability, and implementation availability. Our experiments will prove sufficient to discuss the influence and shortcomings of different metrics (Section 9.2). To make the results easier to read, the metrics were normalized to the range $[eqn]$ and flipped when necessary to report the degree of similarity (see Table 5). Our reasoning is that metrics that report the highest differences in the dataset (excluding trivial metrics) account best for the small variations between the images in our datasets.

To compare our synthetic images to real ones, we needed to remove the influence of background noise and minor offsets that were present from the imprecision of the acquisition system and background scene structures. We first masked out all the regions that did not contain the observed object face, thus removing the influence of sharp edges between the dark background and the object. For each viewpoint, we constructed a common meta-mask from the intersection of face masks corresponding to that viewpoint. To reduce the difference in intensity levels, we manually estimated the parameters of a linear transformation to bring the histograms between matching real and synthetic images closer together. The minimal gray value in the real images was typically larger than 0, while the synthetic images also showed gray values down to 0. Thus, for each texture type, we add the mean minimal gray value observed in the real images to all synthetic images.

Pre-processing the data is an effective and efficient method to overcome some of the perceptual differences between domains. To jointly compensate for the illumination intensity and material absorption differences between synthetic and real data, we estimated an image multiplication factor. This method is justified, since the factor can be extracted from the integral of the rendering equation [49], assuming that the inter-reflectance of the surface mesostructures is negligible. The assumption holds for our case, since we use normal mapping for mesostructures, which does not produce inter-reflectance. The bias value was added to all the pixels to compensate for the ambient illumination caused by the low-intensity inter-reflections of the inspected object and the inspection environment. This can be observed in the image histograms as a gap of zeros between the illumination value 0 and the first next intensity value in the histogram. We estimated the bias by calculating the average gap size. The estimated factors and biases are as follows: $[eqn]$ for sandblasted, $[eqn]$ for parallel, and $[eqn]$ for spiral milling. In Figure 21, examples of the intensity correction and the effect it has on the synthetic images are shown. Finally, since the synthetic pinhole camera model does not include blurring and imperfections of the real camera, we artificially blurred the synthetic images using defocus blur [121] with a radius of $[eqn]$ .

The synthetic data do not model an exact 1:1 replica of the texture found on a single real object, but it models stochastic variations of it. Therefore, we measured the distance to the closest synthetic sample, similarly to a one-sided Chamfer distance. When evaluating the metrics, for each viewpoint, we evaluated the metric between the real image and all the available synthetic object instances and selected the value representing the highest similarity (lowest distance). In Table 5, for each texture type, we reported the metric value averaged across all viewpoints and objects sharing the same texture type.

8.2.2. Task Similarity

Task similarity is affected by concept shift and estimates how well the task function, mapping inputs to labels, obtained on synthetic data, transfers to the target dataset. In our case, we measured how well an image semantic segmentation model trained on synthetic data generalizes to the real data. In the field of domain adaptation [113], a common approach is to report the difference in task metrics. As discussed in [122], this approach measures the difference in biases present in the two datasets, which can be interpreted as domain shifts when working with datasets acquired from different environments.

We trained a UNet [123] with a ResNet-50 [124] backbone from five random parameter initializations per experiment and reported the top obtained result so as to reduce the bias of weight initialization. The model was trained to perform semantic segmentation through pixel classification into the classes background, dent, and scratch. Since the non-background classes are highly underrepresented, we observed the need to use class weighting, which we empirically found to be $[eqn]$ for the real and $[eqn]$ for the synthetic datasets. We optimized the models by using the AdamW optimizer [125] with a learning rate of $[eqn]$ and a $[eqn]$ weight regularization of $[eqn]$ , for a maximum of 1000 epochs, with early stopping at validation loss convergence upon five consecutive validations without at least a $[eqn]$ relative improvement. For model fine-tuning, we used the learning rate of $[eqn]$ using the real train and validation splits. For memory efficiency, during training, we extracted random image crops of a size of $[eqn]$ , and during the evaluation, we ended up splitting the images into a $[eqn]$ grid of patches of a size of $[eqn]$ . All images were zero padded to ensure dimension divisibility correctness when evaluating the UNet and image patching. To speed up model training, we centered the images linearly between $[eqn]$ .

Domain alignment through image rendering is a memory and compute expensive operation, thus we offloaded as much of the domain alignment as possible to online post-processing. We commenced by pre-processing the synthetic images using the factor and bias that were estimated for domain similarity, with a masked application applied to only the main face of interest in the image. To simulate the non-zero background, we artificially added Gaussian noise of amplitudes in the interval $[eqn]$ to the background around the object. To simulate the vertical blooming effect caused by photon leakage within the saturated regions of the CCD sensor array, we extracted the maximum value of each image column, masked the column using a threshold of $[eqn]$ , blurred it by using a box filter of size 64, multiplied it by $[eqn]$ , and added it to the original image with clipping. Furthermore, manually annotated real images contain imprecision in masks, as they often cover the area around the rim of the defect. We emulated this by dilating the synthetic masks by 1 pixel. The synthetic images contain masks in regions that have mostly uniform value, and should be considered invisible. We pre-filtered the masks to remove defects with insufficient visibility. We empirically estimated the visibility as the difference between the mean intensity of the defect and the mean intensity of the surface ring around the defect, calculated as the difference between the defect mask, which was dilated two times, and the original defect mask. Defects are considered visible if the difference in means is over $[eqn]$ .

Data augmentation further increases the diversity of the dataset and helps to regularize the recognition model so as to consider the important features. To simulate the offsets of the camera, we used a random rotation of amplitude 15°. Illumination strength and object material diffusion variation can be jointly controlled using random exposure from the interval $[eqn]$ . To simulate the blurred background structures and impurities in the lens or the light carrying medium, we added Gaussian noise of amplitude in the range of $[eqn]$ to the entire image and then blurred it by using defocus blur [121] of kernel sizes in the interval of $[eqn]$ . Gaussian noise of amplitude in $[eqn]$ was added once more to simulate the static noise of the real images. Finally, images can be randomly flipped horizontally to increase the texture–structure pair diversity. All the augmentations were performed with a probability of $[eqn]$ . Finally, we performed intensity-biased random cropping [45] using a threshold of 15 to ensure production of crops that contained more visible structures in the image.

The results of image segmentation were collected and are displayed in Table 6. When computing the mean score, the contribution of the background class to the mean value was ignored, as it is the majority class that would skew the results away from the defect classes. To evaluate the concept shift, we compare common semantic segmentation metrics of the model generalization when trained on the real subset to the synthetic subset.

9. Discussion

9.1. Pipeline Controllability and Simplicity

The pipeline is very versatile and allows for the generation of a wide variety of texture variations needed for domain randomization. We show that, even in the extremely low-data scenario of this study, we can model and generate a large variety of samples with great similarity to the real data while keeping the pipeline interpretable for further improvements in texture and lighting. The physically linked parametrization of the individual modules allows defining elements using physical measurements, thus reducing the initial expensive search for viable parameters. While rendering is the most computationally demanding component, it can be easily parallelized.

The large number of parameters make it difficult to study the influence of different parameter sampling schemes on recognition performance. However, this pipeline opens the opportunity to design a higher level of abstraction for dataset design through the modeling of lower dimensional constraints over larger sets of parameters. While this is out of the scope of this paper, we strongly encourage its research in the future.

9.2. Domain Similarity

Visually comparing the synthetic and real domains, we observed similar reflection patterns across all three texture types, working both for denser and these patterns. The parallel or spiral patterns corresponded to the ones in the real images, with the striped reflections falling off consistently. The reflection of the light source shape closely matched that in the real images; however, it was prone to slight differences due to texture directions or center offsets. Additionally, the variations in the texture consistently models that of the real data, especially the semi-randomness of the milling patterns, where opposite side rings appear at times as the dominant pattern. However, there are noticeable differences between the images, as well. The synthetic images are much sharper than the real ones due to the missing blur occurring from the imperfections of real camera and offsets of its position in space. In addition, the synthetic materials are more reflective or less rough than the ones in real images, observed by the difference in average intensity gradient along the vertical axis (most visible in sandblasted). However, in the views observed from the views perpendicular to the surface, the intensity drift is not visible. This might require a further study on the viability of existing material BRDF models and the need for finer-grain textures or modifications of the rendering algorithm.

Measuring texture similarity is a difficult problem due to surface and acquisition imperfections being present in the real data. Camera blurring, surface imperfections, and other effects move away from the nominal texture without degrading its performance, thus producing falsely bad results. In addition, it is difficult to generate a nominal image that perfectly aligns with the acquired image due to camera–object and texture–object alignments. In this work, we instead relied on methods that compare global or local pixel statistics of textures to remove alignment. From the results in Table 5, we observed that the sandblasted textures have the highest degree of similarity compared to the parallel and spiral milling textures. This result is related to the simplicity of modeling the sandblasted texture, as the texture appears as mostly uniform noise. More structured textures, such as those of milling, are more difficult to replicate, as they have significantly more parameters that interplay to produce unique patterns.

Comparing the values of metrics is difficult; even though they are bounded between $[eqn]$ and might measure similar properties of images, they do not share the same linearity of response to changes in inputs. A separate study is required to compare which physical properties different metrics attempt measuring over additional datasets. Since such a study targets a different task, it requires different definitions of the dataset parameters, which is out of the scope of this study.

9.3. Task Similarity and Recognition Performance

The defect recognition results in Table 6 show very similar performances in all experiments, except when the model trained on the synthetic domain is evaluated directly on the real domain. The similarity of results, even after fine-tuning, indicates that the models are saturated in their capacity to solve the task. The low generalization to the real domain indicates that there exists a misalignment between the synthetic and real data distributions, both in the form of a covariate shift (as observed by similarity metrics) and concept shift. The results obtained from the models trained on all textures do not surpass the performance on the highest performing sandblasted texture, leading us to the conclusion that training a model on multiple textures at once does not give any benefit over training a separate model for each texture.

When inspecting the model predictions over the datasets, we noticed some patterns. All models tend to predict masks that are slightly wider than the label. Most false positives come from the milled textures, in the lines where the two milling path circles overlap and along the exit lines. In sandblasted textures, tiny glints from the surface are often falsely predicted as defected. False positives are also present in minor deviations of an object’s shape, such as edge scuffing, beveling, deeper sandblasting holes, etched signatures, on paper label stickers, and where the milled pattern has a high contrast. Models also tend to predict defects on out-of-focus faces, which the annotator left as unannotated due to heavy blurring or unrecognizability. When predicting scratches, the models prefer areas where the texture has lower contrast and defect stand out. Some scratches that are not close to being horizontal or vertical are not predicted, even though the visibility is similar to the predicted ones, meaning the recognition models need more filters in lower levels or more explicit rotation invariance. Overall, it seems a higher resolution might help the model distinguish between defect and texture patterns.

9.4. Influence of Domain Similarity on Task Performance

The largest drop in recognition performance is on milled textures, which is consistent with their lower domain similarity when compared to the sandblasted textures. Furthermore, SSIM and LPIPS seem to be linearly correlated to the best obtained recognition performance. If this were true, we could use these metrics when iteratively improving the synthetic data to reduce the domain gap at its source, the data simulation, or the generation process. This forms the questions of whether these metrics are indeed predictive of the domain gap that impedes the recognition performance, which other metrics have a similar property, and why MAE and histogram comparisons do not seem to be. We find these questions are out of the scope of this paper, and we leave their careful treatment for a future study.

10. Conclusions

The presented method is capable of generating synthetic data for structured surface textures for a metal object by using a single sample of a real surface and knowledge about the manufacturing process. It allows precise control over its parameters, and the parameters are modeled to represent physically measurable or intuitively approximate features. The generated data offer the highest degree of realism, yet, for the studied textures, in terms of patterns, the intensity distributions and defects vary. Furthermore, the presented method is capable of generating synthetic data at scale, with support from the physically plausible variations in texture and defect parameters. The separation of modeling steps gave rise to the concept of scales decomposition, and provides a clear backbone on how further developments can improve and extend the capabilities of the pipeline. The decomposition of the dataset quality measurements allowed us to compare its similarity between the synthetic and real datasets as a whole while separating the similarity of the domain from the similarity of the task. The uncovered predictive trend that similarity metrics have on defect detection performance offers an efficient measurement technique for iterative synthetic data development. Finally, the published dual dataset serves the community by offering missing data for inspection of structured surfaces, facilitates domain adaptation research, and contributes a benchmark for the development of industrial inspection models.