Field of the invention

This invention relates to a training system for an ultrasound inspection system.

Background of the invention

Ultrasonic testing methods currently used in the industry typically necessitate physical contact to the specimen and/or a liquid couplant in order to achieve the acoustic impedance matching and improve the ultrasound transmission. They often need to be performed manually by a properly qualified operator however despite strictly defined standards, the factor of the subjective interpretation by the operator cannot be avoided.

Ultrasonic immersion testing is a frequently used method with the disadvantage that the specimen and the ultrasonic probes need to be immersed in a liquid such as water. For many applications, immersion may not be possible or practical, or may be too costly or complex to set up compared to other inspection techniques.

Air-coupled ultrasound avoids the drawbacks of ultrasonic immersion technology however can typically only be used on large plate-like objects in through-transmission setup. The objects of more complex geometry face problems since it is not possible to distinguish between the low-intensity informationcarrying ultrasound transmitted through the solid specimen and ultrasound transmitted directly through air, which disturbs the measurement. Air-coupled ultrasound transducers are narrowband and have low characteristic testing frequencies (typically between 50 kHz and 500 kHz), which causes a significant decrease in resolution and detection sensitivity.

Lasers are currently broadly used in the industry for inspection of external geometry and surface quality of manufactured parts. Although it is per se known to use lasers to generate ultrasound, or to read ultrasound vibrations on the surface of an object, there is no product in the market for industrial ultrasonic inspection of internal properties using lasers for both excitation and detection directed on the surface of a nonimmersed specimen to be inspected, due to the difficulty of obtaining a reliable measurement signal that has sufficient bandwidth and signal to noise ratio using only laser systems for both the excitation and the detection of ultrasound on various surface materials of objects for inspection.

In conventional ultrasound testing methods, a first drawback is often the time required to setup and configure the test system to output reliable results useful for the quality inspection, and a second drawback may lie in the difficulty of accurately determining the exact type of defect, or discriminating with sufficient accuracy between defects of different types. Such problems may be compounded with non-contact methods, especially laser-based testing methods mentioned above.

While it is known to use machine learning algorithms in various industries to improve the automation and interpretation of results, such machine learning methods are typically not employed in the often very narrow product specific implementations of ultrasound inspection procedures that have a very specific inspection protocol and setup performed manually by a qualified operator.

There is a continuous need to reduce the cost of quality inspection procedures while increasing the reliability and accuracy of such procedures.

There would also be an advantage in extending the range of products on which ultrasound testing may be applied, including on products manufactured in large series in an automated manner.

Summary of the invention

An object of the invention is to provide an ultrasound inspection training system to improve an ultrasound inspection system, such that the ultrasound inspection system is economical to implement and that allows reliable and accurate ultrasound inspection of internal properties of an object.

It is advantageous to provide an ultrasound inspection training system to improve an ultrasound inspection system, such that the ultrasound inspection system is versatile and can be easily implemented on objects of diverse geometries and sizes.

It is advantageous to provide an ultrasound inspection training system to improve an ultrasound inspection system, such that the ultrasound inspection system is easy to set up and operate and requires minimal manual intervention.

It is advantageous to provide an ultrasound inspection training system to improve an ultrasound inspection system such that the ultrasound inspection system allows rapid inspection.

It is advantageous to provide an ultrasound inspection training system to improve an ultrasound inspection system, such that the ultrasound inspection system is compact.

It is advantageous to provide an ultrasound inspection training system to improve an ultrasound inspection system, such that the ultrasound inspection system is able to detect a large range of defects or anomalies in an inspected object. Another more specific object of the invention is to provide an ultrasound inspection training system specifically to improve a laser-based ultrasound inspection system with the aforementioned advantages.

Various objects of this invention have been achieved by providing the ultrasonic inspection training system according to the independent claims. Dependent claims set forth advantageous embodiments of the invention.

Disclosed herein, is an ultrasound inspection training system comprising a test object, an ultrasound producing device configured to generate ultrasonic signals in the test object and read ultrasonic response signals emitted from the test object, a property altering device coupled to the test object, and a measurement computing system having program modules installed and executable therein, the program module including a machine learning algorithm, wherein the property altering device is configured to alter the ultrasound propagation characteristics of the test object to generate ultrasonic response signal datasets for training a machine learning algorithm of a measurement computing system and generating a diversified database.

In an advantageous embodiment, the property altering device comprises a mechanical property altering device configured to alter the mechanical properties of the test object.

In an advantageous embodiment, the mechanical property altering device comprises any one or more of a strain or deformation inducing element, a heating element, an inductive element, a capacitive element, a piezoelectric element, an electrostrictive element, or a magnetostrictive element.

In an advantageous embodiment, the mechanical property altering device comprises at least one variable state material configured to alter the ultrasound propagation characteristics of the test object according to a state of said variable state material, the variable state material comprising a material that changes any one or more of form, dimensions, temperature, or material properties when subjected to stimulation comprising any one or more of electrical, magnetic, optical, electromagnetic, or thermal stimulation.

In an advantageous embodiment, the property altering device is mounted on a surface of the test object or embedded within the test object.

In an advantageous embodiment, the property altering device comprises an electro-ultrasonic device configured to alter the ultrasound propagation characteristics of the test object.

In an advantageous embodiment, the electro-ultrasonic device comprises an active receiver component to capture an ultrasonic response, and an active emitter component to provide ultrasonic feedback, the active receiver component and active emitter component mounted on a surface of the test object or embedded within the test object.

In an advantageous embodiment, the electro-ultrasonic device comprises an active receiver control unit with amplifier connected to the active receiver component, and an active emitter control unit with amplifier connected to the active emitter component.

In an advantageous embodiment, the electro-ultrasonic device comprises one or more electro-ultrasonic control units connected on the one hand to a controller of the ultrasound producing device, and on the other hand to the active receiver control unit with amplifier and active emitter control unit with amplifier.

In an advantageous embodiment, the active receiver component and active emitter component comprises or consists of an ultrasound transducer based on any one of piezoelectric, electrostrictive, and magnetostrictive materials.

In an advantageous embodiment, the electro-ultrasonic device comprises an analog or digital electroultrasonic control unit configured to manipulate on demand, for instance amplify, phase delay or filter, a control signal of the electro-ultrasonic property altering device and therefore manipulate on demand the mechanical properties or ultrasound propagation properties of the object of inspection.

In an advantageous embodiment, the ultrasound producing device comprises a laser-based ultrasound device for ultrasound generation and/or detection.

In an advantageous embodiment, the machine learning algorithm is a neural network.

Also disclosed herein is a laser-based ultrasound inspection system comprising an object of inspection, a laser device and a coating arrangement including a laser absorbing layer disposed on a surface of the object of inspection, a backing layer disposed over the laser absorbing layer, and reflective particles or a partially reflective layer coupled to the surface of the object of inspection.

In an advantageous embodiment, the partially reflective layer or reflective particles are in or on the backing layer and/or the laser absorbing layer.

In an advantageous embodiment, the partially reflective layer or reflective particles are disposed in or on a second backing layer positioned on the surface of the object of inspection separately from the laser absorbing layer.

In an advantageous embodiment, the reflective particles are retro-reflective particles. In an advantageous embodiment, the retro-reflective particles are ball-shaped, for instance substantially spherical.

In an advantageous embodiment, the reflective particles have a refractive index that is greater than a refractive index of the backing layer.

In an advantageous embodiment, the partially reflective layer is configured to split a laser beam for ultrasound generation and laser beam for ultrasound detection into a beam passing through the backing layer on to the absorbing layer, and a reflected beam for reading of ultrasound waves.

In an advantageous embodiment, a material of the laser absorbing layer is selected from anyone or more of hydrogenated (or black) hydrogenated titanium dioxide, carbon nanotubes, graphite powder mixed with epoxy resin, gold nanopores, black carbon with polydimethylsiloxane, reduced graphene oxide, polydimethylsiloxane , metals (e.g. aluminum, chromium, copper) and polymers (e.g. high density polyethylene, polycarbonate, acrylonitrile butadiene styrene).

In an advantageous embodiment, a material of the backing layer is selected from anyone or more of silica (silicon dioxide), borosilicate, fluoride, aluminate, borate, phosphate, chalcogenide, sapphire, or similar glasses (crown and flint) and glass ceramics, polyethylene, polyvinyl chloride, terephthalate, polystyrene, polypropylene, polycarbonate, polymethyl methacrylate or similar, wherein the additives may include one or more of the following additives thoriu oxide, lanthanum oxide, lead oxide, cerium oxide, calcium oxide, magnesium oxide, aluminium oxide, boric oxide, sodium carbonate, germinates, nitrates, carbonates, plastics, acrylic, titanates, arsenates, antimonates, tellurites, metals, aluminates, phosphates, chalcogenides, borates, fluorides.

In an advantageous embodiment, a material of the reflective layer or particles is selected from anyone or more of silica (silicon dioxide), borosilicate, fluoride, aluminate, borate, phosphate, chalcogenide, sapphire, or similar glasses (crown and flint) and glass ceramics, polyethylene, polyvinyl chloride, terephthalate, polystyrene, polypropylene, polycarbonate, polymethyl methacrylate, wherein the materials may include one or more of the following additives thoriu oxide, lanthanum oxide, lead oxide, cerium oxide, calcium oxide, magnesium oxide, aluminium oxide, boric oxide, sodium carbonate, germinates, nitrates, carbonates, plastics, acrylic, titanates, arsenates, antimonates, tellurites, metals, aluminates, phosphates, chalcogenides, borates, fluorides.

In an advantageous embodiment, the coating arrangement has a thickness in a range of 0. 1 mm to mm to 2 mm, a thickness of the laser absorbing layer being in a range of 10 pm to 0.5 mm, and the backing layer having a thickness in a range of 0.5 mm to 2 mm.

In an advantageous embodiment, the laser device comprises a laser ultrasound generator and a laser ultrasound reader in a single unit, or as separate units.

Further objects and advantageous aspects of the invention will be apparent from the claims, and from the following detailed description and accompanying figures.

Brief description of the drawings

Figure la is a schematic block diagram illustrating an overview of an ultrasound inspection training system according to an embodiment of the invention;

Figure lb is a schematic block diagram illustrating another overview of an ultrasound inspection training system according to an embodiment of the invention;

Figure 2 is a schematic view in cross-section of an object under inspection with a coating according to an embodiment of the invention configured for laser-based ultrasound inspection;

Figure 3a is a view similar to figure 2 of another embodiment in which the laser ultrasound generation is in a location separate from the laser ultrasound reading of the inspected object;

Figure 3b illustrates a variant of figure 3a in which the laser ultrasound generation and laser ultrasound reading are performed on opposite sides of the inspected object;

Figure 4 is a schematic representation of a laser device for laser ultrasound generation and reading according to an embodiment of the invention;

Figure 5 is a view similar to figure 4 in which the laser ultrasound generator is separate from the laser ultrasound reader;

Figure 6a is a schematic cross-sectional representation of a training object of an ultrasound inspection training system, the training object for training a machine learning process of the ultrasound inspection training system according to an embodiment of the invention;

Figures 6b to 6d are views similar to 6a of variants of test objects; Figures 7a to 7d are schematic representations of a transducer, a property altering test object, and a control system of an ultrasound inspection training system according to various embodiments of the invention;

Figures 8a to 8e are schematic representations of a transducer and a test object of an ultrasound inspection training system according to various embodiments of the invention;

Figure 9 is a circuit diagram of an analog electro-ultrasonic control device of an ultrasound inspection training system according to an embodiment of the invention;

Figures 10.1 to 10.5 are illustrations relating to an experimental setup for machine learning as described below:

Fig. 10.1 : Scheme (a) and photo (b) of a specific example of a property altering device comprising a shape memory polymer (SMP) foil. Ultrasound was generated by a laser pulse and detected by a laser vibrometer at fixed positions. An artificial reconfigurable defect was created by locally reducing the Young’s modulus and increasing the ultrasound attenuation level of the SMP by a heating laser. A specimen (i.e. test object) configuration was changed by applying two masses on the SMP foil.

Fig. 10.2 : Raw experimentally obtained signals used for the model training. In the shades of gray, change of the defect position in the direction x (a-c) and direction y (d-f) is shown for first (a, b), second (b, d), and third (c, f) specimen configuration. Without machine learning algorithms, it is not possible to localize the defect and distinguish between different specimen configurations. Please note that in order to increase figure clarity, only 624 (13 repetitions x 8 defect positions x 2 defect position dimensions x 3 specimen configurations) signals are shown instead of altogether 3840 signals (20 repetitions x 8 defect positions x x 8 defect positions y x 3 specimen configurations) used in our study. Fig. 10.3: Labeling accuracy (red full line) and loss parameter (blue dashed line) for the classification between the three specimen configurations in dependency on the number of defect positions used for training. The defect position is in this case a disturbing parameter, which significantly affects the signal shape. The model was tested on the previously unseen dataset containing signals of all defect positions. Good performance and robustness of the model is achieved for reduced training datasets as well.

Fig. 10.4 : Mean positioning error and labeling accuracy of the defect positions if the model is trained on reduced datasets and different specimen configurations. 20 repetitions of the training process are represented by the mean value (solid line) and the standard deviation range. The vertical orange line represents the case shown in Fig. 10.5 The diversified datasets make the model robust.

Fig. 10.5 : Performance of the model during a single training process (first row) and its final performance on the training (second row) and testing (third row) datasets. No localization is possible if only three defect positions from the dataset obtained at the specimen configuration equivalent to the testing dataset are used for the training process (first column). The performance of the model is partly improved if an additional dataset containing all defect positions is used for the training process even if they were obtained at the specimen configuration different to the one of testing (second and third column). The localization error decreases to 0.25 mm (which delivers the labeling accuracy of 80%) if the robustness of the model is improved by a diversified dataset containing all defect positions obtained at two different specimen configurations and only three defect positions obtained at the specimen configuration equivalent to the testing dataset (fourth column).

Detailed description of embodiments of the invention

Referring to the figures, starting with figure la, an ultrasound inspection training system comprises an ultrasound transducer device 3, a measurement computing system 5, and a test object 1 for inspection. The computing system 5 is connected to the ultrasound transducer device 3 at least during the time required to upload ultrasound measurements results generated by the ultrasound transducer device. The computing system may however comprise a control module and be connected to the ultrasound transducer during an inspection training process in order to control a reading process of the ultrasound transducer on the surface of the inspected training object and receive the measurement results fed back from the ultrasound transducer during the reading process. The computing system 5 is connected to the controller of the property altering device 83, which is connected to the property altering device 8, at least during the time required to generate training datasets. The property altering device 8 is applied in the interior or on the surface of the object of inspection 1, with which it is mechanically interacting. The controller of the property altering device 83 is connected either to electrodes or heating elements 8a or to electro-ultrasonic control unit 81e, depending on the variant.

Referring to figure lb, an example of an ultrasound inspection system that needs to be trained is illustrated. In this example, the ultrasound inspection system is a laser-based ultrasound inspection system comprising an ultrasound transducer device in the form of a laser ultrasound generator 32b, a measurement computing system 5, and an object of inspection 1’ which in this specific advantageous embodiment comprises a coating arrangement 4 on an outer surface of the object of inspection. The computing system 5 is connected to the laser device 3 at least during the time required to upload ultrasound measurements results generated by the laser device. The computing system may however comprise a laser control module and be connected to the laser during an inspection process in order to control a scanning process of the laser on the surface of the inspected object and receive the measurement results fed back from the laser during the scanning process.

The computing system comprises a machine learning program module for interpreting the measurement results from the laser device and that may include atraining phase using test objects 1 that have configurable known properties. Within the scope of the invention, the test objects may be actively controlled and have varying properties to simulate defects and other material properties that allow improved training of the machine learning program module. The computing system may have a reference database 7 stored therein and that may be updated with measurement results based on training data and on measurement data of the inspected objects over time.

Referring to the advantageous laser-based ultrasound inspection system embodiment, it may be noted that laser pulses allow effective broadband and contact-free generation of ultrasound in solids. Broadband laser ultrasound delivers additional and more precise information about the quality inspection properties than conventional air or liquid coupled ultrasound techniques, and benefit from the advantages of contact-free methods that are faster, more robust, and are easier to be automated than contact-based methods. With laser technology it is possible to generate very short pulses (ns, ps, fs) of high peak power (e.g. hundreds MW) and it is possible to generate ultrasound without physical contact directly with the object of inspection. Two solid-to-air interfaces are thus omitted (transmitter-to-air and air-to-specimen), in comparison to a more traditional solution of air-coupled transducers, for which typically only around 0.1% of the ultrasound energy is transmitted at each such interface.

For laser-based ultrasonic excitation in the thermoelastic regime, ultrasound is generated due to the fast thermal expansion and contraction of a thin material layer on the specimen surface. For higher pulse energies, small explosions on the specimen surface and ablated particles generate ultrasound (ablative regime).

Efficiency of ultrasound generation by laser pulses may be improved, for instance by a factor of up to 100, by introducing a backing-mass layer. The backing layer may comprise a solid or liquid (partly) transparent layer applied on the surface of the object of inspection and which preferably has mechanical properties (acoustic impedance) similar to the object of inspection. In contrast to air backing (when no backing-mass layer is present), ultrasound is generated at the transition between this transparent layer and an opaque object of inspection; and not at the transition between the air and the object of inspection. As a consequence, a higher share of laser pulse energy will be converted to the ultrasound energy propagating in the object of inspection and less of the energy will be lost to the surrounding media e.g. by created plasma, kinetic energy of ablated particles, or air-propagating ultrasound. In this case, the ultrasound is generated in the bulk of the object of inspection by an effect similar to a subsurface explosion.

An absorptive layer with high coefficient of thermal expansion is applied between the object of inspection and the transparent backing-mass layer. Efficiency of ultrasound detection by a laser beam probe (laser- Doppler vibrometers, interferometers) is conditioned by the amount of light captured by the measuring device. A high power of the laser beam reflected back to the receiving optics of the laser reader is advantageous. In many cases, light scattered in all directions on the rough object of inspection surface does not suffice.

Referring now to figures 2, 3a and 3b, coating arrangements according to embodiments of the invention are illustrated.

In a first variant, the coating arrangement 4 includes a single coating for both laser ultrasound generation and laser ultrasound reading. In this configuration, the laser device for ultrasound generation 3a generates a laser source beam 11 that is absorbed and generates ultrasound in thermoelastic or ablative regime. A laser beam for ultrasound detection 12 is generated and captured by the laser device for ultrasound detection 3b.

The laser source beam 11 is configured to generate ultrasound in the object of inspection 1 and the laser beam for ultrasound detection 12 transmits vibration of the surface 2 of the object of inspection 1 to the laser device 3. The reflected signal represents the object’s response to ultrasound stimulation that is dependent on the internal properties of the object and that allows to distinguish objects with defects or other anomalies from objects without defects or anomalies.

The coating arrangement 4 according to a first embodiment illustrated in figure 2 comprises a laser absorbing layer 4a positioned on the surface 2 of the object of inspection 1, a backing layer 4b mounted over the laser absorbing layer 4a. The backing layer is transparent or at least partially transparent to the laser source beam 11 to allow the laser source beam 11 to at least partially reach and interact with the laser absorbing layer 4a. Interaction of the laser source beam with the laser absorbing layer generates ultrasound due to the laser energy absorbed in the laser absorbing layer 4a.

The coating arrangement 4 further includes a reflective layer or reflective particles 4c configured to reflect part of the laser source beam back to the laser device. The reflective particles 4c may advantageously be embedded in a backing layer 4b, which may also serve as the backing layer positioned over the laser absorbing layer, or a backing layer positioned directly on the surface of the object of inspection, depending on the embodiment. The reflective particles 4c may also be provided in a reflective particle support layer material that is different from the material of the backing layer 4b.

Since the reflective layer or reflective particles are coupled to the surface 2 of the object of inspection (e.g. via the backing layer), vibrations of the surface 2 due to ultrasound waves within the material of the object of inspection are captured in the laser beam for ultrasound detection 12. In embodiments as illustrated in figures 3a and 3b, the coating arrangement 4 may comprise a laser absorbing layer 4a positioned against the surface 2 of the object of inspection and a backing layer 4b mounted over the absorbing layer similar to the previously described backing layer, and a separate reflective layer arrangement positioned on the surface 2 of the object of inspection in a separate position from the absorbing and backing layer arrangement 4a, 4b. In this variant, the laser source beam 11 is generated by a laser ultrasound generator 3a that is separate from a laser ultrasound reader 3b that generates a separate laser beam and receives in return a laser beam for ultrasound detection 12 for reading the vibrations of the surface 2 of the object of inspection at a separate position from the position where the ultrasound is generated. The functions of ultrasound generation and ultrasound reading on the surface of the object of inspection are thus separated into different zones. These zones may be positioned adjacently or in relative proximity on a same side of a surface of the object of inspection as illustrated in figure 3a, or on opposite sides of the object of inspection, as illustrated for instance in figure 3b. Various other positional arrangements such as top or bottom, and lateral sides may for instance be configured and the object of inspection although shown as a substantially planar with a constant thickness may have various 3- dimensional shapes.

In a preferred embodiment, the reflective particles 4c comprises retro-reflective particles, for instance in the form of balls, for instance substantially spherical balls of transparent material with a refractive index that is different to the refractive index of the backing layer 4b and that causes the light beam entering into the reflective particles to be internally reflected until the internal reflected light exits the particle on a side where the light enters the particle. The retroreflective properties can also be achieved by particles of other shapes such as tetrahedrons, or by various shaped particles with a reflective coating on the surface thereof.

In a variant, the reflective layer may have a partially reflective coating that splits the source laser beam into a transmission beam that traverses the reflective layer to impinge upon the absorbing layer, and a reflected beam used to measure the ultrasound vibration of the coating coupled to the surface of the object of inspection.

Advantageously, the combination of the laser absorbing layer with a backing layer to improve the transmission of ultrasound energy captured in the absorbing layer to the object, and the use of a reflective layer or reflective particles to increase the amplitude of the reflected signal from the surface of the object of inspection significantly improves the signal to noise ratio and also allows for a broad band ultrasound generation and capture for accurate and reliable inspection of material properties within the object of inspection.

The material of the absorbing layer 4a is configured to increase laser light absorption and to generate ultrasound by its expansion and contraction in a thermoelastic regime. The material of the absorptive layer is thus selected to have a high light absorption level at the wavelength of the laser for ultrasound excitation and at the same time a high thermal expansion coefficient, while also providing sufficient bonding properties between the object of inspection and the backing layer.

According to embodiments of the invention, materials of the absorbing layer include hydrogenated (or black) hydrogenated titanium dioxide, carbon nanotubes, graphite powder mixed with epoxy resin, gold nanopores, black carbon with polydimethylsiloxane, reduced graphene oxide, polydimethylsiloxane , metals (e.g. aluminum, chromium, copper) and polymers (e.g. high density polyethylene, polycarbonate, acrylonitrile butadiene styrene). The absorbing layer may consist of or comprise a solid form, or may consist of or comprise a liquid or gel form.

According to embodiments of the invention, the materials of the backing layer 4b may advantageously include

- transparent and semi-transparent materials such as silica (silicon dioxide), borosilicate, fluoride, aluminate, borate, phosphate, chalcogenide, sapphire, or similar glasses (crown and flint) and glass ceramics;

- transparent or semi-transparent polymers such as polyethylene, polyvinyl chloride, terephthalate, polystyrene, polypropylene, polycarbonate, polymethyl methacrylate or similar;

- the backing layer materials may include additives such as thoriu oxide, lanthanum oxide, lead oxide, cerium oxide, calcium oxide, magnesium oxide, aluminium oxide, boric oxide, sodium carbonate, germinates, nitrates, carbonates, plastics, acrylic, titanates, arsenates, antimonates, tellurites, metals, aluminates, phosphates, chalcogenides, borates, fluorides, or similar. The backing layer may consist of or comprise a solid form, or may consist of or comprise a liquid or gel form.

According to embodiments of the invention, the acoustic impedance of the material of the backing layer, and optionally also of the absorbing layer, is preferably similar to the material of the object of inspection as possible, typically in the range from 1.5x 106 kg m-2 s-1 and 50x 106 kg m-2 s-1. The backing layer material may thus be selected according to its acoustic impedance in order preferably to match or substantially match the acoustic impedance of the object of inspection to improve the transmission of ultrasound energy generated in the laser absorbing layer into the material of the object of inspection.

The coating arrangement 4 may be typically in a thickness range of 0.1 mm to 5 mm and therefore in many applications the inspected objects may be left with the coating, or in certain applications the coating may be removed from the inspected object by various processes such as subtractive machining, laser ablation, chemical etching, thermal combustion, and any other processes that may remove the coating from the object if needed. The laser-based ultrasound inspection system according to embodiments of the invention may advantageously be used for inspection of a large variety of objects such as: aircraft components, turbine blades, high-pressure vessels, pipelines, support structures, and other high-load or high-velocity components. The inspection system is applicable on a broad spectrum of materials - the most typical are metals, ceramics, concrete, polymers and various composites e.g. carbon-fiber-reinforced polymers. Typical production processes that require particular quality surveillance are welding, adhesive, soldering and other similar joints, additive manufacturing and casting parts. The inspection system is useful for continuous quality surveillance of rolled metal sheets and profiles.

According to an aspect of the invention, the interpretation of ultrasound signal measurement results may be assisted by a machine learning programs, in particular using neural networks in order to reduce the training data required when setting up the inspection system for a new object to be inspected.

The natural response of a test object in the ultrasonic frequency range is altered by internal damages, the change in geometric or material properties. The ultrasound generation, propagation, and detection properties are coded in trainable parameters of the neural network and the ultrasonic signals are interpreted without actually knowing the physical model of the specimen and the measurement setup, which is typically the case for industrial applications.

Machine learning algorithms are especially suitable for the interpretation of ultrasound obtained by contact- free methods, because the mechanical response of the test object is not disturbed by a contact probe, and provide advantages of fast, automated, flexible, and robust information processing.

Raw ultrasonic signals may be converted to the frequency domain by the fast Fourier transform. The first half of the real and the imaginary component values are merged together and used as a parallel input for the fully connected neural network. Multiple hidden layers are linear and have a rectified linear unit activation function. An Adam optimization algorithm may be used. The last (output) layer has the number of elements equal to the number of classes (e.g. presence of damage, damage type, adequate quality of the product, etc.). The loss parameter may be calculated by a cross entropy loss function for the classification problem. The neural network is trained by multiple passes of the entire training dataset, with the batch size equal to the whole training dataset size.

For the defect localization problems, the last layer has one element for each of the defect location dimensions. They are analog and directly represent the defect position. The loss function is defined as mean absolute error of the defect position. The training datasets size and diversity may be obtained by repeating the acquisition of ultrasonic signals under different conditions and different configurations of a test object (also named herein “specimen”). Varying parameters include: locations, energy, and shape of the ultrasound generation; position and mechanical boundary conditions of the test objects; together with the location and other properties (frequency range) of the ultrasound detection. The properties of the test object according to embodiments of the invention can be varied as described hereinbelow.

Nonlinear behavior in the ultrasonic frequency range is a reliable indicator for the presence of internal damages. Delaminations, kissing bonds, porosity, or microcracks induce nonlinear effects such that ultrasound does not propagate in the same way if the ultrasound power is changed, and the spectrum of the detected ultrasound is significantly different from the excited ultrasound (different frequency distribution). This can be caused by micro-friction, clapping, or other mechanical effects at the defect zone.

The ultrasonic signals obtained at different conditions or configurations to detect nonlinearities (as described above) are converted to the frequency domain and used as a parallel input to the neural network for the ultrasound excitation. The number of input layer’s elements of the neural network therefore equals to the number of sampling points (amplitude in time) multiplied by the number of signals obtained at different setup configurations. After applying the training process on the diversified dataset obtained on a low number of test objects, the neural network is able to extract nonlinear components signifying the presence of the damage. They are interconnected parameters extractable only by the change of system conditions.

The quality of the test object can be estimated by measuring its mechanical response in the ultrasonic frequency range during the change of the test object’s temperature or its mechanical properties. For example, by applying a pressure on the test object, the properties of the internal damage might be altered and the ultrasound propagation properties change. Similarly as when observing nonlinear behavior, both signals, before (during) and after the change, may be used in parallel as an input to the neural network.

By providing diversified training datasets as proposed in the present invention, the robustness of neural networks to the change of system properties can advantageously be achieved and the target information can be efficiently extracted from the ultrasonic signals.

Certain system properties can be varied during the repetition of the measurement on a same specimen - e.g. ultrasound excitation (e.g. energy, position, form) and detection (e.g. position, gain) properties. However, this does not provide us the robustness on the change of the test object properties, which is essential since the neural network should be efficient on the whole series of the objects for inspection, which are in practical cases partly dissimilar due to the production anomalies. For example, a small and uncritical change of the product geometry can affect the ultrasonic signal more than the presence of safety-critical internal damages. Typically, neural networks must be trained on a sufficiently high number of different specimens in order to achieve its efficiency. This is expensive because the training samples must be inspected by a destructive reference inspection method (e.g. microscopy of the cut section), which is often time-demanding and expensive. Embodiments of the invention overcome this problem by providing test objects with property altering devices that are easily configurable to simulate varied conditions and properties in an efficient manner.

According to an aspect of the invention a test object coupled to a property altering device with actively adjustable properties is used to generate diversified datasets of ultrasonic signals read from the test object that are relevant for quality inspection.

In a first variant, a reconfigurable test object with similar properties than the industrial object for which the robust neural network for the ultrasonic signal interpretation is provided, with the difference that specific (e.g. mechanical) properties can be varied using different mechanisms described hereinafter.

A second variant is a property altering device which can be applied directly (e.g. adhesively, magnetically or mechanically) on the industrial product in order to make it reconfigurable (e.g. on its surface to achieve a change of its geometry or stiffness). In a variant, a part of the industrial product (e.g. critical joint, link) can be exchanged by a test object with actively adjustable properties coupled to a property altering device.

A change in properties of the test object with the property altering device can be achieved by mechanisms based on different principles: changes in material properties of the test objet can be varied by a property altering device comprising a polymer with highly temperature-dependent properties (heat can be locally provided by a laser, resistor, coil or other heating elements), geometry and internal tensions of the test object can be varied by a property altering device comprising dielectric elastomers, mechanical switches, piezoelectric or piezostrictive elements.

The presence of a defect (internal crack, delamination, void, etc.) in a test object can be simulated by a property altering device simulating a local reduction of material stiffness or by locally increasing the ultrasound attenuation level.

In other embodiments, precise and localized variations of properties of the test object can be achieved by a property altering device comprising a high-frequency ultrasound generator coupled to the test object’s surface, which is controlled to provide a real-time ultrasonic response analogous to the ultrasonic response in an object of inspection with defects. In other words, the high-frequency ultrasound generator coupled to the test object’s surface is controlled such the ultrasound response signal simulates a defect in the test object. An advantage of the property altering device comprising a high-frequency ultrasound generator, is that large and diverse training datasets can be obtained using a single test object. In comparison to numerically generated training datasets obtained by numerical (finite elements method, finite differences method) or analytical models, the ultrasonic signals obtained directly on a physical test object with actively adjustable properties will be much closer to the real inspection scenario, which otherwise cannot be numerically modelled precisely enough.

The invention opens a possibility to validate and optimize the architecture of machine learning algorithms for the application of ultrasonic signal interpretation. It helps extracting the parameters of the ultrasonic signals, which are indicators for defects or damage in the object of inspection, and to differentiate between the abnormal parameters and the normal (expected) properties of the object of inspection. The test object coupled to a property altering device provides the possibility to extract information on components of the ultrasonic signals, in particular nonlinear components, which are indicative for the presence of damage or defects and may at the same time be invariant with respect to the object’s shape, material and other properties.

The invention has multiple quality control applications applicable in the fields of precision engineering, precious metals, energy sector, aeronautics, automotive industry, healthcare and certification of safety- critical infrastructures.

Machine learning algorithms provide robust, flexible and automated ultrasound signal interpretation. The algorithm can be trained to identify relevant signal parameters and to correlate them to the quality state of the product. The mechanism behind cannot be fully interpreted by human, as well as it is in most cases impossible to visually differentiate between ultrasonic signals carrying different information about the product quality. The relevant information can be extracted from the ultrasonic signals without knowing the physical model of the system i.e. without actually understanding the wave propagation (differentiating between the wave types, reflections etc.). An additional advantage of machine learning -based ultrasound interpretation is that it is not necessary to calibrate the ultrasound excitation and detection device. The training process itself is to some extent the calibration process of the whole system including wave propagation properties in the specimen, ultrasound excitation, detection, and coupling properties, together with the signal processing elements, which can all be undefined. Only several test objects are necessary to be inspected destructively for the purpose of the model training, while the state of the quality of the rest of the test objects can be estimated with high fidelity by non-destructive inspection methods and ultrasound signal interpretation based on machine learning. This has practical advantages for modem lean manufacturing industries demanding high flexibility. An example of a training process for an ultrasound inspection training system may comprise the following steps:

1) Integration of active elements (one or multiple) on (or in) the test object.

2) Application of the ultrasonic transducers (one or multiple) on the test object surface (contact transducers) or close to the test object (air-coupled transducers). Alignment of the laser beams on the test object surface in the case of laser-based ultrasound generation and detection.

3) Connection of the test object with actively adjustable properties with the controller of the test object with actively adjustable properties and ultrasonic transducers with the transducer control devices. Their connection with the computing system comprising a generator of the control signal for the test object with actively adjustable properties, a generator of the ultrasonic signal, a signal pre-processing unit, a machine learning algorithm, a control unit and a database.

4) Generation and detection of the ultrasound at the initial state of the test object.

5) Saving of the initial ultrasonic signal in the database.

6) Change of the test object properties by control and activation of the active element integrated on the test object.

7) Change ultrasound generation and detection properties.

8) Generation and detection of the ultrasound at the new state of the test object and the measurement system.

9) Saving of the initial ultrasonic signal and the state of the test object in the database.

10) Repetition of the points 6 to 9 at different states of the test object and different ultrasound generation and detection properties as long as the sufficient data size and diversity is achieved and non-linear effect caused by the defects are extracted.

11) Repetition of the points 1 to 10 on an additional test object with a different quality state.

12) Training of the machine learning algorithm on the obtained datasets, which will make it robust on the specific change of properties.

13) Re-evaluation of the training dataset to extract the effect of a specific change on the signal shape.

14) Application of the trained inspection system for the industrial inspection. Extraction of the quality states of the test objects.

Referring to figures 6a to 6c, for the purposes of improving the measurement of defects and anomalies in objects to be tested, various objects with various properties may be used for training the machine learning algorithms. These test objects may advantageously include objects with configurable property altering devices 8 that alter the properties of a test object 1 in a controlled manner.

The property altering device may comprise a mechanical property altering device 80 configured to alter the mechanical properties of the test object, the mechanical property altering device comprising any one or more of a strain or deformation inducing element, a heating element, an inductive element, a capacitive element, a piezoelectric element, an electrostrictive element, or a magnetostrictive element.

The property altering device may be based on reconfigurability achieved by the temperature, strain, or stress control or by electro-ultrasonic control devices. For instance, the active elements of the property altering devices may include heating elements, magnetic elements, capacitive elements, piezoelectric elements, and any other form of active element that generates heat, magnetic field, electrical field, or a mechanical stress on the test object in which these property altering devices are integrated or mounted on, in order to vary the material properties of the test object. Property altering devices elements may include materials that change form, dimensions, temperature, and material properties when subjected to stimulation. These allow, for instance to simulate certain material changes in a controlled predictable manner such that the ultrasound inspection of the test object may be used fortraining the machine learning algorithm. It may be noted that the computing system may also be fed measurement results from objects for inspection that are reference objects without anomalies or defects as a reference for subsequent testing of objects.

A change in properties of the test object with the property altering device (e.g. local drop of material stiffness, increase of ultrasound attenuation level) can be achieved by providing the property altering device with a material with temperature-dependent properties as for example shape-memory polymer, or other polymers (e.g. polyurethane, polychlorotrifluoroethylene, polypropylene, polyvinyl acetate, polyamide, polyethylene terephthalate, polyvinyl chloride), which have glass transition temperature in the range from 20°C to 200°C (typically at around 50°C).

Property altering devices 8 according to an embodiment may comprise one or more heating sources or electrodes 8a and one or more layers of a temperature dependent material piezoelectric, piezostrictive (or similar) material(s) 8b. In an embodiment, the heating source may be embedded in or mounted on the temperature dependent material layer(s). In another embodiment, the heating source may be remotely arranged, configured to heat the temperature dependent material in a non-contact manner, for instance by radiation.

The heating elements may in a variant be electric heating elements, for instance including coils or wires with the diameter of around 50-200 pm, for instance around 100 pm, embedded in or mounted on the material with temperature-dependent mechanical properties.

The heating elements can be arranged in multiple layers in the temperature dependent material.

Heating units ranging from 0.1 mm ² to several cm ² (typical size of around 1 mm ² ) can be controlled separately in order to locally control the ultrasound propagation properties. An embodiment of property altering devices with heating elements can be produced for instance by: (a) applying a conductive (e.g. made of copper) fdm on the material with temperature-dependent mechanical properties, (b) laser structuring of the conductive fdm in a shape to form a heating resistance or an induction coil (c) applying a next level of the material with temperature-dependent properties.

The property altering device may then be applied, for instance by bonding or clamping, on the test object.

In an embodiment, the temperature of the property altering device can be controlled by a laser illuminating the material with temperature-dependent properties. A laser beam can be used instead of the heating coil to locally increase the temperature of the material with temperature-dependent properties, which is included in the test object, whose mechanical properties can therefore be altered.

If the test object comprises a property altering device with temperature-dependent properties, the change of the ultrasound propagation properties can be achieved by heating the test object material. This principle can also be used to measure the local thermal dependency of the test object material on the temperature change.

A change in properties of the test object with the property altering device may be achieved by providing the property altering device with one or more strain or stress inducing elements such as piezoelectric, electrostrictive, or magnetostrictive elements, or dielectric elastomers.

Geometry and internal tensions of the test object can be varied by applying an electric or magnetic field in the piezoelectric, electrostrictive, or magnetostrictive elements of the property altering device.

The property altering devices can have multiple electrodes applied on the surface or embedded in the active material, which may include: ceramics (e.g. lead zirconate titanate), crystalline materials (e.g. quartz), polymers (polyvinylidene chloride, polyimide, polyvinylidene fluoride, polyamides), porous polymeric film or polymer composites, magnetostrictive materials and composites (e.g. Terfenol, Galfenol, Alfer, Cobalt ferrite), electrostrictive materials (e.g. lead magnesium niobate), or dielectric elastomers (graphite powder, silicone graphite mixtures, rubbers, acrylic elastomers). Property altering devices can be applied on the test object surface, in the interior of the test object, or at an interface between joining portions of a test object.

A change in mechanical properties of the test object with the property altering device may be achieved by providing the property altering device with mechanical (e.g. micromachined) switches applied on the test object surface or in its interior. The mechanical switch comprises a linear motor with the moving part pressed against the specimen surface. By releasing the contact, the mechanical properties of the test object changes. The ultrasonic signals are obtained before and after the release of the mechanical switch and included in the database for the training process of machine learning algorithms for ultrasonic signal inspection. The machine learning algorithm is therefore made robust to a specific change applied by the mechanical switch.

Referring to figures 7a to 7d, a change in properties of the test object with the property altering device 8 may be achieved by providing the property altering device with an electro-ultrasonic control system.

In this embodiment, the property altering device 8 comprises at least one ultrasound producing device 3 coupled to the test object 1 in a contact or non-contact manner, an active receiver component 8 la to capture an ultrasonic response, an active emitter component 81b to provide ultrasonic feedback, and one or more electro-ultrasonic control unit 8 le connected on the one hand to the active reader and feedback components, and on the other hand to a controller of the ultrasound producing device 3.

The ultrasound producing device 3 may be in the form of an electro-ultrasonic transducer 3 la, 3 lb, whereby the electro-ultrasonic transducer may include both ultrasound generation and ultrasound reading functions, or the ultrasound producing device 3 may comprise at least two electro-ultrasonic transducers, a first electro-ultrasonic transducer 31a for generating ultrasound signals, and a second electro-ultrasonic transducer 3 lb for reading ultrasound signals.

The electro-ultrasonic transducer(s) 3, 31a, 31b may be applied on the surface 2 of the test object 1, or located close to the test object for the case of air-coupled transducers.

In a variant, a non-contact ultrasound producing device may comprise one or more lasers for ultrasound generation and detection similar to the laser devices described in relation to figures lb to 5.

The operation frequency range of the ultrasound producing device 3 and property altering device 8 depends on the size of the features, in particular the defects, required to be detected in the test object. Suitable frequency ranges are: i. a frequency range below 100 kHz is suitable for the feature sizes above 5 mm, ii. a frequency range between 100 kHz and 1 MHz for the approximate feature sizes from 0.5 mm to 5 mm, and iii. a frequency above 1 MHz for the feature sizes below 0.5 mm.

If integral test object properties are to be estimated, the frequency range can extend below 20 kHz. The active elements 81a, 81b to capture the ultrasonic response and to provide ultrasonic feedback (emitter 8 lb and receiver 81a) are mounted on the test object surface 2 or inside the test object 1. They are connected to a high-frequency, electro-ultrasonic control unit 81e to provide a real-time ultrasonic feedback, which can be modulated at will. This embodiment allows to simulate the response of an object with defects or to a change of geometry or material of the object, subjected to ultrasound inspection.

The active receiver component 81a to capture the ultrasonic or acoustic response can comprise or consist of various devices to detect ultrasound, for instance ultrasound transducers based on piezoelectric, electrostrictive, and magnetostrictive materials.

The active receiver component 81a may be applied on a test object 1 by adhesives or a coupling media (liquid or gel), whose purpose is to increase the amount of ultrasound transmitted from the transducer to the test object. The active receiver component 8 la is connected to an active receiver control unit 81c which comprises an amplifier.

Alternatively, a laser vibrometer, a laser interferometer, or a laser deflection probe can also be used as an active receiver component 81a. The laser probe beam is illuminating the specimen surface and detecting the vibrations of the specimen surface eventually in the ultrasonic frequency range. A laser vibrometer is consisting of a photo detector, beam splitters, Bragg cell, and a laser source. It detects the velocity of the surface of the test object using the principle of Doppler shift. The laser deflection probe is consisting of a laser source and a photodiode with a sharp boundary edge of the sensitive area. The laser beam is reflected from the surface of the test object. The deflection of the laser beam is detected by the change of the light intensity illuminating the sensitive area of the photo diode on the area close to the boundary edge.

The ultrasonic signal may be digitalized by a sampling frequency which is at least two times higher than the highest frequency components of the ultrasound excited by the ultrasound generator, which is propagating in the test object in order to detect its properties (typically in the range of several 10 MHz). The ultrasound is afterwards digitally manipulated by the electro-ultrasonic control unit 81e on demand e.g. it is filtered, delayed in phase, amplified, etc. The type of manipulation depends on the type of diversity (different properties of the test specimen), which one would like to include in the reference database. Phase delay at different frequencies is representing having a similar effect than the geometry change or the change in the material of the test object. Attenuation at different frequencies is representing the presence of porosity or microcracks. Additional signal components at various frequencies (to simulate nonlinearities) can be added to the ultrasonic signals detected by the active receiver component 81a before being transmitted back to the test object by the active emitter component 81b. The type of additional signal components depends on the type of nonlinearity which one would like to include in the reference training database, for example doubling of the signal components of the highest amplitude in the signal. Each of the defects has its specific non-linear signature, which can separately be measured by comparing the signal with and without the detects on reference samples. The property altering device can be used to combine measurements on different reference samples in a single reference database by eliminating (achieved by the reconfigurability) the influence of disturbing parameters, which might not be the same for all the reference samples. An appropriate signal is generated (considering the previously measured transfer function properties of the active emitter component 81b) using a digital to analogue converter. Ultrasonic mechanical feedback (with respect to the mechanical input obtained by the active receiver component 81 and processed by the electro-ultrasonic control unit 8 le) is provided to the test object with small latency times (time delay between the emitting and receiving the ultrasound at the test object) using active emitter component 81b applied on the test object.

Electro-ultrasonic control unit 81e is connecting the active receiver control unit 81c (it includes an amplifier, an analogue to digital converter) and an active emitter control unit 8 Id (it includes a digital to analogue converter and an amplifier), which are furthermore connected to the active receiver component 81a and the active emitter component 81b. By capturing and reemitting the ultrasonic/mechanic vibration of the object of inspection 1, property altering device 8 is providing a real-time mechanical feedback, which therefore changes the mechanical properties of the object of inspection 1. The vibrations of the object of the inspection in acoustic or ultrasonic frequency range can be altered by changing the control parameters of the electro-ultrasonic control unit 81e. This allows the manipulation of the mechanical properties of the test object. The test object can therefore mechanically respond similarly as if it would contain damages, or have induced changes in its geometry or material, which can in this case be easily manipulated. This opens the possibility for generation of diversified datasets obtained on real test objects and improvement of the training process of machine learning algorithms, which can therefore be made robust to the specific change of mechanical or other system properties.

A change in properties of the test object with the property altering device may be achieved by providing the property altering device with an analog electro-ultrasonic control device. The analog electro-ultrasonic control device is similar to the above described digital electro-ultrasonic control device except that the analog to digital and digital to analog converters are omitted and the signal is processed analogously.

An amplified ultrasonic signal may be manipulated at will(e.g. it is filtered, delayed, amplified or similar) using analog electrical components (including operational amplifiers, resistors, inductors, transistors, capacitors, diodes, and others) by adapting resistance, inductance, capacitance, etc. The type of manipulation depends on the type of diversity (different properties of the test specimen), which one would like to include in the reference training database. Phase delay at different frequencies is representing having a similar effect than the geometry change or the change in the material of the test object. Attenuation at different frequencies is representing the presence of porosity or microcracks. Nonlinear electrical components are included in the analog electro-ultrasonic control device to simulate mechanical nonlinearities and other indicators of damages. Examples of nonlinearities are: change of the ultrasound or vibration amplitude if the excitation frequency is changed, generation of frequency components by the object of inspection 1 including the property altering device 8 that are not included in the excitation form, presence of excitation modes, which are (depending on the geometry of the object of inspection 1) not present in the damage-free object, clapping of delamination, micro friction of cracks, and others. The electrical signal is amplified and adjusted to match the requirements of the Active emitter component 81b to provide ultrasonic feedback to the object of inspection 1.

Figure 9 shows an exemplary circuit of an analog electro-ultrasonic control device comprising a piezoelectric sensor applied on the test object, a charge input amplifier, a proportional -integral -derivative (PID) controller, a phase shifter, a generator of additional signal components, an output amplifier, and a piezoelectric actuator applied on the test object. The specifications of resistors R1 to RIO and capacitors Cl to C2 depend on the frequency range (10 Hz to 10 MHz). Resistors RT1 to RT4 and compensators CT1 to CT2 are tunable and controlled by the controller of the property altering device. The amplifier proportional-integral-derivative (PID) controller is used to simulate the changes of the ultrasonic signal representing the presence of defects or geometric/material changes in the test object. A generator of additional signal components may be included in the circuit to induce nonlinear effects, typically indicating presence of internal damages (e.g. frequency shift if amplitude of the signal amplitude is changed). A phase shifter in the circuit allows to simulate the change of test object geometry, which can therefore be tuneable (e.g. plate thickness).

Referring to figures 8a to 8e, exemplary embodiments of an ultrasound producing device 3 comprising an ultrasound generator 3a and an ultrasound reader 3b are illustrated. The transducers of the ultrasound generator 3a and ultrasound reader 3b may be based on a piezoelectric effect, a piezostrictive effect, a magnetostrictive effect, or a capacitance (capacitive micromachined ultrasonic transducers). Alternatively, thermoacoustic emitters can be used.

The transducers of the ultrasound generator 3a and ultrasound reader 3b can be placed on the test object surface 2 as illustrated in figures 8d and 8e, or in the vicinity of the test object 1 for the air-coupled transducers as illustrated in figures 8a, 8b and 8c. The ultrasound generator 3a and ultrasound reader 3b may be positioned on the same side of the test object 1 as illustrated in figures 8a, 8b and 8e, or on opposite sides of the test object as illustrated in figures 8c and 8d.

All combinations of the following methods are possible. Ultrasound/vibration can be excited by laser pulses, contact transducers, or air-coupled transducers and ultrasound/vibration can be detection by a laser interferometer, a laser vibrometer, a beam deflection probe, contact transducers, or air-coupled transducers. Nonlinear effects may be detected using air-coupled ultrasonic transducers by varying the ultrasound power or specimen-to-transducer distance. If in doing so the spectrum of the detected ultrasound changes, the presence of internal damages is highly probable.

Ultrasound can be detected by an ultrasound reader 3b with a broadband transducer or a plurality of narrowband transducers, sensitive at different characteristic frequencies, which can, for example, be a multiple of the ultrasound generation frequency (frequency of the peak amplitude in the spectrum of the ultrasound emitted by the active emitter component 81a) or for several kHz different (slightly lower or higher) from the ultrasound generation frequency.

Example of an experimental setup to generate large datasets of ultrasonic signals relevant for duality inspection using a laser-based inspection system.

A reconfigurable defect is simulated by a heating laser projecting a short line on a shape memory polymer foil, which has a special property that its Young’s modulus and ultrasound attenuation level can locally be controlled by its temperature field. The shape memory polymer foil is a specific example of a property altering device referred to previously. The ultrasound is generated in this example by a laser pulse at one fixed position and detected in this example by a laser vibrometer at another fixed position for 64 different defect positions and 3 different configurations of the same disordered specimen. The term “specimen” used in this example represents the test object referred to previously.

We study robustness of neural networks in cases of reduced and partly disassociated training datasets. In our first study, we classify the specimen configurations with the defect position being the disturbing parameter. The model shows high performance on a dataset of signals obtained at all the defect positions also if only trained on a fully different dataset containing signals obtained at few defect positions. In our second study we reverse the problem and perform the defect localization. The model become robust to the change of the specimen configuration if a reduced dataset containing signals obtained at two different specimen configurations is used for the training process. The following conclusions may be drown, which are also true for industrial quality control processes where the reconfigurable specimen is exchanged by a high number of test objects with different properties: (i) it is not necessary to decode the wave propagation and the measurement system can be uncalibrated - their properties will be contained in the parameters of the trained model; (ii) considerable performance is achievable also by a training process on small-sized datasets or on datasets obtained under dissimilar conditions; (iii) model robustness on specific parameters can be achieved by adequately diversified datasets.

Emerging technologies based on machine learning algorithms or artificial neural networks are transforming the way of information processing [1-3], Audio signal interpretation by machine learning algorithms is a broadly addressed topic with typical applications for already highly reliable speech recognition and sound classification [4-6], Can ultrasonic signals be processed in a similar way and provide advantages of automation and improved performance, as it is the case for implementing image recognition algorithms for visual detection and classification of surface damages [7-9] and analysis of radiography images [10, 11],

Insufficient amount of appropriate training data is the main reason why machine learning algorithms have not successfully penetrated the sector of industrial ultrasonic inspection yet. It is easier to generate labelled dataset for a problem, which can easily be solved by a human, than a dataset of ultrasonic signals labelled with the target information. In the latter case, it is difficult to obtain diversified and large dataset, as we require a reference inspection method, which is in most cases destructive (e.g. microscopy of the cut section).

Most of the current ultrasonic inspection methods are relaying on relatively basic principles based on, for example, observing reflections from internal features or increase of ultrasound attenuation levels caused by delaminations or porosity in through transmission setup [12-15], More advanced methods uses spectral analysis of the signal to extract additional information [16-20],

State of the art: Machine learning for ultrasonic inspection

Analytically and numerically supported training process

Large datasets for training the neural network for ultrasonic signal interpretation can be generated numerically or analytically. However, this approach is facing a strong assumption that the exact model of the inspection system is known. Geometric and material properties of a specimen and target features, as well as characteristics of ultrasound generation and detection must be numerically or analytically describable as precisely as possible. Previous realizations typically built a training dataset on a high number of signals obtained on a theoretical model. Some included a small number of experimentally obtained signals. This is clearly an elegant way, which is however raising a question if the machine learning data interpretation is really necessary in case the physical model is already known. It is more relevant for practical application to use machine learning on a dataset containing undecoded ultrasonic signals. In this case the ultrasound generation, propagation, and detection properties will be coded in trainable parameters of the neural network and the ultrasonic signals can be interpreted without actually known the physical model of the measurement setup.

The following studies performed the training process on theoretically obtained datasets consisting of laser ultrasonic signals. In one of the early realizations, numerically simulated surface wave dispersion curves of layered structures were used as training and testing datasets [21], The model was experimentally tested on two specimens. Numerical simulations including signal pre-processing based on wavelet decomposition are used as an input to train the neural network [22], The method is tested on a single composite plate specimen to determine elastic constants of the material. Theoretical dispersion curves were used for the network training [23], The method was tested experimentally on one plate by measuring its plate thickness, Young’s modulus, and Poisson’s ratio. Dynamic responses calculated by a finite-element method were used for the network training [24] . In order to test the method, location and size of the cuts on the specimen size opposite to the ultrasound excitation and detection were experimentally determined on two specimens.

Numerical simulations of wave propagation in the specimen were used to train the neural network [25], Maximum, minimum, and peak-to-peak values of Rayleigh waves together with the signal peak frequency and its bandwidth were used as input parameters to determine location and size of subsurface cracks. Part of the simulated signals and three experimental signals were used to validate the model. Numerical model was utilized to generate ultrasonic signals, which were converted to images and used for training an already pretrained convolutional neural network, which was derived from visual recognition tasks [26], Four experimentally obtained signals were included in the training and the validation dataset in order to determine location and size of subsurface defects.

An interesting method for training data augmentation was based on a technique used for human inspector trainings, which allowed generation of virtual flaws with variable parameters (e.g. depth or size) by numerical modification of experimentally obtained signals [27, 28], As it was the case for other methods based on numerically or analytically trained networks, prior knowledge (relevant physical model) on the measurement system and the specimen properties were required for this method as well.

Training process using experimentally obtained data

It is technically challenging to produce a high number of reference specimens with known target parameters in order to train the neural network on purely experimental dataset. In previous studies, the larger amount of the training data was typically obtained by repeating the measurement (eventually at different positions) on a small number of specimens. Consequently, the low diversity of the training datasets limited robustness of the model, which typically provided good performance, but only for a specific configuration of the experimental system and for one specific inspection task.

Defect classification of porosity, lack of fusion, tungsten inclusions, and intact specimens were performed applying wavelet pre-processing of ultrasonic signals obtained by a piezoelectric transducer [29], The training and testing datasets were altogether consisting of 240 ultrasonic signals obtained on 9 specimens. Presence of a notch in a metallic plate was detected by a neural network trained on a dataset of 216 numerically and 24 experimentally generated signals, obtained by varying the location of the transducers [30], An elegant way to obtain larger dataset of ultrasonic signals is to use phased array ultrasonic transducers [31-34], Signals of individual piezoelectric elements at different excitation parameters have only limited diversification if applied on low number of test objects. Crack orientation and depth were evaluated on a single specimen using wavelet packet decomposition [31], 12 different defect shapes provided 240 signals of different steering beams angles, which were divided to the training and testing datasets. The presence of 68 different defects (various holes and crack types) in 6 steel blocks was estimated [32], The training and testing datasets were consisting of more than 4000 linear scan images obtained by phased-array probe and augmented by flipping, random cropping, translation, and visual color adjustments. The performance of neural networks can be improved by a hybrid training process, as shown by two studies addressing detection of holes [33] and pipeline cracks [34], Both used convolutional neural networks, which were consecutively trained on simulated data and experimental signals obtained by a phased-array probe.

Machine learning algorithms have been used to support ultrasonic quality control of spot welding by classifying them into four quality levels [35] and to predict their static and fatigue behavior [36], X- ray computed tomography scans were used as a reference to train a neural network to estimate porosity level in carbon fiber reinforced polymers [37], Similarly, electron backscattered scattering diffraction served as a reference measurement to measure grain size of polycrystalline metals using laser ultrasound [38],

Ultrasonic signals were captured at 10,000 different locations on four samples before and after applying a damage (in form of a mass) [39], The training dataset was obtained by randomly picking different snapshots of a two-dimensional scan [40], These two training data augmentations had only limited contribution to the robustness of the machine learning algorithms.

Subwavelength information extraction by machine learning algorithms

Machine learning algorithms can be used to extract specific information with the resolution below the diffraction limit. Typical approach is to record wave responses scattered from the object in the far- field and perform learning process for different arrangements of the target information (e.g. position or shape of the imaged object). The inconveniences are that an arrangement variation is not always achievable and the method is limited to datasets closely related to the training conditions.

A study was performed using waves on a plate with differently shaped holes as defects using six numerical simulations and five physical specimens [41], Size and variability of the training datasets were augmented by a random cropping, zooming, flipping, and rotating. The method is able to classify between different shapes, which were used during the training process, however, it is not able to distinguish arbitrary and unknown subwavelength shapes of defects. It was shown that coupling of subwavelength information to the far field can be improved by placing randomly distributed resonators in the near field of an object [42], Subwavelength images of digits and numbers drawn by a two-dimensional array of speakers (emitting in the audible frequency range) were reconstructed from the signals of four microphones placed in the far field.

In the microwave domain, an object was localized with a subwavelength resolution in a chaotic cavity. Coded-aperture imaging effect was achieved by a metasurface consisting of an array of individually tunable boundary conditions. The signal was captured at a single frequency and at a single location but for a fixed series of random configurations of the metasurface for each of the object locations [43],

Ultrasonic localization based on machine learning

Two studies were addressing the localization problem by ultrasound analysis based on machine learning. The first was demonstrating a system able to localize a finger touch on a metal plate [44] . The sensing area was surrounded by 4 transmitters and 12 receivers and all the 144 (12 x 12) touch positions were used in the training process. The second addressed the source localization of ultrasound emitted by pencil lead- breaks on the surface of a composite plate [45], 8 sensors were used to classify between five zones. In both cases, the target localization resolution was not significantly below the training resolution.

Knowledge gap

The main challenge is that typically the machine learning algorithms only work reliably under the conditions closely related to those of the training process. They get ineffective if ultrasound acquisition parameters, ultrasound coupling properties, specimen characteristics, probe location, or other conditions change.

Majority of the previous machine learning experimental studies in the domain of ultrasonic inspection suffered of low data diversity, as it is practically challenging to obtain a suitable experimental database containing multiple target feature (defect) types and system configurations.

Our study is the first using a reconfigurable defect to experimentally obtain large volumes of diverse datasets of ultrasonic signal. We are able to vary both the defect position and the specimen configuration, which both affect the ultrasound propagation. This opens the possibilities to study the robustness of the model and what size of the training dataset is required for certain performance and how related it should be to the testing dataset.

Our method is able to achieve the subwavelength localization with the resolution up to 5 times below the one of the training data. Methods

Specimen description

The key element of the specimen was a foil with 0.2-mm thickness made of a shape memory polymer (SMP), (manufacturer: SMP Technologies Inc, Tokyo), which had a special property that its glass transition temperature laid in the range between 25°C and 90°C. As its Young’s modulus continuously fell at least for factor 20 by increasing its temperature for several 10°C above the room temperature [46], it was an appropriate material to simulate a property altering defect by local heating. A localized decrease of Young’s modulus and increase of ultrasound absorption has similar effect on the ultrasound as a crack, local porosity, or a local change of material or geometric properties. Its advantage exploited in this study is that size shape and location of such a defect are easily reconfigurable, if we have control over the temperature field of the SMP foil.

The SMP foil was placed in an arbitrary shaped frame (Fig. 10.1) consisting of two aluminium plates pressed together. The shape of a frame hole was providing us disordered wave reflections and propagation. One part of the specimen was covered by carbon powder in order to achieve high light absorption and another part was covered by a retroreflective foil. The mechanic properties of the specimen were changed by sticking a first (100 mg) and a second mass (20 mg) on the SMP foil at the position marked in Fig. 10. la. This provided us three specimen configurations: the first configuration is without mass, the second configuration is with both masses, and the third configuration is with the mass number 1 alone. In order to additionally increase the chaotic shape of the specimen, a notch with the length of 2.5 mm was induced in the middle of the positions of the reconfigurable defects (simulated by heated spots).

Experimental setup

Ultrasound was excited at a single arbitrary chosen location by a laser pulse with wavelength of 500 nm, energy of 9 mJ, duration of 5 ns (full width at half-maximum), and repetition rate of 20 Hz using a Surelite SL 1-20 pump laser together with a Surelite OPO Plus optical parametric oscillator (manufacturer of both: Continuum). The precise wavelength of the ultrasound excitation laser is not relevant and was chosen arbitrary. An advantage of using an optical parametric oscillator was that we were able to detune its optimal optical configuration and cause the excitation laser to become unstable. For each of the ultrasound excitation, pulse energy, shape, and diameter of the laser beam were randomly varying for up to 50%. This decreased the repeatability of the ultrasound generation, bring more noise in the measurement and provide us chaotic signals, which are more challenging but more interesting to be evaluated by machine learning algorithms.

Ultrasound was detected at a single location using PSV-F-500-HV laser vibrometer (manufacturer: Polytec) with 15 signal averaging. A third laser used in our experiment was a SuperK supercontinuum white light laser with a SuperK Varia tunable wavelength filter (manufacturer of both: NKT Photonics). Its power was approximately 0.2 W and the wavelength range was arbitrary chosen to range from 400 nm and 500 nm. A single wavelength laser could also be used instead. XG210 2-axis galvanometer scan head (manufacturer: Mecco) was used to project a 2 mm long and 0.7 mm broad line and locally heat the SMP foil. By the local drop of the Young’s modulus and local increase of the ultrasound absorption, a reconfigurable defect was created.

Acquisition of datasets

Altogether 3840 signals were captured with the length of 250 points and the sampling frequency of 625 kHz. 20 measurements were repeated for each of the 64 reconfigurable defect positions (8x8 positions in two dimensions defined as x and y) and for each of the three specimen configurations (without mass, with both masses, and with a mass number 1 alone). The step distance in the x and y directions between the two defect positions were 0.4 mm.

The 20 repeated signal measurements for each of the defect positions and each of the specimen configurations were firstly divided to 13 signals exclusively used for the model training and remaining 7 signals exclusively used for the model testing. This was true for all the studies of our work. As described in the following section, the number of defect positions and specimen configurations was additionally reduced for the model training.

A part of raw measured signals is shown in Fig. 10.2. Change in the direction x of the defect positions at y equals 1.2 mm is coded by the shades of gray in the left-hand column (x = 0 mm for the black, x = 3.2 mm for the bright gray). Change in the direction y of the defect positions at x equals 1.2 mm is coded by the shades of gray in the right-hand column (y = 0 mm for the black, y = 3.2 mm for the bright gray). The signals of the same shade of gray were consecutively obtained by the repeated measurement at the same defect position. Each of the three rows represents from the top to the bottom: the first specimen configuration (without additional mass), the second specimen configuration (with two masses), and the third specimen configuration (with only mass number 1 attached on the SMP foil). Please note that for the purpose of transparency, a single x and a single y position was chosen to demonstrate the signal variation in the y and x direction, respectively. For the same reason, only 13 repetitions, which were used for the model training, are shown instead of all the 20 measurements at each of the defect position and at each of the specimen configuration.

The shape of the signal has complex and unknown dependency on the target parameters - the specimen configuration (presence of mass) and the defect position. From the raw signals shown in Fig. 10.2, it visually appears that the latter has stronger influence on the signal. The goal is to interpret the signals by extracting the target parameters without actually knowing the wave propagation and the measurement system properties.

Results and discussion

Two studies were performed on the datasets obtained as described above. The first was a classification of the specimen configuration and the second was a localization of the reconfigurable defect. While the classification problem has a discrete output (label number) the localization has two analog outputs (defect positions x and y).

Classification of the specimen configurations

The aim of the first study is to classify the ultrasonic signals obtained at three different specimen configurations (three rows in Fig. 10.2). In this case, a disturbing factor is the defect position, which significantly affects the signal shape and should be eliminated (marked by the shades of gray in Fig. 10.2).

Model architecture and training process

At first, the raw signals were converted to the frequency domain by the fast Fourier transform. The first half of the real and imaginary component values (125 values each) were merged together and used as a parallel input for the neural network.

The model consisted of a fully connected neural network with an input of 250 elements, two hidden layers

- the first with 200 elements and the second with 20 elements. The last (output) layer had 3 elements for 3 classes: first, second, and third specimen configuration. All three layers were linear and have a rectified linear unit activation function. An Adam optimization algorithm was used. The loss parameter was calculated by a cross entropy loss function. The neural network was trained by 50 epochs - passes of the entire training dataset. The batch size was always equal to the whole training dataset size, which was a changing variable in our research as described below.

In Fig. 10.3, we show results of the classification between the three different configurations ofthe specimen

- between the three rows of Fig. 10.2. We test the performance of the machine learning algorithm on the diversified dataset if a reduced amount of training data is used. Signals obtained at a limited number of defect positions (which is here the disturbing parameter) was used for the model training (starting with a single defect position), while it was tested for all the defect positions. The number of signals used for the training was reaching from 13 (13 repetitions at a single defect position) to 832 (13 repetitions at 64 defect position). The model was always tested on the dataset obtained at all the 64 defect positions with 7 repetitions each - altogether 448 signals, which were always all different from the training dataset. The final performance of the model tested for all the defect position depends on the defect positions we choose for the model training. The defect positions lying in the middle of the defect position area deliver better performance comparing to those lying on the edge of the scan. In our statistical approach, we repeated the training process for 20 randomly chosen defect positions and determined the labelling accuracy (red full line) and the loss parameter (blue dashed line) at the end of the learning process (Fig. 10.3). They are represented by the mean value and the shadowed standard deviation range.

The model has a moderate performance on the testing dataset containing signals of all defect positions if the training is performed on a dataset only containing the signals of one defect position (56% labeling accuracy). If additional defect positions are included in the training dataset, the accuracy rapidly increases and reaches 90% if only 8 defect positions are used for the training process. This is remarkable as the defect position (shades of gray in Fig. 10.2) visually appears to have higher influence on the signal shape than the specimen configuration (three rows in Fig. 10.2). Influence of the disturbing factor (changing defect position) is eliminated also if training is performed on datasets with limited diversity (limited number of detect locations) and the model robustness is extended to related, but previously unseen data.

Defect localization

The second study of our work is using the same experimental data as the first study, but we reverse the problem. Its goal is to localize the defect (which affects the signal as marked by the shades of gray in Fig. 10.2) and a disturbing factor is the change of the specimen configuration (three rows in Fig. 10.2), on which the model should be make robust.

Model architecture and training process

Signal pre-processing for the second study of this work was the same as for the first study: signal conversion to the frequency domain and parallel merging of the real and imaginary components.

The model consisted of a fully connected neural network with an input of 250 elements, two hidden layers - the first with 500 elements and the second with 100 elements. The last layer had only two elements which were analog and directly represented the defect positions x and y. The loss function was defined as mean absolute error of these two values. Similarly, as for our first study, linear layers and a rectified linear unit activation function were chosen, together with the Adam optimization algorithm and a scheduler reducing the initial learning rate of 0.01 for 10% every 100 epochs. Because of a higher range of possible outputs (defect positions in two dimensions), and because it is about localization and not classification, the learning process for the second study of this work was approximately hundred times longer than for the first study. The training process comprised 1000 epochs. The batch size was always equal to the whole training dataset size, which was a changing variable (described below), as it was the case for our first study as well. Less efficient alternatives of model architecture and training procedure

Typically, a higher number of neural network layers and a higher number of elements in each layer improved the performance of the model. However, higher model complexity made the training process slower. The performance of the model was significantly reduced if only one hidden layer was used and a neural network with a single layer (similar to multi-variable regression) was inefficient. If the raw signals are directly used as the input for the neural network (without conversion to the frequency domain), its performance decreases for around 10%. Stochastic gradient descent was less efficient than the Adam optimizer.

Reduced training datasets and model robustness

In order to estimate the robustness of the model for the defect localization, we tested its performance when reduced datasets and datasets significantly different to the testing dataset were used for the training process. Please note that during the second study of our work, the 7 signal repetitions measured at each of the defect positions obtained at the third specimen configuration were always used for the model testing. These signals were never included in the training datasets.

For the first training process, we used a dataset consisting of 13 repetitions at a single randomly chosen defect position obtained at the first specimen configuration. The second training process used dataset consisting of 13 repetitions at two randomly chosen (different from each other) defect positions at the first specimen configuration; and so on until the 64th training process, where we used a dataset consisting of 13 repetitions at all of the 64 defect positions at the first specimen configuration. In order to reduce the uncertainty of randomly reducing the training dataset, we repeated this procedure for 20 times. In order to obtain the black dotted curve in Fig. 10.4a and Fig. 10.4c, we performed altogether 1280 training processes - at 20 randomly chosen defect positions with the number of different defect positions used for the training process reaching from 1 to 64.

The same procedure was repeated for the second specimen configuration and for the combination of the first and the second specimen configuration. For the latter, we used the same randomly chosen defect positions for datasets obtained at both specimen configurations. This delivered us the blue dashed line and the red dash-dot line in Fig. 10.4a and Fig. 10.4c, respectively.

Subsequently, we gradually added the ultrasonic signals obtained at the third specimen configuration. As described in the previous section, the training and the testing data remained not to be fully different because of the varying properties of the ultrasound excitation.

The first training process used a dataset consisting of 13 repetitions at a single randomly chosen defect position at the third specimen configuration, together with 13 repetitions at all defect positions at the first specimen configuration - altogether 13x(64+l) signals. The second training process used a dataset consisting of 13 repetitions at two always randomly chosen (different from each other) defect positions at the third specimen configuration, together with 13 repetitions at all defect positions at the first specimen configuration - altogether 13x(64+2) signals; and so on until the 64th training process, where we used a dataset consisting of 13 repetitions at 64 defect positions at third specimen configuration and 13 repetitions at all of the 64 defect positions at the first specimen configuration - altogether 13 x (64+64) signals. In order to obtain the black dotted curve in Fig. 10.4b and Fig. 10.4d, we performed altogether 1260 training processes - at 20 randomly chosen defect positions with the number of different defect positions at the third specimen configuration used for the training process reaching from 1 to 64.

The same procedure of gradually adding more of the defect positions obtained at the third specimen configuration was repeated while exchanging all the defect positions at the first specimen configuration with all the defect positions at the second specimen configuration (blue dashed line in Fig. 10.4b and Fig. 10.4d). The procedure was followed for the third time using all the defect positions at the first and the second specimen configuration together (red dash-dot line in Fig. 10.4b and Fig. 10.4d). As a reference, we repeated the same procedure for the fourth time without any training data of the defect positions obtained at the first and the second specimen configuration (green full line in Fig. 10.4b and Fig. 10.4d).

The positioning error in Fig. 10.4 a and b represents the mean value of absolute errors between the predicted and the real defect positions in the dimensions x and y. If this absolute error is smaller than 0.2 mm for both dimensions, which is the half of the minimal distance between the two defects, its position is labelled correctly. The percentages of correct labelling are shown in Fig. 10.4 c and d.

In Fig. 10.4 a and Fig. 10.4 c, we can observe that the performance of the model on the testing dataset, obtained at the specimen configuration different from the training dataset (third specimen configuration), is increased if the model is trained on a mixed dataset containing signals obtained at two different specimen configurations (red dash-dot line). In this case, the model is robust to the change of the specimen configuration and the mean positioning error reaches the values below the half of the minimal distance between two defect positions used for the training process in this study (0.4 mm). Consequently, the labelling accuracy increases, while it remains low if the undiversified training dataset is used (black dotted and blue dashed line).

If the training process is performed on the dataset obtained at the specimen configurations different to the testing dataset, around 21 randomly chosen defect positions already deliver performance close to the one when all the defect positions are used for training. The end values of Fig. 10.4 a and c are equal to the start values of Fig. 10.4 b and d. If the model is trained by a dataset containing all defect positions obtained at two different specimen configurations (red dash-dot line), both different to the testing datasets, the positioning error is decreased almost for the factor of 2 and the labelling accuracy is improved for the factor of 5, in comparison to the case when a dataset only containing signals obtained at the first (black dotted line) or the second (blue dashed line) specimen configurations.

In this case the model performance is high already when including only a few additional defect positions from the dataset obtained at the third specimen configuration equivalent to the testing. After 11 additional defect positions, positioning error below 0.1 mm and labeling accuracy of 80% is achieved (red dash-dot line in Fig. 10.4d). Neural network without the training on previous (different) datasets achieves the same performance when 35 different defect positions are used for the training.

Please note that the datasets of the ultrasonic signals obtained at the three specimen configurations are significantly different from each other, as proven by the classification problem described in the first study of this work.

In the first row of Fig. 10.5, we show the performance of the model during the training process at the situation marked with the orange vertical line in Fig. 10.4. The final performance (after 1000 epochs) for the defect localization on the training dataset (second row) and the testing dataset (third row) is visually represented. In order to provide better comparison, the training dataset always comprised the same 3 defect positions at the third specimen configuration (red dots with full outline). Similar as for Fig. 10.4, where the defect positions were always chosen randomly, 13 ultrasonic signals repetitions at each of these three positions were used for the first column of Fig. 10.5. We additionally included all the defect positions obtained at the first specimen configuration (green dots with dashed outline) for the second column in Fig. 10.5, all the defect positions obtained at second specimen configuration (orange dots without outline) for the third column in Fig. 10.5, and both datasets (obtained at the first and second specimen configurations) for the fourth column in Fig. 10.5.

The testing dataset was always the same - all defect positions at the third specimen configuration. In order to better distinguish between the different defect positions, red and blue colors are alternating in x dimension and full and empty filling are alternating in y dimension. In the background, the gray shading shows the mean absolute error of each of the defect with the black representing the largest error.

The localization is not possible if only three defect position are used to train the model (first column in Fig. 10.5), because of the lack of the information about the rest of the defect positions. The performance of the model significantly improves, if other defect positions are included in the dataset, even though that they were not obtained at the same specimen configuration. Comparing the second and third column of Fig. 10.5, we can observe that the second specimen configuration is more similar to the third specimen configuration, because of its slightly better training performance. However, labelling accuracy does not lift above 50% and the mean absolute error is not much below 0.17 mm.

The model became robust to the change of the specimen configuration if all the defect positions obtained at the first and second specimen configuration are used for training. Labelling accuracy of 80% and mean absolute error of 0.25 mm is achieved already if only three defect positions at the specimen configuration equivalent to the testing dataset are used for training.

Predicted defect positions are scattered also when the method is tested on the training dataset. This is showing the high stochastic variability of all the datasets because of the low repeatability of the ultrasound excitation, which was induced artificially.

Conclusions

We introduced a novel experimental method to generate large labelled datasets of ultrasonic inspection signals. This opened a possibility to validate and optimize the architecture of machine learning algorithms for the application of ultrasound signal interpretation. We studied their performance and robustness if limited and/or only partly related training datasets are used for the model training.

We proved that by the use of machine learning algorithms, it is possible to extract information inaccessible by conventional ultrasonic signal processing methods. The algorithm can be trained to identify relevant signal parameters with their interdependences and to correlate them to the desired output. The mechanism behind cannot be fully interpreted by human, as well as it is in the most cases impossible to visually differentiate between differently labelled ultrasonic signals.

The relevant information can be extracted from the ultrasonic signals without knowing the physical model of the system i.e. without actually understanding the wave propagation (differentiating between the wave types, reflections etc.). In the case of our study, it would not be possible because of the disordered specimen shape and undefined and unstable laser excitation properties. The additional advantage of machine learning ultrasound interpretation is that the ultrasound excitation and detection device is not necessary to be calibrated. In a certain way, the training process itself is the calibration process of the whole system including wave propagation in the specimen, ultrasound excitation, detection, and coupling properties, together with the signal processing elements, which can all be undefined. In the first study of our work, the model trained on a reduced dataset (reduced number of different defect positions) is able to discern with high reliability the specimen configurations and is robust to the change of the defect position, which is otherwise having a stronger effect on the signal shape.

We reversed the problem for the second study of our work. If a mixed dataset of two specimen configurations is used for the training process, the model for defect localization become robust to the different related change of the specimen configuration. The localization error is smaller than 0. 1 mm (four times less than the minimum resolution of the training dataset) if only 10 additional training defect positions obtained at the specimen configuration equivalent to the one of the testing dataset are included.

This is showing that if the training dataset is sufficiently diversified, it is possible to make the neural network for ultrasound interpretation robust to the change of the specimen shape, material properties, ultrasound excitation (e.g. energy, position, form) detection (e.g. position, gain) and other system properties. Likewise, it can be robust to disturbances (in our case variation of ultrasound excitation properties) and to the certain degree of noise level. Our results show also that model performance, sufficient for multiple applications, can also be achieved if the model is trained on a small-sized dataset partly different from the testing dataset.

These outcomes of our research show a high potential of the machine learning algorithms for the ultrasound signal interpretation for the industrial inspection. Reduced size of training datasets would suffice for various cases and the training is not even necessary to be performed on the dataset closely related to the testing dataset. If the testing object or the inspection system is moderately changed, only a small size of the training data is required to achieve the same correctness. After this change and an additional training, the model becomes increasingly more robust to this type of change.

This has practical advantages for modem lean industries demanding high flexibility. Only several test objects are necessary to be inspected destructively for the purpose of the model training, while the quality state of the rest can be with high fidelity estimated by non-destructive inspection method and ultrasound signal interpretation based on machine learning.

We expect that the results of our work are transferable to different ultrasonic other inspection methods (e.g. a single-point piezoelectric transducer or radiography methods), as well as to additional defect characteristics (e.g. size or orientation).

Literature References

[1] X.W. Chen, X. Lin, Big Data Deep Learning: Challenges and Perspectives, IEEE Access 2 (2014) 514-525. https://doi.org/10.1109/ACCESS.2014.2325029 [2] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature 521(7553) (2015) 436-444. https ://doi .org/ 10.1038/nature 14539

[3] J.R.R. Uijlings, K.E.A. van de Sande, T. Gevers, A.W.M. Smeulders, Selective Search for Object Recognition, International Journal of Computer Vision 104(2) (2013) 154-171. https://doi.org/10.1007/sl l263-013-0620-5

[4] G. Guodong, S.Z. Li, Content-based audio classification and retrieval by support vector machines, IEEE Transactions on Neural Networks 14(1) (2003) 209-215. https://doi.org/10.1109/TNN.2002.806626

[5] I. McLoughlin, H. Zhang, Z. Xie, Y. Song, W. Xiao, Robust Sound Event Classification Using Deep Neural Networks, IEEE/ACM Transactions on Audio, Speech, and Language Processing 23(3) (2015) 540-552. https://doi.org/10.1109/TASLP.2015.2389618

[6] K. Noda, Y. Yamaguchi, K. Nakadai, H.G. Okuno, T. Ogata, Audio-visual speech recognition using deep learning, Applied Intelligence 42(4) (2015) 722-737. https://doi.org/10.1007/sl0489- 014-0629-7

[7] A. Caggiano, J. Zhang, V. Alfieri, F. Caiazzo, R. Gao, R. Teti, Machine learning-based image processing for on-line defect recognition in additive manufacturing, CIRP Annals 68(1) (2019) 451-454. https://doi.Org/10.1016/j.cirp.2019.03.021

[8] H.Y.T. Ngan, G.K.H. Pang, N.H.C. Yung, Automated fabric defect detection — A review, Image and Vision Computing 29(7) (2011) 442-458. https://doi.Org/10.1016/j.imavis.2011.02.002

[9] X. Tao, D. Zhang, W. Ma, X. Liu, D. Xu, Automatic Metallic Surface Defect Detection and Recognition with Convolutional Neural Networks, Applied Sciences 8(9) (2018). https://doi.org/10.3390/app8091575

[10] P. Fuchs, T. Kroger, C.S. Garbe, Defect detection in CT scans of cast aluminium parts: A machine vision perspective, Neurocomputing 453 (2021) 85-96. https : //doi .org/ 10.1016/j .neucom .2021.04.094 14

[11] M. Schlotterbeck, L. Schulte, W. Alkhaldi, M. Krenkel, E. Toeppe, S. Tschechne, C. Wojek, Automated defect detection for fast evaluation of real inline CT scans, Nondestructive Testing and Evaluation 35(3) (2020) 266-275. https://doi.org/10.1080/10589759.2020.1785446

[12] J. Rus, A. Gustschin, H. Mooshofer, J.-C. Grager, K. Bente, M. Gaal, F. Pfeiffer, C.U. Grosse, Qualitative comparison of non-destructive methods for inspection of carbon fiber-reinforced polymer laminates, Journal of Composite Materials 57(27) (2020) 4325-4337. https://doi.org/10.1177/0021998320931162

[13] J. Rus, D. Kulla, J.C. Grager, C.U. Grosse, Air-coupled ultrasonic inspection of fiber-reinforced plates using an optical microphone, Proceedings of German Acoustical Society, DAGA Rostock (2019) 763-766

[14] E. Blomme, D. Bulcaen, F. Declercq, Air-coupled ultrasonic NDE: Experiments in the frequency range 750 kHz-2 MHz, NDT & E International 35 (2002) 417-426. https://doi.org/10. 1016/80963-8695(02)00012-9

[15] ASTM E797/E797M-15 Standard Practice for Measuring Thickness by Manual Ultrasonic Pulse- Echo Contact Method, ASTM International, West Conshohocken, PA (2015). https://doi.org/10.1520/E0797_E0797M-15

[16] J. Rus, C.U. Grosse, Thickness measurement via local ultrasonic resonance spectroscopy, Ultrasonics 109 (2021) 106261. https://doi.Org/10.1016/j.ultras.2020.106261

[17] J. Rus, C.U. Grosse, Local ultrasonic resonance spectroscopy: A demonstration on plate inspection, Journal of Nondestructive Evaluation 39(2) (2020). https://doi.org/10.1007/sl0921- 020-00674-5

[18] A. Migliori, J.L. Sarrao, Resonant ultrasound spectroscopy : applications to physics, materials measurements, and nondestructive evaluation, A Wiley-Interscience publication, New York, Weinheim (1997)

[19] A. Jtingert, C. Grosse, M. Kruger, Local Acoustic Resonance Spectroscopy (LARS) for Glass Fiber-Reinforced Polymer Applications, Journal of Nondestructive Evaluation 33(1) (2014) 23- 33. https://doi.org/10.1007/sl0921-013-0199-3

[20] I. Solodov, J. Bai, G. Busse, Resonant ultrasound spectroscopy of defects: Case study of flat- bottomed holes, Journal of Applied Physics 113(22) (2013) 223512. https://doi.org/10.1063/E4810926 [21] C.D. Tsai, T.T. Wu, Y.H. Liu, Application of neural networks to laser ultrasonic NDE of bonded structures, NDT & E International 34(8) (2001) 537-546. https://doi.org/10.1016/S0963- 8695(01)00015-9

[22] J. Yang, J. Cheng, Y.H. Berthelot, Determination of the elastic constants of a composite plate using wavelet transforms and neural networks, The Journal of the Acoustical Society of America 111(3) (2002) 1245-1250. https://doi.org/10.1121/L1451071

[23] F. Lefevre, F. Jenot, M. Ouaftouh, M. Duquennoy, P. Poussot, M. Ourak, Laser ultrasonics and neural networks for the characterization of thin isotropic plates, Review of Scientific Instruments 80(1) (2009) 014901. https://doi.org/10.1063/L3070518

[24] A. Oishi, K. Yamada, S. Yoshimura, G. Yagawa, S. Nagai, Y. Matsuda, Neural Network-Based Inverse Analysis for Defect Identification with Laser Ultrasonics, Research in Nondestructive Evaluation 13(2) (2001) 79-95. https://doi.org/10.1080/09349840109409688

[25] K. Zhang, G. Lv, S. Guo, D. Chen, Y. Liu, W. Feng, Evaluation of subsurface defects in metallic structures using laser ultrasonic technique and genetic algorithm-back propagation neural network, NDT & E International 116 (2020) 102339. https ://doi .org/ 10.1016/j .ndteint.2020.102339

[26] S. Guo, H. Feng, W. Feng, G. Lv, D. Chen, Y. Liu, X. Wu, Automatic Quantification of Subsurface Defects by Analyzing Laser Ultrasonic Signals Using Convolutional Neural Networks and Wavelet Transform, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 68(10) (2021) 3216-3225. https://doi.org/10.1109/TUFFC.2021.3087949 15

[27] I. Virkkunen, T. Koskinen, O. Jessen-Juhler, J. Rinta-aho, Augmented Ultrasonic Data for Machine Learning, Journal of Nondestructive Evaluation 40(1) (2021) 4. https://doi.org/10.1007/sl0921-020-00739-5

[28] T. Koskinen, I. Virkkunen, O. Siljama, O. Jessen-Juhler, The Effect of Different Flaw Data to Machine Learning Powered Ultrasonic Inspection, Journal of Nondestructive Evaluation 40(1) (2021) 24. https://doi.org/10.1007/sl0921-021-00757-x

[29] S. Sambath, P. Nagaraj, N. Selvakumar, Automatic Defect Classification in Ultrasonic NDT Using Artificial Intelligence, Journal of Nondestructive Evaluation 30(1) (2011) 20-28. https://doi.org/10.1007/sl0921-010-0086-0

[30] A. Rai, M. Mitra, Lamb wave based damage detection in metallic plates using multi-headed 1- dimensional convolutional neural network, Smart Materials and Structures 30(3) (2021) 035010. https://doi.org/10.1088/1361-665x/abdd00

[31] X. Yang, S. Chen, S. Jin, W. Chang, Crack Orientation and Depth Estimation in a Low-Pressure Turbine Disc Using a Phased Array Ultrasonic Transducer and an Artificial Neural Network, Sensors 13(9) (2013). https://doi.org/10.3390/sl30912375

[32] D. Medak, L. Posilovic, M. Subasic, M. Budimir, S. Loncaric, Automated Defect Detection From Ultrasonic Images Using Deep Learning, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 68(10) (2021) 3126-3134. https://doi.org/10.1109/TUFFC.2021.3081750

[33] T. Latete, B. Gauthier, P. Belanger, Towards using convolutional neural network to locate, identify and size defects in phased array ultrasonic testing, Ultrasonics 115 (2021) 106436. https ://doi .org/ 10.1016/j .ultras.2021.106436

[34] R.J. Pyle, R.L.T. Bevan, R.R. Hughes, R.K. Rachev, A.A.S. Ali, P.D. Wilcox, Deep Learning for Ultrasonic Crack Characterization in NDE, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 68(5) (2021) 1854-1865. https://doi.org/10.1109/TUFFC.2020.3045847

[35] 6. Martin, M. Lopez, F. Martin, Artificial neural networks for quality control by ultrasonic testing in resistance spot welding, Journal of Materials Processing Technology 183(2) (2007) 226-233. https://doi.Org/10.1016/j.jmatprotec.2006.10.011

[36] N. Amiri, G.H. Farrahi, K.R. Kashyzadeh, M. Chizari, Applications of ultrasonic testing and machine learning methods to predict the static & fatigue behavior of spot-welded joints, Journal of Manufacturing Processes 52 (2020) 26-34. https://doi.org/10.1016/jjmapro.2020.01.047

[37] D. Chen, Y. Zhou, W. Wang, Y. Zhang, Y. Deng, Ultrasonic signal classification and porosity testing for CFRP materials via artificial neural network, Materials Today Communications 30 (2022) 103021. https://doi.Org/10.1016/j.mtcomm.2021.103021

[38] R. Xue, X. Wang, Q. Yang, D. Xu, Y. Sun, J. Zhang, S. Krishnaswamy, Grain size distribution characterization of aluminum with a particle swarm optimization neural network using laser ultrasonics, Applied Acoustics 180 (2021) 108125. https://doi.Org/10.1016/j.apacoust.2021.108125

[39] J. Melville, K.S. Alguri, C. Deemer, J.B. Harley, Structural damage detection using deep learning of ultrasonic guided waves, AIP Conference Proceedings 1949(1) (2018) 230004. https://doi.Org/10.1063/l.5031651

[40] K. Shukla, P.C. Di Leoni, J. Blackshire, D. Sparkman, G.E. Kamiadakis, Physics-Informed Neural Network for Ultrasound Nondestructive Quantification of Surface Breaking Cracks, Journal of Nondestructive Evaluation 39(3) (2020) 61. https://doi.org/10.1007/sl0921-020- 00705-1

[41] H. Song, Y. Yang, Super-resolution visualization of subwavelength defects via deep learning- enhanced ultrasonic beamforming: A proof-of-principle study, NDT & E International 116 (2020) 102344. https://doi.Org/10.1016/j.ndteint.2020.102344 16

[42] B. Orazbayev, R. Fleury, Far-Field Subwavelength Acoustic Imaging by Deep Learning, Physical Review X 10(3) (2020) 031029. https://doi.org/10.1103/PhysRevX.10.031029

[43] M. del Hougne, S. Gigan, P. del Hougne, Deeply Subwavelength Localization with Reverberation-Coded Aperture, Physical Review Letters 127(4) (2021) 043903. https://doi.Org/10.l 103/PhysRevLett. 127.043903

[44] C.-S. Chang, Y.-C. Lee, Ultrasonic Touch Sensing System Based on Lamb Waves and Convolutional Neural Network, Sensors 20(9) (2020). https://doi.org/10.3390/s20092619

[45] S. Sikdar, D. Liu, A. Kundu, Acoustic emission data based deep learning approach for classification and detection of damage-sources in a composite panel, Composites Part B: Engineering 228 (2022) 109450. https://doi.Org/10.1016/j.compositesb.2021.109450

[46] A. Firouzeh, M. Salerno, J. Paik, Stiffness control with shape memory polymer in underactuated robotic origamis, IEEE Transactions on Robotics 33(4) (2017) 765-777. https://doi.org/10.1109/TRO.2017.2692266

List of feature references:

Object of inspection 1 ’

Surface 2

Test object 1

Property altering device 8

Mechanical property altering device 80

Electrodes or heating elements 8a

Variable state material 8b

Thermally-sensitive elements piezoelectric elements electrostrictive elements Capacitive elements Magnetostrictive elements Inductive elements

Electro-ultrasonic device 81

Active receiver component 81a

Active emitter component 81b

Active receiver control unit 81c with amplifier

Active emitter control unit 8 Id with amplifier

Electro-ultrasonic control unit 81e

Control unit(s) 83

Controller of the property altering device

Ultrasound producing device 3

Ultrasound generator 3a

First electro-ultrasonic transducer 31a

Laser ultrasound generator 32a

Laser source beam 11

Laser beam for ultrasound detection 12

Laser reflection beam

Galvanometer-driven scanning mirror 13

Optical or opto-mechanical unit to focus the laser beams 14

Optical unit to combine the paths of the laser beams 15

Ultrasound reader or detector 3b Second electro-ultrasonic transducer 3 lb

Laser ultrasound reader 32b

Coating arrangement 4

Laser absorbing layer 4a Backing layer 4b

Laser transparent layer

Reflective particles 4c

Retroreflective particles

Measurement computing system 5 Program modules 6

Machine learning program module

Databases 7

Test objects reference database