Login| Sign Up| Help| Contact|

Patent Searching and Data


Title:
METHOD AND APPARATUS FOR DETECTING CONDITION OF A BOLT ON A BOLTED STRUCTURE
Document Type and Number:
WIPO Patent Application WO/2019/190405
Kind Code:
A1
Abstract:
A method and an apparatus for detecting condition of a bolt on a bolted structure, the method comprising: inputting an image comprising a bolt for detection; detecting that the bolt is a circular or hexagonal bolt; performing perspective distortion correction of the bolt in the image to eliminate error arising from angle of capture of the image; in case that the bolt is detected to be a circular bolt, performing wirelock detection to detect presence of wirelock between more than one detected bolts in the image; and in case that the bolt is detected to be a hexagonal bolt, performing crimp detection to detect whether each bolt edge of the bolt is crimped by a crimping plate or is missing the crimping plate.

Inventors:
CHENG, Jierong (Institute for Infocomm Research1, Fusionopolis Way,,#21-01 Connexis, Singapore 2, 138632, SG)
XIONG, Wei (Institute for Infocomm Research1, Fusionopolis Way,,#21-01 Connexis, Singapore 2, 138632, SG)
DU, Jia (Institute for Infocomm Research1, Fusionopolis Way,,#21-01 Connexis, Singapore 2, 138632, SG)
CHEN, Wenyu (Institute for Infocomm Research1, Fusionopolis Way,,#21-01 Connexis, Singapore 2, 138632, SG)
Application Number:
SG2019/050177
Publication Date:
October 03, 2019
Filing Date:
March 29, 2019
Export Citation:
Click for automatic bibliography generation   Help
Assignee:
AGENCY FOR SCIENCE, TECHNOLOGY AND RESEARCH (1 Fusionopolis Way, #20-10 Connexis North Tower, Singapore 2, 138632, SG)
International Classes:
G06T7/13; F16B39/02; F16B39/10; F16B39/20; G06K9/46; G06N20/00
Foreign References:
CN106485694A2017-03-08
CN106778740A2017-05-31
Other References:
CHA, Y.-J. ET AL.: "COMPUTER-IMAGE-BASED LOOSENED BOLT DETECTION USING SUPPORT VECTOR MACHINES", 5TH INTERNATIONAL STRUCTURAL SPECIALTY CONFERENCE, CSCE, 4 June 2016 (2016-06-04), pages 1 - 8
PARK, J.-H ET AL.: "Novel bolt-loosening detection technique using image processing for bolt joints in steel bridges", THE 2015 WORLD CONGRESS ON ADVANCES IN STRUCTURAL ENGINEERING AND MECHANICS, ASEM15, 29 August 2015 (2015-08-29), pages 1 - 19, XP055640557
Attorney, Agent or Firm:
CHANG, Jian Ming (80 Raffles Place#16-01 UOB Plaza 1, Singapore 4, 048624, SG)
Download PDF:
Claims:
Claims

1. A method for detecting condition of a boit on a bolted structure, the method comprising:

inputting an image comprising a bolt for detection;

detecting that the bolt is a circular or hexagonal boit;

performing perspective distortion correction of the bolt in the image to eliminate error arising from angle of capture of the image;

in case that the bolt is detected to be a circular bolt, performing wirelock detection to detect presence of wire!ock between more than one detected bolts in the image; and

in case that the bolt is detected to be a hexagonal boit, performing crimp detection to detect whether each bolt edge of the bolt is crimped by a crimping plate or is missing the crimping plate.

2. The method as claimed in claim 1 , wherein the method further comprises:

using pre-trained classifiers of an Artificial Intelligence (A.l.) system for the wirelock detection and/or the crimp detection.

3. The method as claimed in claim 1 , wherein the method further comprises:

detecting whether the boit is a circular bolt by applying circular hough transform on the image.

4. The method as claimed in claim 3, wherein the method further comprises:

when circular bolt is detected, fitting boundary of the bolt to an elliptical outline, parameterised by centre of the boit, orientation of the boit, major and minor axes of the bolt;

determining an affine transform to represent a transform of a circular outline to the fitted elliptical outline;

normalising the affine transform, and cropping around the boundary of the bolt to generate sub images of the bolt; and

comparing the generated sub-images of the bolt to a database of labelled images of bolts to classify status of the bolt to be wirelocked or otherwise.

5. The method as claimed in claim 4, wherein classification of status of the detected circular bolt is based on rotation invariant feature extraction that involves providing training images of circular bolt captured at different angles for Artificial Intelligence (A.!.) training.

6. The method as claimed in claim 1 , wherein the method further comprises:

applying collinearity measure and using a line detector to detect status of wirelock or wirelocks between detected bolts to be straight or otherwise.

7. The method as claimed in claim 1 , wherein the method further comprises:

detecting whether the bolt is a hexagonal bolt by:

applying a line segment detector, which locate lines in the image that are candidates of bolt edges;

filtering the located candidates of bolt edges according to angles between each pair of located lines; and

estimating candidate hexagon or hexagons from the candidates of bolt edges, wherein a perfect hexagon is created by estimating most probable parameters of the bolts centre, rotation angle, and radius,

wherein success in locating lines in the image that are candidates of boit edges, filtering of located candidates of bolt edges according to the angles, and estimation of candidate hexagon or hexagons leads to detection of the hexagonal bolt.

8. The method as claimed in claim 7, wherein the method further comprises:

when hexagonal boit is detected, calculating affine transform between the estimated perfect hexagon and the candidates of bolt edges;

finding line correspondences from the calculated affine transform to detect the bolt in the image; and estimating perspective transformation from the found line correspondences to correct perspective distortion of the detected bolt in the image due to angle of capture of the image;

extracting a number of sub-images of the detected bolt from the image by cropping regions outside of bolt edges of the detected bolt;

normalising the extracted sub-images of the detected bolt;

comparing the extracted sub-images of the detected bolt to a database of labelled images of bolts to classify a status of each bolt edge of the detected bolt to be crimped, uncrimped or is missing the crimping plate; and

combining statuses of the bolt edges of the detected bolt to form a joint pattern to provide a final status of the hexagonal bolt.

9. The method as claimed in claim 2, wherein the method further comprises:

using a line segment based registration algorithm to align images captured when the A.i. system is online to pre-co!lected images to obtain a spatial transformation representative of highest degree of alignment between images captured when the A.I. system is online and the pre-collected images, wherein the pre-collected images refer to images pre-collected for A.I. training in bolt edge and/or straight wirelock estimation and prediction;constructing a 3D model from the pr e-coi!ecied images; estimating a geometric transform between two pre-collected images comprising bolts to detect; and

transferring locations of features of the pre-collected images on the constructed 3D model to an image captured when the A.I. system is online based on the estimated geometric transform; and using the spatial transformation and the estimated geometric transform to determine exact 3D location of features in an image captured when the A.I. system is online to facilitate the wirelock detection and/or the crimp detection.

10. An apparatus for detecting condition of a bolt on a bolted structure, wherein the apparatus comprises:

a processor configured to execute instructions to control the apparatus to:

input an image comprising a bolt for detection;

detect that the bolt is a circular or hexagonal bolt;

perform perspective distortion correction of the bolt in the image to eliminate error arising from angle of capture of the image;

in case that the bolt is detected to be a circular bolt, perform wirelock detection to detect presence of wirelock between more than one detected bolts in the image; and

in case that the bolt is detected to be a hexagonal bolt, perform crimp detection to detect whether each bolt edge of the bolt is crimped by a crimping plate or is missing the crimping plate.

1 1. The apparatus as claimed in claim 10, wherein the apparatus is controllable to:

use pre-trained classifiers of an Artificial Intelligence (A.I.) system for the wirelock detection and/or the crimp detection.

12. The apparatus as claimed in claim 10, wherein the apparatus is controllable to:

detect whether the bolt is a circular bolt by applying circular hough transform on the image.

13. The apparatus as claimed in claim 12, wherein the apparatus is controllable to:

when circular bolt is detected, fit boundary of the bolt to an elliptical outline, para eterised by centre of the bolt, orientation of the bolt, major and minor axes of the bolt;

determine an affine transform to represent the transform of a circular outline to the fitted elliptical outline;

normalise the affine transform, and crop around the boundary of the bolt to generate sub-images of the bolt; and

compare the generated sub-images of the bolt to a database of labelled images of bolts to classify status of the bolt to be wirelocked or otherwise.

14. The apparatus as claimed in claim 13, wherein classification of status of the detected circular bolt is based on rotation invariant feature extraction that involves providing training images of circular bolt captured at different angles for Artificial Intelligence (A.!.) training.

15. The apparatus as claimed in claim 10, wherein the apparatus is controllable to:

apply collinearity measure and use a line detector to detect status of wirelock or wirelocks between detected bolts to be straight or otherwise.

16. The apparatus as claimed in claim 10, wherein the apparatus is controllable to:

detect whether the bolt is a hexagonal bolt by:

applying a line segment detector, which locate lines in the image that are candidates of bolt edges;

filtering the located candidates of bolt edges according to angles between each pair of located lines; and

estimating candidate hexagon or hexagons from the candidates of bolt edges, wherein a perfect hexagon is created by estimating most probable parameters of the bolt's centre, rotation angle, and radius,

wherein success in locating lines in the image that are candidates of bolt edges, filtering of located candidates of bolt edges according to the angles, and/or estimation of candidate hexagon or hexagons leads to detection of the hexagonal bolt.

17. The apparatus as claimed in claim 16, wherein apparatus is controllable to:

when hexagonal bolt is detected, calculate affine transform between the estimated perfect hexagon and the candidates of bolt edges;

find line correspondences from the calculated affine transform to detect the bolt in the image; and estimate perspective transformation from the found line correspondences to correct perspective distortion of the detected bolt in the image due to angle of capture of the image;

extract a number of sub-images of the detected bolt from the image by cropping regions outside of bolt edges of the detected bolt;

normalise the extracted sub-images of the detected bolt;

compare the extracted sub-images of the detected bolt to a database of labelled images of bolts to classify a status of each bolt edge of the detected bolt to be crimped, uncrimped or is missing the crimping plate; and

combine statuses of the bolt edges of the defected bolt to form a joint pattern to provide a final status of the hexagonal bolt.

18. The apparatus as claimed in claim 1 1 , wherein the apparatus is controllable to:

use a line segment based registration algorithm to align images captured when the A.l. system is online to pre-collected images to obtain a spatial transformation representative of highest degree of alignment between images captured when the A.l. system is online and the pre-collected images, wherein the pre-collected images refer to images pre-collected for A.L training in bolt edge and/or straight wirelock estimation and prediction;

construct a 3D model from the pre-collected images;

estimate a geometric transform between two pre-collected images comprising bolts to detect; and transfer locations of features of the pre-collected images on the constructed 3D model to an image captured when the A.! system is online based on the estimated geometric transform; and use the spatial transformation and the estimated geometric transform to determine exact 3D location of features in an image captured when the A.! system is online to facilitate the wirelock detection and/or the crimp detection.

Description:
Method And Apparatus for Detecting Condition of A Bolt on A Bolted Structure

Field

The present invention relates to a method and an apparatus for detecting a condition of a bolt on a bolted structure.

Background

In some industries, there is present a visual inspection procedure performed by human operators to inspect statuses of fasteners used in an equipment, machine or structure. Such inspection is performed to ensure that the fasteners are not damaged, and are tightened and securely fastened at their respective locations. Such visual inspection can be time consuming, tedious, and subject to human attention and knowledge of the equipment, machine or structure.

Summary

According to an example of the present disclosure, there are provided a method and apparatus as claimed in the independent claims. Some optional features are defined in the dependent claims.

Brief Description of the Drawings

Embodiments of the present disclosure will be better understood and readily apparent to one skilled in the art from the following written description, by way of example only and in conjunction with the drawings, which are briefly described as follows.

Fig. 1 illustrates an overall framework of a proposed solution for fastener detection.

Fig. 2 illustrates circular bolt detection in the example of Fig. 1.

Fig. 3 illustrates hexagonal bolt detection in the example of Fig. 1.

Fig. 4A illustrates an idea! hexagonal bolt, and six image patches to be cropped for feature extraction.

Fig. 4B illustrates a projective transform of the image in Fig. 4A.

Fig. 5A illustrates a joint pattern of six edges statuses of an uncrimped bolt.

Fig. 5B illustrates a joint pattern of six edges statuses of a crimped bolt.

Fig. 6 illustrates a result of detection of 6 circular bolts.

Fig. 7 illustrates a result of detection of straight wirelock for securing 2 circular bolts.

Fig. 8 illustrates a result of supervised wirelock detection.

Fig. 9A illustrates hexagonal bolt detection with candidate edges identified as bolt edges.

Fig. 9B illustrates registration of an ideal hexagon to an actual hexagonal bolt using affine transform.

Fig.10 illustrates supervised uncrimped bolt detection and status of each bolt edge tested is marked on each of them.

Fig. 1 1 is a block diagram illustrating system architecture of a system or an apparatus according to an example of the present disclosure. Detailed Description

An example of the present disclosure is related to industries pertaining to automotive, construction, and heavy machinery. For example, with reference to the automotive/heavy machinery industry, in a routine engine external check, typically there are three types of bolt-related faults to be detected for an exterior of the engine. They are 1 ) missing bolts, 2) un-crimped bolts, and 3) bolts with missing locking wire or wire!ock. The visual inspection of such faults is typically performed by human operators, which can be time consuming, tedious, and subject to human attention and knowledge of the engine. It may, for instance, take one day for a human inspector to examine a fairly large engine. The present example proposes an automated image analysis method to be performed on digital images captured from the engine’s exterior. The advantages of the method include improvement to inspection efficiency, consistency and accuracy of bolt-related fault detection of the engine’s exterior.

The bolts on an engine surface can vary largely in size and shape (e.g. circular, hexagonal, and wavy). Its appearance on images depends on the viewing angle, capturing distance, surface material and lighting conditions. Existing bolt detection methods only apply for bolts of known shape and on a homogeneous background. These methods may face difficulties to detect bolts fixed on an engine surface with complex structure and hence forming heterogeneous backgrounds.

A system is proposed for the example of the present disclosure for analysing digital engine images captured by, for instance, mobile devices or moving cameras in order to detect the aforementioned three types of bolt-related faults i.e. 1 ) missing bolts, 2) un-crimped bolts, and 3) bolts with missing locking wire or wirelock. The system may include one or more components in the form of software, hardware, and/or firmware to operate. Although a system is proposed for the present example, it should be appreciated that in another example, an apparatus (e.g. a central control server or computer, or a mobile device) including a processor or processing unit that executes instructions stored in a memory may be proposed to control the apparatus to perform all the functions of the system. The system may comprise such apparatus. Such apparatus may be the system itself. Mobile device mentioned in the present disclosure includes, for example, smart phone, tablet device, portable personal computer, and the like.

There is proposed a method for the system including offline Artificial Intelligence (A.l.) training and online A.l. assisted detection to inspect bolt faults from 2 Dimensional (2D) images. The offline A.l. training stage aims to build a database including prior knowledge such as 3 Dimensional (3D) engine models, a master template of bolt types and positions, pre-coilected images aligned with a model as a bridge for 2D to 3D mapping, and trained classifiers to distinguish between crimped and uncrimped or wirelocked and unwirelocked bolts. In an online detection stage, bolt fault detection can be performed using an A.l. mode! having pre-trained classifiers. The A.l. portion of the system may comprise of a circle and line detector, an image feature extractor, and a trained classifier. For example, the A.L mode! can be a pre-trained classifier, which takes a feature vector extracted from an image patch as input and classifies it into a bolt status (e.g. wirelocked or not).

For example, Fig. 1 illustrates an overall framework of the proposed solution of the system 100 as mentioned above. The system 100 takes in a current 2D image 102 captured by an imaging device such as a camera as input. The current 2D image 102 captures a surface of an engine that comprises one or more bolts. This camera may one integral in a mobile device such as a smartphone, or a robotic moving camera that is configured to move to pre-defined positions to capture images. From the current 2D image 102, shape of the one or more bolts are identified. For instance, they are subjected to circular bolt detection 104 and/or hexagonal bolt detection 106. Upon detection that they are circular or hexagonal bolt, perspective distortion correction 108 is performed on the current 2D image 102 to enable supervised wirelock detection 1 10 to be performed in the case of detection of circular bolt and/or to enable supervised crimp detection 1 12 to be performed in the case of detection of hexagonal bolt. Perspective distortion correction 108 is performed to eliminate error arising from angle of capture of the image by the camera. In the present example, wirelock should be present for securing any circular bolt. During hexagonal bolt defection, line segment matching 1 14 is performed for each hexagonal bolt to determine parameters of the bolt such as bolt’s centre, rotation angle, radius etc. line segment matching 1 14 may also be applied to detect straightness of wirelock during supervised wirelock detection 1 10. The current 2D image 102 may be subjected to geometric transform 1 16 to facilitate line segment matching 1 14. The system 100 comprises an offline system 124 that can be switched online in actual operation. The offline system 124 comprises a database 1 18 for storing one or more of the current 2D image 102. The database 1 18 can also store prior knowledge such as 3 Dimensional (3D) engine models, a master template of bolt types and positions, pre-collected images aligned with a model as a bridge for 2D to 3D mapping, and trained classifiers to distinguish between crimped and uncrimped or wirelocked and unwirelocked bolts. The offline system 124 can be configured to perform 2D to 3D correspondence (e.g. determine the necessary coordinate systems and parameters) and to construct a 3D model of the detected bolt or bolts in the current 2D image 102. In the present example, the offline system 124 is an Artificial Intelligence (A.l.) system based on neural network, and uses data in the database 1 18 to learn and to make predictions. When offline, the offline system 124 can be operated to perform A.! training. After training, the offline system 124 comprises pre-trained classifiers 122 derived from A.! training. When the offline system 124 is switched online, the pre-trained classifiers 122 are utilized during actual supervised wire!ock detection 1 10 and supervised crimp detection 1 12 to facilitate accurate predictions for wirelock detection and crimp detection.

Fig. 2 illustrates circular bolt detection 104, perspective distortion correction 108 and supervised wirelock detection 1 10 of Fig. 1.

In the example of Fig. 2, there is provided a method 200 for detecting a condition of a bolt from an image 202 of the bolt disposed on a bolted structure, wherein the bolt is a circular bolt with wirelock. The method comprises the following steps.

At a step 204, the method 200 locates or detects the bolt by applying circular Hough transform on the image 202 of the bolt. If at step 204, a circle is not detected, it is determined that a circular bolt is not present. Otherwise, it is determined that the circular bolt is present.

At a step 206, the method 200 fits boundary of the bolt to an elliptical outline, parameterised by centre of the bolt, orientation of the bolt, major and minor axes of the bolt.

At a step 208, the method 200 determines an affine transform to represent the transform of a circular outline to the fitted elliptical outline.

At a step 210, the method 200 normalises the affine transform, and crops the boundary of the bolt to generate sub-images of the bolt. After normalizing the affine transform, the boundary of the bolt becomes approximately a circle. The sub-images are generated by cropping a square window around the circular boundary of the bolt.

At a step 212, the method 200 compares the generated sub-images of the bolt to a database (e.g. the database 1 18 of Fig. 1 ) of labelled images of bolts to classify status of the bolt to be wirelocked or otherwise, wherein the classification of the status of the detected circular bolt is based on rotation invariant feature extraction. Such rotation invariant feature extraction method would be described in more detail later.

The method 200 further comprises a step 214 of applying collinearity measure and using a line detector to detect status of wirelock or wirelocks between detected bolts to be straight or otherwise. The collinearity measure is computed from an angle between two line segments detected in the image 202 and perpendicular distances from midpoint of each of the two detected line segments to the other one of the two detected line segments. When the computed collinearity measure is above a predefined threshold, the two detected line segments are considered coiiinear and therefore they can be merged as a single line to avoid duplicate line detection and to avoid detection of incomplete lines. One example of a line detection method for detecting line segments is found in Rafael Grompone von Gioi, J ' er ' emie Jakubowicz, J.-M. Morel, and Gregory Randall,“LSD: a line segment detector,” Image Processing Online, Mar 2012. Steps 204 to 212 can be applied to more than one bolt to extract features and perform classification for each detected circular bolt. Fig. 3 illustrates hexagonal bolt detection 106, perspective distortion correction 108, supervised crimp detection and line segment matching 1 14 of Fig. 1.

In the example of Fig. 3, there is provided a method 300 for detecting a condition of a bolt from an image 302 of the bolt disposed on a bolted structure, wherein the bolt is hexagonal. The method 300 comprises the following steps.

At a step 304 for line segment detection of a hexagonal bolt, the method 300 locates or detects the hexagonal bolt by applying a line segment detector, which locate lines in the image 302 that are candidates of bolt edges “line segment detector” refers to an image processing too! that is used to locate lines in a given image.

At a step 306, the method prunes or filters the located candidates of bolt edges according to angles between each pair of located lines. For example, it is known that a hexagon is made up of 6 lines that form 6 corners, and each pair of lines forms an angle of 120 degrees relative to each other at each corner. Therefore, the pair of lines that forms an angle closest to 120 degrees relative to each other is likely to be the bolt edges.

At a step 308, the method 300 estimates candidate hexagon or hexagons from the candidates of bolt edges, wherein a perfect or ideal hexagon is created by estimating most probable parameters of the bolt's centre, rotation angle, and radius.

If in steps 304 to 308, there is success in locating lines in the image that are candidates of bolt edges, filtering of located candidates of bolt edges according to the angles, and estimation of candidate hexagon or hexagons, such success leads to detection of the hexagonal bolt. Otherwise, it is determined that the hexagonal bolt is not detected.

At a step 310, the method 300 uses two dimensional (2D) line segment registration to calculate affine transform between the estimated perfect or idea! hexagon and the candidates of bolt edges. In one example, the 2D line segment registration may use intersection points of line segments as key points and finds an optima! affine transform between two sets of such key points. The line segments refer to the candidates of bolt edges and lines of the ideal hexagon. Each set of key points consists of a set of 2D points represented by their x and y coordinates on the image 302. The two sets of key points to be used to find the optima! affine transform include one set of key points on the ideal hexagon and the other set of key points on the candidates of bolt edges. The optimal affine transform can be estimated to be the same affine transform obtained between two sets of line segments. Each set of line segments consists of six line segments represented by their starting and ending points. The key points of the line segments are found by extending the line segments and computing their intersection points. For the ideal hexagon, one can obtain 12 key points in total i.e. 6 points from the 6 corners and the rest of the 6 points from the intersection of every two edges which are neither in parallel nor neighboring. In the same manner, 12 key points can be generated from the candidates of bolt edges. The two sets of line segments to be used to find the optimal affine transform include one set of line segments of the idea! hexagon and the other set of line segments of the candidates of bolt edges.

At a step 312, the method 300 finds line correspondences from the calculated affine transform to detect the bolt in the image. Finding line correspondences refer to detection of line correspondence between each of six candidate bolt edges and each of the six sides of the idea! hexagon respectively. Once such corresponding lines are found, the bolt is detected.

At a step 314, the method 300 estimates perspective transformation i.e. final camera pose estimation using the found line correspondences to correct perspective distortion of the detected bolt in the image 302 due to angle of capture of the image 302. At a step 316, a number of sub-images of the detected bolt is extracted from the image 302 by cropping regions 402 outside of bolt edges of the detected bolt 400, and the extracted sub-images are subsequently normalised into squares of pre-defined size.

At a step 318, the method 300 compares the extracted sub-images of the detected bolt to a database of labelled images of bolts to classify a status of each bolt edge of the detected bolt to be crimped, uncrimped or missing a crimping plate, for example, using A.i. that had been subjected to supervised learning, and combines statuses of the (six) bolt edges of the detected bolt to form a joint pattern to provide a final status of the hexagonal bolt. The final status of the hexagonal bolt can include (1 ) partially crimped and not fully secured, (2) fully crimped and secured, (3) not crimped and not secured, and the like. For example, a joint pattern can be a combined clockwise pattern of bolt edge statuses starting from an edge labelled as“503” in Fig. 5B that is crimped, missing, crimped, crimped, missing, and crimped, in such order respectively. Such a joint pattern is evident of a fully crimped and secured condition.

Based on the above method 300, the classification of the bolt edge is based on a scale invariant feature extraction, wherein the detection of the bolt edge is invariant to the bolt size. Such scale invariant feature is contributed by the normalization processing step described in step 316.

More details that can be present in examples of the present disclosure are described as follows.

Image processing methods can be proposed to detect the number of bolts on the engine, uncrimped bolts, and bolts with missing locking wire, from the acquired images of the system 100 of Fig. 1 . Such image processing methods are applicable to, for instance, the methods 200 and 300 of Fig. 2 and 3 respectively. The system 100 can be controlled via software and can be able to detect bolt-related faults based on reference images which provide prior knowledge.

With regard to circular bolts, circular bolts in a target image may be approximately located by using Circular Hough Transform. The boundary of the bolt is then fitted to an ellipse parametrized by its centre, orientation, major and minor axes. An affine transform can be calculated to represent the transform from an ideal circle to the fitted ellipse. After bolt detection, image patches are automatically cropped around the boundary of bolts for Histogram of Gradient (HoG) feature extraction. A HoG descriptor counts occurrences of gradient orientation in an image patch and uses a histogram of directions of gradients as features. For circular bolts, sub-images are cropped from the target image around the centre of the bolt. These sub-images are normalized using the computed affine (ellipse to circle) transform before feature extraction. A Support Vector Machine (SVM) classifier is trained from manually labelled ground truth and thereby able to distinguish circular bolts status (wire!ocked/unwirelocked). The SVM classifier is a discriminative classifier formally defined by a separating hyperplane. In other words, given labelled training data, the algorithm of an SVM model outputs an optima! hyperplane, which categorizes the training data according to their labels. An SVM model is a representation of examples (training data) as points in space, mapped so that the examples (training data) of separate categories are divided by a clear gap that is as wide as possible. New examples (test data) are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall in. Straight wireiocks between bolts are detected by applying collinearity measure combined with a line detector on the target image. Such collinearity measure combined with line detector has been described in step 214 of Fig. 2.

Furthermore, a rotation-scale invariant feature extraction method for machine learning based bolt status classification can be proposed. Histogram of Gradient (HoG) is not a rotation-scale invariant descriptor. However, by accurately detecting bolt boundary and normalizing the captured images of a bolt for detection according to a pre-defined bolt size before feature extraction, the trained detector of the system (e.g. the system 100 of Fig. 1 ) becomes invariant to the bolt size. Due to different viewing angles and distances, the shape of a bolt in images can be modelled as ellipses of different sizes and orientations. By fitting a bolt boundary to an ellipse, and normalizing the fitted bolt boundary from an ellipse to a circle of pre-defined radius, features extracted from training images are detectable through A.I. even when changes are made to the image’s scale (i.e. image size), as long as the shape of bolt In the tested Images are normalized to the same circular shape. Moreover, rotating training images of circular bolt captured at different angles for A.I. training can help to achieve rotation invariance. With regard to hexagonal bolts, to locate hexagonal bolts in a target image, first a !ine segment detector may be used to prune candidates for bolt edges according to angles between each pair of detected lines. Then, from all the possible hexagons formed by the candidate edges, a perfect hexagon is created by estimating the most probable parameters of its centre, rotation angle, and radius. To correct the perspective distortion on the target image caused by picturing angle of a camera capturing the target image, the affine transform between the estimated (perfect) hexagon and the candidate bolt edges is calculated by using a robust 2D line segment registration method, followed by final camera pose estimation from line correspondences.

Perspective distortion correction can also be introduced, wherein a robust hexagon fitting algorithm is used for hexagonal bolt detection from captured images. Projective transform between a perfect or ideal hexagon and bolt edges can be estimated. Therefore, perspective distortion on the image caused by picturing angle of a camera or image capturing device used to capture the bolt for detection can be corrected by cropping the transformed image patches (also called “sub-images”) before feature extraction. More details on perspective distortion correction are described with reference to Figs. 4A and 4B below.

In hexagonal bolt status classification, instead of classifying it directly from the features of the whole bolt image, each edge of the bolt can be classified independently. Whether the bolt is crimped or not is determined by the joint pattern of edge status. This is much more robust than a single round classification for predicting true bolt status.

In an example, existing key point detectors such as scale invariant feature transform (SIFT) and speed up robust feature (SURF) are not applicable for texture-less engine images. The robust 2D line segment registration method described earlier can be used to align the images captured for bolt detection in an online stage of, for instance, the offline system 124 of Fig. 1 , to pre-co!!ected images (for A.! training) or a master template (i.e. a pre-defined reference image set), so that missing bolts and missing wireiocks can be detected.

With reference to Figs. 4A and 4B, there is shown an image of a hexagonal bolt 400 with six bolt edges 404. Six sub-images (or image patches) 402 are cropped from the outside of the bolt edges 404. Fig. 4B is a projective transform 401 of Fig. 4A. These cropped sub-images are normalized using a computed projective transform before feature extraction at the step of 318 of Fig. 3. Specifically, during feature extraction and classification, a Histogram of Gradient (HoG) feature is computed from each of the cropped sub-images. A classifier is trained from manually labelled ground truth and thereby able to distinguish bolts edge status (crimped/uncrimped/missing). In the present example, the classification accuracy of the training data is above 95%. The final status of the hexagonal bolt is determined by a joint pattern of its edges as shown in Fig. 5A and 5B.

Figs. 5A and 5B illustrate joint patterns of six edge statuses of an uncrimped hexagonal bolt 500 in Fig. 5A and a crimped hexagonal bolt 501 in Fig. 5B. Fig. 5A shows a crimping plate 504 that can be bent to crimp 4 edges of the bolt 500 to secure it. in the case of Fig. 5A, all 4 edges to be crimped are uncrimped and thus 4 uncrimped edges 502 are detected. The remaining 2 edges of the bolt 500 are regarded as 2 edges 506 with crimping plate 504 missing. Similarly, Fig. 5B shows a crimping plate 510 that can be bent to crimp 4 edges of the bolt 501 to secure it. In the case of Fig. 5B, all 4 edges to be crimped are crimped and thus 4 crimped edges 508 are detected. The remaining 2 edges of the bolt 501 are regarded as 2 edges 512 with crimping plate 510 missing.

In an example of a bolted structure, such as on a panel of an engine (e.g. of an automotive), the surface is metallic and smooth, which is difficult for key point detection. Hence, as taught earlier in step 310 of Fig. 3, the edges of hexagonal bolts and their intersection points can be used as key points. A robust line segment based registration algorithm can be used to align the images captured in an A.l. online stage (i.e. when the A.L system is in actual operation) to pre-collected images that are for reference. Features extracted from the pre-collected images for reference can be used to train an A.l. model to classify crimp status and/or wirelock status. For example, the line segment based registration can involve the following steps.

1) A step to randomly choose three pairs of line segments of hexagonal bolts from an image captured in the A.l. online stage and choose three pairs of line segments of hexagonal bolts from one of the pre-collected images.

2) A step to find three intersection points for every three pairs of chosen line segments. Each intersection point is a pair of key points i.e. two dimensional x and y coordinates.

3) A step to infer spatial transformation between the three pairs of key points of the three pairs of line segments of a hexagonal bolt from the image captured in the A.l. online stage and the three pairs of key points of a hexagonal bolt from the pre-collected image.

4) A step to calculate a score to measure how the line segments are aligned between the two images considered by the spatial transformation. For example, higher score may indicate better alignment between the image captured in the A.l. online stage and the pre-collected image. Each score can indicate degree of alignment.

5) The process of steps 1 ) to 4) above can be repeated for selected line segments of several images captured in the A.L online stage and several pre-collected image until no higher score can be achieved. The spatial transformation with highest score {i.e. highest degree of alignment) would be a useful spatial transformation result for consideration.

Furthermore, a 3D model can be constructed from a set of the pre-collected images (e.g. images constructed by the offline system 124 of Fig. 1 ) in which the bolted structure is in an ideal condition. Then, by estimating a geometric transform between a pre-collected image and an online image (e.g. a target image considered when the offline system 124 of Fig. 1 is online), locations of features of the pre collected images on the constructed 3D model can be transferred to the online image based on the estimated geometric transform.

The spatial transformation result of the line segment based registration algorithm is useful for missing bolt detection and un-wire!ocked bolt detection (which is determinable when a reference model is provided), as well as for accurate localization of faulty bolts in the constructed 3D model. Specifically, the spatial transformation result of the line segment based registration provides a reference for where to look for certain features. For example, where exactly is a 3D location of a wirelock in an online image. Hence, in the manner as described, wirelock detection and crimp detection can be performed using the spatial transformation result and the estimated geometric transform.

Fig. 6 illustrates detection of 6 circular bolts 602 on a bolted structure 600 using the system of Fig. 1 .

Fig. 7 illustrates detection of a straight wirelock 702 for securing two detected circular bolts 704 and 706 on a bolted structure 700 using the system of Fig. 1. 3 other circular bolts 708, 710 and 712 are actually detected in Fig. 7 as well. Markings such as dashed lines 714 shown in Fig. 7 refer to straight lines between every detected pair of bolts in Fig. 7 and each dashed line indicates no wire between the detected pair of bolts. A solid line 716 indicates that there is a wire between the detected circular bolts 704 and 706 used to lock and secure them. The system 100 of Fig. 1 can be configured to display such dashed lines 714 and solid line 716 on a display such as Liquid Crystal Display (LCD) monitor, Light Emitting Diode (LED) monitor, and the like. Such markings shown in the display is useful indication of wirelock detection result to a user of the system of Fig. 1.

Fig.8 further illustrates detection of two other circular bolts 802 and 804 by the system of Fig. 1 in addition to the 6 circular bolts 602 detected in the bolted structure 600 of Fig. 6. Specifically, Fig. 8 illustrates result of supervised wirelock detection 1 10 of Fig. 1. Markings such as circles or rings 806 circulating or highlighting each of the 6 circular bolts 602 can be displayed on a display (e.g. Liquid Crystal Display (LCD) monitor, Light Emitting Diode (LED) monitor, and the like) of the system of Fig. 1 . The circular bolts 802 and 804 are circulated or highlighted by circles or rings 808 in the similar manner as well and displayed accordingly. The colour scheme (e.g. Red) of the circles or rings 806 of the 6 circular bolts 602 and the colour scheme (e.g. Blue) of the circles or rings 808 of the circular bolts 802 and 804 are configured to be different because circular bolts 802 and 804 are not wirelocked or unwirelocked but the 6 circular bolts 602 are wired!ocked. Such markings shown in the display is useful indication of wirelock detection result to a user of the system of Fig. 1. As mentioned earlier, existing bolt detection methods only apply for bolts of known shape and on a homogeneous background. These methods may face difficulties to detect bolts fixed on an engine surface with complex structure and hence forming heterogeneous backgrounds. Fig. 8 illustrates an advantage of the capability to apply the system of Fig. 1 in a heterogeneous background comprising a surface with complex structure.

Results of hexagonal bolt and crimp detection are illustrated in Figs. 9A, 9B and 10. Fig. 9A shows detection of a hexagonal bolt 900. There are present straight line markings 902 indicative of detected candidates for bolt edges of the hexagonal bolt 900. A line marking 904 residing between centre of the hexagonal bolt 900 and one of the 6 pointed corners of the hexagonal bolt 900 is detected candidate indicating radius of the hexagonal bolt 900. Fig. 9B shows the same hexagonal bolt 900 of Fig. 9A with registration of an ideal hexagon 906 (dashed line) to an actual hexagonal bolt outline marking 908 produced using affine transform. The system 100 of Fig. 1 can be configured to display such line markings 902, 904, 906 and 908 on a display such as Liquid Crystal Display (LCD) monitor, Light Emitting Diode (LED) monitor, and the like. Such markings shown in the display is useful indication of hexagonal bolt defection result to a user of the system of Fig. 1.

Fig. 10 illustrates supervised crimp bolt detection 1 12 of the system 100 of Fig. 1. The status (i.e.‘crimp’, ‘uncrimped’ or‘missing’) of each tested bolt edge is marked out in Fig. 10. In the present example, the marking of statuses of ail bolt edges of a detected bolt is referred to as a joint pattern and a final status describing the bolt condition can be given based on the joint pattern. For example, if the joint pattern is such that there are 2 bolt edges with missing crimping plate and 1 out of 4 bolt edges with crimping plate is not crimped, a final status indicating that maintenance is required may be notified by the system 100 to alert a user to take maintenance action to crimp the uncrimped bolt edge. Specifically, Fig. 10 shows 2 bolts 1000 and 1001 . 4 uncrimped bolt edges 1002 of bolt 1000 have been marked out. 2 bolt edges 1004 with missing crimping plate 1006 are also marked out. Each bolt edge of bolt 1001 is also similarly marked with‘crimp’,‘uncrimped’ or‘missing’ statuses. Bolt 1001 is only partially shown and shall not be further elaborated. The system 100 of Fig. 1 can be configured to display such status indications i.e. ‘crimp’,‘uncrimped’ or‘missing’ on a display such as Liquid Crystal Display (LCD) monitor, Light Emitting Diode (LED) monitor, and the like. Such markings shown in the display is useful indication of supervised crimp bolt detection result to a user of the system of Fig. 1.

The architecture of a system or an apparatus proposed in an example of the present disclosure may be an apparatus 1 102 in Figure 1 1 and comprises a number of individual components including, but not limited to, processing unit 1 1 16 (or processor), a memory 1 1 18 (e.g. a volatile memory such as a Random Access Memory (RAM) for the loading of executable instructions 1 120, the executable instructions defining the functionality the apparatus 1 102 carries out under control of the processing unit 1 1 16. The apparatus 1 102 also comprises a network module 1 125 allowing the apparatus 1 102 to communicate over the communications network 1 108 (for example the internet, Bluetooth network). User interface 1 124 is provided for user interaction and may comprise, for example, conventional computing peripheral devices such as display monitors, computer keyboards and the like. The apparatus 1 102 may also comprise a database 1 126 to store data contained in the database 1 18 of Fig. 1 . The database 1 126 may not be local to the apparatus 1 102. The database 1 126 may be a cloud database.

The processing unit 1 1 16 is connected to input/output devices such as a computer mouse, keyboard/keypad, a display, headphones or microphones, a video camera and the like (not illustrated in Figure) via Input/Output (I/O) interfaces 1 122. The components of the processing unit 1 1 16 typically communicate via an interconnected bus (not illustrated in Figure 1 1 ) and in a manner known to the person skilled in the relevant art.

The processing unit 1 1 16 may be connected to the network 1 108, for instance, the Internet, via a suitable transceiver device (i.e. a network interface) or a suitable wireless transceiver, to enable access to e.g. the Internet or other network systems such as a wired Local Area Network (LAN) or Wide Area Network (WAN). The processing unit 1 1 16 of the apparatus 1 102 may also be connected to one or more external wireless communication enabled device 1 104 through the respective communication links 1 1 10 and 1 1 12 via the suitable wireless transceiver device e.g. a WiFi transceiver, Bluetooth module, Mobile telecommunication transceiver suitable for Global System for Mobile Communication (GSM), 3G, 3.5G, 4G telecommunication systems, or the like.

In another example, a system or an apparatus proposed in an example of the present disclosure may have a system architecture of a device 1 104 in Figure 1 1. The device 1 104 may comprise a number of individual components including, but not limited to, microprocessor 1 128 (or processor), a memory 1 130 (e.g. a volatile memory such as a RAM) for the loading of executable instructions 1 132, the executable instructions defining the functionality the device 1 104 carries out under control of the microprocessor 1 128. The device 1 104 also comprises a network module (not illustrated in Figure) allowing the device 1 104 to communicate over the communications network 1 108. User interface 1 136 is provided for user interaction and control that may be in the form of a touch panel display and presence of a keypad as is prevalent in many smart phone and other handheld devices. The device 1 104 may also comprise a database (not illustrated in Figure), which may not be local to the device 1 104 but a cloud database. The database may contain the same data as that of the database 1 126. The device 1 104 may include a number of other Input/Output (I/O) interfaces 1 134 as well but they may be for connection with headphones or microphones or projectors, Subscriber identity module (SIM) card, flash memory card, USB based device, and the like, which are more for mobile device usage.

The software and one or more computer programs stored may include, for example, applications that may include one or more applications for e.g. internet accessibility, operating the device 1 104 and device 1 106 (i.e. operating system), network security, file accessibility, database management, which are applications typically equipped on a desktop or portable (mobile) device. The software and one or more computer programs may be supplied to a user of the device 1 104 and/or the device 1 106 encoded on a data storage medium such as a CD-ROM, on a flash memory carrier or a Hard Disk Drive, and are to be read using a corresponding data storage medium drive for instance, a data storage device (not illustrated in Figure 1 1 ). Such application programs may also be downloaded from the network 1 108. The application programs are read and controlled in its execution by the processing unit 1 1 16 or microprocessor 1 128. Intermediate storage of program data may be accomplished using RAM 1 120 or 1 130.

Furthermore, one or more of the steps of the computer programs or software may be performed in parallel rather than sequentially. One or more of the computer programs may be stored on any machine or computer readable medium that may be non-transitory in nature. The computer readable medium may include storage devices such as magnetic or optical disks, memory chips, or other storage devices suitable for interfacing with a general purpose computer or mobile device. The machine or computer readable medium may also include a hard-wired medium such as exemplified in the Internet system, or wireless medium such as exemplified in the Wireless LAN (WLAN) system. The computer program when loaded and executed on such a general-purpose computer effectively results in a system or an apparatus that implements the steps of the computing methods in examples herein described.

Examples of the present disclosure may have the following features.

A method for detecting condition of a bolt on a bolted structure, the method comprising: inputting an image comprising a bolt for detection; detecting that the bolt is a circular or hexagonal bolt; performing perspective distortion correction of the bolt in the image to eliminate error arising from angle of capture of the image; in case that the bolt is detected to be a circular bolt, performing wire!ock detection to detect presence of wirelock between more than one detected bolts in the image; and in case that the bolt is detected to be a hexagonal bolt, performing crimp detection to detect whether each bolt edge of the bolt is crimped by a crimping plate or is missing the crimping plate.

The method may comprise using pre-trained classifiers of an Artificial Intelligence (A.!.) system for the wirelock detection and/or the crimp detection.

The method may further comprise detecting whether the bolt is a circular bolt by applying circular hough transform on the image. The method may further comprise: when circular bolt is detected, fitting boundary of the bolt to an elliptical outline, parameterised by centre of the bolt, orientation of the bolt, major and minor axes of the bolt; determining an affine transform to represent a transform of a circular outline to the fitted elliptical outline; normalising the affine transform, and cropping around the boundary of the bolt to generate sub images of the bolt; and comparing the generated sub-images of the bolt to a database of labelled images of bolts to classify status of the bolt to be wire!ocked or otherwise.

Classification of status of the detected circular bolt may be based on rotation invariant feature extraction that involves providing training images of circular bolt captured at different angles for Artificial Intelligence (A.l.) training.

The method may further comprise applying co!linearity measure and using a line detector to detect status of wirelock or wirelocks between detected bolts to be straight or otherwise.

The method may further comprise: detecting whether the bolt is a hexagonal bolt by: applying a line segment detector, which locate lines in the image that are candidates of bolt edges; filtering the located candidates of bolt edges according to angles between each pair of located lines; and estimating candidate hexagon or hexagons from the candidates of bolt edges, wherein a perfect hexagon is created by estimating most probable parameters of the bolt's centre, rotation angle, and radius, wherein success in locating lines in the image that are candidates of bolt edges, filtering of located candidates of bolt edges according to the angles, and estimation of candidate hexagon or hexagons leads to detection of the hexagonal bolt.

The method may further comprise: when hexagonal bolt is detected, calculating affine transform between the estimated perfect hexagon and the candidates of bolt edges; finding line correspondences from the calculated affine transform to detect the bolt in the image; and estimating perspective transformation from the found line correspondences to correct perspective distortion of the detected bolt in the image due to angle of capture of the image; extracting a number of sub-images of the detected bolt from the image by cropping regions outside of bolt edges of the detected bolt; normalising the extracted sub-images of the detected bo!t; comparing the extracted sub-images of the detected bolt to a database of labelled images of bolts to classify a status of each bolt edge of the detected bolt to be crimped, uncrimped or is missing the crimping plate; and combining statuses of the bolt edges of the detected bolt to form a joint pattern to provide a final status of the hexagonal bolt.

The method may further comprise: using a line segment based registration algorithm to align images captured when the A.l. system is online to pre-col!ected images to obtain a spatial transformation representative of highest degree of alignment between images captured when the A.L system is online and the pre-collected images, wherein the pre-collected images refer to images pre-coliected for A.l. training in bolt edge and/or straight wirelock estimation and prediction ;constructing a 3D model from the pr e-coliected images; estimating a geometric transform between two pre-collected images comprising bolts to detect; and transferring locations of features of the pre-collected images on the constructed 3D model to an image captured when the A.l. system is online based on the estimated geometric transform; and using the spatial transformation and the estimated geometric transform to determine exact 3D location of features in an image captured when the A.L system is online to faci!itate wire!ock detection and/or crimp defection.

An apparatus for detecting condition of a bolt on a bolted structure, wherein the apparatus comprises: a processor configured to execute instructions to control the apparatus to: input an image comprising a bolt for detection; detect that the bolt is a circular or hexagonal bolt; perform perspective distortion correction of the bolt in the image to eliminate error arising from angle of capture of the image; in case that the bolt is detected to be a circular bo!t, perform wire!ock defection to detect presence of wire!ock between more than one detected bolts in the image; and in case that the bolt is detected to be a hexagonal bolt, perform crimp detection to detect whether each bolt edge of the bolt is crimped by a crimping plate or is missing the crimping plate. The apparatus may be controllable to: use pre-trained classifiers of an Artificial Intelligence (A.I.) system for the wirelock detection and/or the crimp detection.

The apparatus may be controllable to: detect whether the bolt is a circular bolt by applying circular hough transform on the image.

The apparatus may be controllable to: when circular bolt is detected, fit boundary of the bolt to an elliptical outline, parameterised by centre of the bolt, orientation of the bolt, major and minor axes of the bolt; determine an affine transform to represent the transform of a circular outline to the fitted elliptical outline; normalise the affine transform, and trace around the boundary of the bolt to generate sub-images of the bolt; and compare the generated sub-images of the bolt to a database of labelled images of bolts to classify status of the bolt to be wirelocked or otherwise.

Classification of status of the detected circular bolt may be based on rotation invariant feature extraction that involves providing training images of circular bolt captured at different angles for Artificial Intelligence (A.i.) training.

The apparatus may be controllable to: apply co!linearity measure and use a line detector to detect status of wirelock or wirelocks between detected bolts to be straight or otherwise.

The apparatus may be controllable to: detect whether the bolt is a hexagonal bolt by: applying a line segment detector, which locate lines in the image that are candidates of bolt edges; filtering the located candidates of bolt edges according to angles between each pair of located lines; and estimating candidate hexagon or hexagons from the candidates of bolt edges, wherein a perfect hexagon is created by estimating most probable parameters of the bolt's centre, rotation angle, and radius, wherein success in locating lines in the image that are candidates of bolt edges, filtering of located candidates of bolt edges according to the angles, and/or estimation of candidate hexagon or hexagons leads to detection of the hexagonal bolt.

The apparatus may be controllable to: when hexagonal bolt is detected, calculate affine transform between the estimated perfect hexagon and the candidates of bolt edges; find line correspondences from the calculated affine transform to detect the bolt in the image; and estimate perspective transformation from the found line correspondences to correct perspective distortion of the detected bolt in the image due to angle of capture of the image; extract a number of sub-images of the detected bolt from the image by cropping regions outside of bolt edges of the detected bolt; normalise the extracted sub-images of the detected bolt; compare the extracted sub-images of the detected bolt to a database of labelled images of bolts to classify a status of each bolt edge of the detected bolt to be crimped, uncrimped or is missing the crimping plate; and combine statuses of the bolt edges of the detected bolt to form a joint pattern to provide a final status of the hexagonal bolt.

The apparatus may be controllable to: use a line segment based registration algorithm to align images captured when the A.I. system is online to pre-collected images to obtain a spatial transformation representative of highest degree of alignment between images captured when the A.I. system is online and the pre-collected images, wherein the pre-collected images refer to images pre-collected for A.! training in bolt edge and/or straight wirelock estimation and prediction; construct a 3D mode! from the pre collected images; estimate a geometric transform between two pre-co!lected images comprising bolts to detect; and

transfer locations of features of the pre-collected images on the constructed 3D model to an image captured when the A.I. system is online based on the estimated geometric transform to facilitate wirelock detection and/or the crimp detection; and use the spatial transformation and the estimated geometric transform to determine exact 3D location of features in an image captured when the A.I. system is online to facilitate wirelock detection and/or crimp detection.

In the specification and claims, unless the context clearly indicates otherwise, the term“comprising” has the non-exclusive meaning of the word, in the sense of “including at least” rather than the exclusive meaning in the sense of “consisting only of”. The same applies with corresponding grammatical changes to other forms of the word such as“comprise”,“comprises” and so on.

While the invention has been described in the present disclosure in connection with a number of embodiments and implementations, the invention is not so limited but covers various obvious modifications and equivalent arrangements, which fall within the purview of the appended claims. Although features of the invention are expressed in certain combinations among the claims, it is contemplated that these features can be arranged in any combination and order.