RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 61/515,040, filed 04 August 2011 , and entitled "Contactless Vision-Based Crack Thickness Quantification by Incorporating Depth Perception"; U.S. Provisional Patent Application No. 61/515,024, filed 04 August 2011 , and entitled "Adaptive Vision-Based Crack Detection by Incorporating Depth Perception"; and, U.S. Provisional Patent Application No. 61/515,022, filed 04 August 2011 , and entitled "Multi-Image Stitching and Scene Reconstruction for Assessment of System Conditions"; the entire contents of all of which applications are incorporated by reference herein.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

[0002] This invention was made with government support under Grant No. CNS-032875, awarded by the National Science Foundation (NSF). The Government has certain rights in the invention.

BACKGROUND

TECHNICAL FIELD

[0003] This disclosure relates to image processing and pattern cognition in general, and in particular to image-based crack detection and quantification.

DESCRIPTION OF RELATED ART

[0004] Visual inspection of structures is a highly qualitative method in which inspectors visually assess a structure's condition. If a region is inaccessible, typically, optical devices such as binoculars must be used to detect and characterize defects. Although several Non-Destructive Testing (NDT) methods have been proposed for inspection purposes, they are nonadaptive and cannot quantify crack thickness reliably.

SUMMARY [0005] Illustrative embodiments are now discussed and illustrated. Other embodiments may be used in addition or instead. Details which may be apparent or unnecessary may be omitted to save space or for a more effective presentation. Conversely, some embodiments may be practiced without all of the details which are disclosed.

[0006] In general terms, the present invention provides contact-less remote-sensing crack detection and/or quantification methodologies that are based on three-dimensional (3D) scene reconstruction, image processing, and pattern recognition. Systems and methodologies according to the invention can utilize depth perception for detecting and/or quantifying cracks. These methodologies can provide the ability to analyze images captured from any distance and using any focal length or resolution. This unique adaptive feature may be especially useful for incorporation into mobile systems, such as unmanned aerial vehicles (UAV) or mobile autonomous or semi-autonomous robotic systems such as wheel-based or track-based radio controlled robots, as utilizing such structural inspection methods onto those mobile platforms may allow inaccessible regions to be properly inspected for cracks. In exemplary applications, crack detection and/or quantification as described in the present disclosure, can be used for concrete structures such as buildings, damns, bridges, and/or nuclear reactors, and the like.

[0007] In one aspect, the present invention provides contactless methods of detecting cracks, such as those observed in concrete structures.

[0008] In a further aspect, the present invention provides contactless crack quantification methodologies for determining or quantifying the thickness of detected cracks.

[0009] In exemplary embodiments, the approaches may be incorporated with autonomous or semi-autonomous robotic systems.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings are of illustrative embodiments. They do not illustrate all embodiments. Other embodiments may be used in addition or instead. Details that may be apparent or unnecessary may be omitted to save space or for more effective illustration. Some embodiments may be practiced with additional components or steps and/or without all of the components or steps that are illustrated. When the same numeral appears in different drawings, it refers to the same or like components or steps.

[0011] FIG. 1 shows a pair of juxtaposed images: (a) a picture of a crack in concrete, and (b) segmented crack pixels derived from the picture, in accordance with the invention.

[0012] FIG. 2 shows a high-level scheme for crack detection, in accordance with exemplary embodiments of the invention.

[0013] FIG. 3 illustrates the geometric relation between image acquisition parameters of a simple pinhole camera model, as used in accordance with invention.

[0014] FIG. 4 shows a schematic overview and components of the Structure from Motion (SfM) problem, as used in accordance with the present disclosure.

[0015] FIG. 5 is graph showing the relationship between structuring element size, camera focal length, working distance, crack size, camera sensor size, and camera sensor resolution for a simple camera model, in accordance with the present disclosure.

[0016] FIG. 6 is a graph showing the effect of decision making threshold on different performance indices for a neural network classifier, according to an embodiment of the invention.

[0017] FIG. 7 depicts a collection of crack orientation kernels, in accordance with an embodiment of the invention.

[0018] FIG. 8 depicts an example of crack thickness quantification, in accordance with an embodiment of the invention.

[0019] FIG. 9 depicts a collection of crack orientation kernels, in accordance with an exemplary embodiment of the invention.

[0020] FIG. 10 depicts an example of crack thickness quantification, in accordance with an exemplary embodiment of the invention: in (a) the white squares are crack pixels of a larger crack image; (b) shows a strip kernel, 135°, corresponding to the minimum correlation value for a centerline pixel.

[0021] FIG. 1 1 shows three views (a)-(c) illustrating the effect of perspective error for imaging.

[0022] FIG. 12. shows a collection of views (a)-(e) of a scene including a crack, along with (f) the sparse 3D scene reconstruction and recovery of the camera poses.

[0023] FIG. 13 illustrates a high-level block diagram of a system suitable for processing data in accordance with the present invention.

DETAILED DESCRIPTION

[0024] Illustrative embodiments are now described. Other embodiments may be used in addition or instead. Details that may be apparent or

unnecessary may be omitted to save space or for a more effective

presentation. Some embodiments may be practiced with additional

components or steps and/or without all of the components or steps that are described.

[0025] In general terms, the present invention provides contact-less remote-sensing crack detection and/or quantification methodologies that are based on three-dimensional (3D) scene reconstruction, image processing, and pattern recognition. Systems and methodologies according to the invention can utilize depth perception for detecting and/or quantifying cracks. These methodologies can provide the ability to analyze images captured from any distance and using any focal length or resolution. This unique adaptive feature may be especially useful for incorporation into mobile systems, such as unmanned aerial vehicles (UAV) or mobile autonomous or semi-autonomous robotic systems such as wheel-based or track-based radio controlled robots, as utilizing such structural inspection methods onto those mobile platforms may allow inaccessible regions to be properly inspected for cracks. In exemplary applications, crack detection and/or quantification as described in the present disclosure, can be used for concrete structures such as buildings, damns, bridges, and/or nuclear reactors, and the like. [0026] In one aspect, the present invention provides contactless methods of detecting cracks, such as those observed in concrete structures.

[0027] In a further aspect, the present invention provides contactless crack quantification methodologies for determining or quantifying the thickness of detected cracks.

[0028] In exemplary embodiments, the approaches may be incorporated with autonomous or semi-autonomous robotic systems.

[0029] For crack detection and quantification methods in accordance with the invention, one or more cracks is/are segmented from the related background. For example, FIG. 1 shows a pair of juxtaposed images: (a) a picture of a crack in concrete, and (b) segmented crack pixels derived from the picture, in accordance with the invention. The white pixels in FIG. 1 (b) are the extracted crack pixels.

[0030] One aspect of the present invention involves crack detection. Various embodiments of the present invention can receive image data and from such date produce a crack map, depicting one or more detected cracks.

[0031] FIG. 2 shows a high-level scheme for a crack detection system/method 200, in accordance with exemplary embodiments of the invention. First, several pictures of a scene (having one or more cracks visible in a structure or feature) are captured from different views, as indicated by "data sensing" at 210. Next, a three-dimensional (3D) sparse model of the scene is reconstructed, e.g., by solving the structure-from- motion (SfM) problem for the scene, as shown at 220. See, e.g., K.N. Snavely, "Scene Reconstruction and Visualization from Internet Photo Collections, Ph.D. thesis, University of Washington, Seattle, Washington USA (2008). By solving the SfM problem set up for the scenario, the sparse structure of a scene as well as the camera's position, orientation, and internal parameters for each view are determined. By scaling the reconstructed sparse 3D model of a scene, the depth perception is obtained. Subsequently, a morphological crack segmentation operator can be used to segment the whole crack, as shown at 230. For this operator, a structuring element parameter can be automatically adjusted based on the camera focal length, object-camera distance, camera resolution, camera sensor size, and the desired crack thickness. Appropriate features may be extracted and selected for each segmented pattern, as indicated at 240, e.g., using the Linear Discriminant Analysis approach. See, e.g., R.A. Fisher, "The Use of Multiple Measurements in Taxonomic Problems, Annals of Eugenics 7, pp. 179-188 (1936). As shown at 250, one or more classifiers (or trained classifiers) can be used to classify real cracks. Suitable classifiers can include but are not limited to a trained neural network (NN), a support vector machine (SVM), and/or a nearest-neighbor classifier. A multiscale approach can then be introduced to obtain a resulting crack map. Detection system/method 200 is adaptive because, based on the image acquisition specifications, camera-object distance, focal length, and image resolution, parameters are adjusted to detect cracks of interest.

[0032] Crack detection and/or quantification according to the present invention can utilize depth perception. For such depth perception, it can be useful in exemplary embodiments for the three-dimensional (3D) structure of a scene to be constructed or recovered, e.g., from multiple images of the scene. From such a 3D structure of a scene, the geometric relation between image acquisition system parameters and imaged object(s) can be obtained. Alternatively, the geometric relation between image acquisition parameters (e.g., working distance from camera to object, focal length, etc.) and imaged object(s) can be done by other suitable means, e.g., with manual or user- obtained measurements such as with a laser range-finder or other measuring apparatus such as measuring tape.

[0033] FIG. 3 illustrates the geometric relation between image acquisition parameters of a simple pinhole camera model 300, as used in accordance with invention.

[0034] Using the simple pinhole camera model 300 shown in FIG. 3 the relation between different image acquisition parameters is shown in the following: ( WD\( SS

[0035] SF = ) \n , (EQ. 1 )

{ FL \SR

[0036] where SF (mm) is the size of a pattern (e.g., crack thickness) represented by n pixels in an image, WD (mm) is the working distance (camera-object distance), FL (mm) is the camera focal length, SS (mm) is the camera sensor size, and SR (pixels) is the camera sensor resolution. The camera sensor size can be obtained from the manufacturer and the camera sensor resolution is known from the image size. The measurements for the working distance, and the camera focal length are used to quantify an n- pixels feature. These two parameters can by estimated as described below, for exemplary embodiments. Other suitable parameter estimation techniques may of course be utilized.

[0037] For some applications, in order to obtain good or optical crack quantification results, based on experiment, it may be desirable to select the image acquisition system parameters in a way that the thickness of the thinnest crack would be represented by six pixels or more in an image.

Scene Reconstruction - Exemplary Embodiments

[0038] To create depth perception from a collections of images, the 3D structure of a scene can be recovered. For this approach, first, several overlapping images of the object are captured from different views. The SfM approach aims to optimize a 3D sparse point cloud and viewing parameters simultaneously from a set of geometrically matched key points taken from multiple views. FIG. 4 shows a schematic overview of the SfM problem 400.

[0039] In the SfM problem/system 400, scale-invariant feature transform (SIFT) are detected in each image (410) and then matched between all pair of images (420). See D.G. Lowe, "Distinctive Image Features from Scale- Invariant Keypoints," Int'l J. Computer Vision 60, pp. 91 -1 10 (2004). The random sample consensus (RANSAC) algorithm can be used, in some applications, to exclude outliers (430). Other suitable algorithms may be used instead of or in addition to the RANSAC algorithm. These matches can be used to recover focal length, camera center and orientation; and also radial lens distortion parameters (two parameters corresponding to a 4th order radial distortion model can be estimated) for each view, as well as the 3D structure of a scene. This optimization process is referred to herein as bundle adjustment (440).

[0040] Since measuring the camera-object distance is not always an easy or practical task, the reconstructed 3D cloud and camera locations from the SfM problem are used to estimate the working distance; however, the Sal problem estimates the relative 3D point coordinates and camera locations. By knowing how much the camera center has moved between just two of the views, the reconstructed 3D points and camera locations can be scaled. To obtain the absolute camera-object distance, a plane is fitted to the 3D points seen in the view of interest. This can be done by using the RANSAC algorithm to exclude the outlier points. By retrieving the equation of the fitted plane, one can find the intersection between the camera orientation line passing through the camera center and the fitted plane. The distance between this intersection point and the camera center is computed as the working distance.

[0041] Furthermore, the estimated focal lengths from the SfM problem are in pixels. To scale these quantities, the estimated focal length for each view is scaled by the ratio of the sensor size to the sensor resolution. This means that EQ. 1 can be simplified to:

f WD\

SF = — n , (EQ. 2)

Where FL is in pixels.

[0042] Note that if scene reconstruction is impossible (e.g., not enough views are available), the approximate focal length can be extracted from the image Exchangeable Image File Format (EXIF) file. In this case, EQ. 1 can be used to estimate the interaction of the image acquisition parameters provided that the working distance is given.

[0043] Embodiments above are described as utilizing the SfM problem to derive a reconstructed 3D structure to obtain image acquisition parameters or geometry of a scene; as was noted previously, however, the geometric relation between image acquisition parameters (e.g., working distance from camera to object, focal length, etc.) and imaged object(s) can be obtained by other suitable means, e.g., with manual or user-obtained measurements such as with a laser range-finder or other measuring apparatus such as measuring tape. Thus, these other methods and their equivalents may be used with crack detection and/or crack quantification according to the present invention.

[0044]

CRACK DETECTION

[0045] Crack detection methodologies according to the present invention may utilize the scene reconstruction approach described previously. The main elements of the proposed crack detection procedure are segmentation, feature extraction, and decision making. Note that before processing any image, it is preferably undistorted using the distortion coefficients obtained from the SfM problem.

Segmentation

[0046] Segmentation is a set of steps that isolate the patterns that can be potentially classified as a, defined defect. The aim of segmentation is to reduce extraneous data about patterns whose classes are not desired to be known. Several segmentation techniques have been evaluated by the inventors, and it has been concluded that a proposed morphological operation by Salembier as modified and described herein works best for crack detection purposes in components that are typically encountered in civil infrastructure systems.

Morphological Operation

[0047] Morphological image processing, which is based on mathematical morphology, is used to extract useful information about the scene objects. The foundation of morphological image processing is based on previous studies by Minkowski and Metheron on set algebra and topology, respectively. [0048] The morphological operation by Salembier is modified here to enhance its capability for crack extraction in different orientations. See Salembier, P. , "Comparison of Some Morphological Segmentation Algorithms Based on Contrast Enhancement: Application to Automatic Defect Detection," Proceedings of the EUSIPCO-90, Fifth European Signal Processing Conference, pp. 833-836 (1990). The morphological operation used for exemplary embodiment of the subject technology is shown in EQ. 3 as follows I , (EQ. 3)

[0049] where / is the grayscale image, S is the structuring element that defines which neighboring pixels are included in the operation, Ό' is the morphological opening, and '·' is the morphological closing. The output image T is then binarized, e.g. , using Otsu's thresholding method or other suitable binarization schemes, to segment potential crack-like dark regions from the rest of the image. See Otsu, N. , "A Threshold Selection Method from Gray-Level Histograms," IEEE Trans. On Systems, Man, and Cybernetics, pp. 62-66 ( 1979). This nonlinear filter extracts the whole crack as opposed to edge detection approaches where just the edges are segmented.

[0050] Furthermore, small extracted patterns can be eliminated as noise. For this purpose, if the length of a segmented pattern is less than a, minimum length, specified by the user, that pattern is eliminated In order to convert minimum length of interest in unit length to minimum length in pixels, (4) is used:

FL

I = , (EQ. 4)

\ WD

[0051] where / is the defined length by the user in unit length, FL and WD (obtained from SfM and scaling, as described in Section 3) are in pixels and unit length, respectively, and l _p is the length in pixels.

Structuring Element [0052] By choosing the size and shape of the structuring element (i.e., neighborhood), a filter that is sensitive to a specific shape can be constructed. When the structuring element has a line format, it can segment cracks that are perpendicular to it. If the length of the structuring element (in pixels) exceeds the thickness of a dark object in an image, then this object can be segmented by the operation in EQ. 3. Consequently, in exemplary embodiments, linear structuring elements are defined in 0°, 45°, 90°, and 135° orientations.

[0053] The challenge is to find the appropriate size for the structuring element. By having the scaled working distance, the derived formula in EQ. 1 can be used to compute the appropriate structuring element. Using this equation, the size of the appropriate structuring element can be computed based on the crack size of interest (where n is the structuring element size). FIG. 5 shows the relationship between these parameters and can be used to determine appropriate image acquisition system parameters.

Feature Extraction

[0054] After segmenting the patterns of interest, they can be assigned a set of finite values representing quantitative attributes or properties called features. These features should represent the important, characteristics that help identify similar patterns. To determine discriminative features useful for classification purposes, the inventors initially defined and analyzed twenty nine features. Eleven of these features were selected as potentially appropriate features for further analysis. Using the LDA approach, the following five features were found to be discriminately appropriate (i.e., preserving 99.4% of the cumulative feature ranking criteria) for classification: (1 ) eccentricity (a scalar that specifies the eccentricity of the ellipse that has the same second-moments as the segmented object), (2) area of the segmented object divided by the area of the above ellipse, (3) solidity (a scalar specifying the proportion of pixels in the convex hull that also belong to the segmented object), (4) absolute value of the correlation coefficient (here, correlation is defined as the relationship between the horizontal and vertical pixel coordinates), and (5) compactness (the ratio between the square root of the extracted area and its perimeter). The convex hull for a segmented object can be defined as the smallest convex polygon that can contain the object. The above features were computed for each segmented pattern under examination.

Classification

[0055] To evaluate methodologies of crack detection, a feature set consisting of 1 ,910 non-crack feature vectors and 3,961 synthetic crack feature vectors was generated to train and evaluate the classifiers. About 60% of this set was used for training, while the remaining feature vectors were used for validation and testing. Note that due to the lack of access to a large number of real cracks, randomized synthetic cracks were generated to augment the training database. For this reason, real cracks were manually segmented and an algorithm was developed to randomly generate cracks from them. The non-crack feature vectors were extracted from actual scenes. The performance of several SVM and NN classifiers was evaluated. Eventually, a SVM with a 3rd order polynomial kernel and a 3-layer feedforward NN with 10 neurons in the hidden layer and 2 output neurons were used for classification. A nearest-neighbor classifier was used to evaluate the performance of the above classifiers.

[0056] Performances of these three classifiers were analyzed, with the analysis showing that the SVM and NN approaches have very close performances, which were better than a nearest-neighbor classifier.

[0057] Note that the SVM method is a discrete classifier, whereas the NN approaches typically needs a threshold to act as a discrete classifier. In an implemented embodiment, if the value of the crack output neuron was found to be greater than the value of the non-crack neuron, the pattern was classified as a crack, otherwise, as a non-crack. This is identical to set the threshold equal to 0.5.

[0058] FIG. 6 shows the effect of changing the decision making threshold on different performance indices for the specific NN used for an implemented embodiment. In the figure, 'positive predictive value' is the proportion of the correctly classified positives (i.e., cracks), and 'negative predictive value' is the proportion of the correctly classified negatives (i.e., non-cracks). For applications where it is expensive to miss a crack (e.g., inspection purposes), it is recommended to select a more conservative threshold (i.e., a threshold less than 0.5). As a threshold moves toward zero, specificity and positive predictive rates increase while sensitivity and negative predictive rates decrease. This means there will be more false negatives and less false positives. For less sensitive applications, one may select a threshold greater than 0.5. Moreover, FIG. 6 helps decide about the appropriate threshold for a specific application by considering the performance indices. It is worth noting that if the training set, size is infinite, the outputs of the above back- propagation NN can converge to the true a posteriori probabilities].

Multi-Scale Crack Map

In order to obtain a crack map, the crack detection procedure described above was repeated using different structuring elements (i.e., different scales). Note that the extracted multi-scale binary crack map is the union of the detected cracks using different, structuring elements. The proposed crack map can be formulated as:

J ^m (u, v) = {\,3k [S _min , m}c ^k {u,v) = l, and 0 otherwise; (EQ. 5) where J ^m is the crack map at scale (i.e., structuring element) m, Smin, is the minimum structuring element size, C ^k is the binary crack image obtained by using k as the structuring element, and u and v are the pixel coordinates of the crack map image.

In an implemented embodiment, the structuring elements of Γ ,1+ ^{2 to} r were used for generating the crack map where [ ^{~ ~} | is the ceiling function, and n _min and n _max are the structuring element sizes corresponding to the minimum and maximum crack size of interest, respectively. The crack map was can be used for crack localization as well as quantification. Of course while the crack detection methodologies according to the subject technology may be used in conjunction with crack quantification methodologies according to the subject technology, these detection methods may be used with other crack quantification methodologies.

CRACK QUANTIFICATION

[0063] A further aspect of the present invention (or subject technology) includes methodologies (which term includes reference to systems and methods) for crack quantification.

[0064] Utilizing an crack map (one that includes a segmented crack), crack quantification methods of the invention calculate a crack thickness in pixels, along the crack, and then provide for scaling the computed thickness to a unit length.

[0065] FIG. 7-8 illustrate details of an embodiment of crack quantification according to the invention. FIG. 7 depicts a collection of crack orientation kernels while FIG. 8 depicts an example of crack thickness quantification using the kernels of FIG. 7.

[0066] Referring now to FIGS. 7-8, a segmented crack can be thinned using morphological thinning, e.g. in accordance with an embodiment of the subject technology. The remaining pixels can be considered as the centerlines of the cracks. In order to measure a crack thickness, the perpendicular orientation to the crack pattern at each centerline pixel is identified. To reach this goal, the thinned segmented crack can be correlated with a set of collection of orientational kernels. In an exemplary embodiment, 35 kernels are used, where these kernels represent equally- incremented orientations from 0° to 175°. FIG. 7 shows the kernels from 0° to 45°; other kernels can be constructed based on these kernels.

[0067] For each centerline pixel, the kernel corresponding to the maximum correlation value represents the tangential orientation of the centerline. Thickness orientation was then defined as the perpendicular orientation to the detected tangential direction. Next, for each centerline pixel, the pixels in the original segmented crack that are aligned with the corresponding thickness orientation were counted in the horizontal and vertical directions. Using these two values, the hypotenuse was computed and considered to be the crack thickness in pixels. Finally, the crack thickness was converted to a unit length by knowing the camera-object distance and the focal length of the camera.

[0068] The white squares shown in FIG. 8 are crack pixels of a larger crack image, as shown in FIG. 8(a). Such a crack image may be provided or derived from any suitable methodology e.g., such as one resulting from a crack map output from any of the systems or methods described previously for FIGS. 1 -6; other methodologies may of course be used. The shaded squares in FIG. 8(b) represent the centerline obtained by thinning the crack object. The kernel corresponding to 45°, centered at the dark center square, has the highest correlation with the thinned pixels, as shown in FIG. 8.(c). Consequently, the shaded squares in-line with the center dark square as shown in FIG. 8(d), which correspond to 135° direction, indicate the thickness orientation at the center square. As shown in FIG. 8, the number of the thickness pixels in the horizontal and vertical directions are both six (6) pixels, and the crack thickness at the center square is estimated as 8.5 pixels (as the square root of the sum of the squares).

[0069] FIG. 9-10 illustrate details of an exemplary embodiment of crack quantification according to the invention. FIG. 9 depicts a collection of crack orientation kernels useful for the embodiment while FIG. 10 depicts an example of crack thickness quantification, in accordance with an exemplary embodiment of the invention: in (a) the white squares are crack pixels of a larger crack image; (b) shows a strip kernel, 135°, corresponding to the minimum correlation value for a centerline pixel.

[0070] Referring now to FIGS. 9-10, in accordance with an exemplary embodiment of crack quantification, a segmented crack can be correlated with a number of kernels, e.g., as described previously using 35 kernels, where these kernels represent equally-incremented strips from 0° to 175°. FIG. 9 shows the strip kernels from 0° to 45°, where the size of these kernels is 71 x71 pixels. As shown in FIG. 9, for 0° to 45° and 135° to 1 75° kernels, eleven orientational kernels can be arranged vertically to form the strip kernels, where each column consists of eleven non-zero values. For 50° to 130° kernels, the orientational kernels may be arranged horizontally.

[0071] For each centerline pixel, obtained from morphological thinning, the strip kernel corresponding to the minimum correlation value represents the thickness orientation. Each correlation value is the area of the crack that is bounded between the strip edges. Since an eleven-pixel length is relatively small (along specified vertical or horizontal direction in the drawing), the crack thickness does not dramatically change in such a small region, and consequently the areas bounded by the strips can all be assumed to be trapezoids. Furthermore, the minimum correlation value corresponds to the area of the trapezoid that is approximately a rectangle. The length of this rectangle is the projection of the eleven-pixel length on the line which is perpendicular to the thickness orientation (i.e., the tangent at the centerline pixel). Finally, the crack thickness at each centerline pixel can be estimated by dividing the corresponding minimum correlation value by this length.

[0072] FIG. 10 shows an example of the thickness quantification method described above for FIG. 9. In FIG. 10(a), the white squares are crack pixels of a larger crack image, which may be supplied by embodiments according to the present invention or other techniques. The shaded squares, shown in FIG. 10(b), represent the strip kernel, centered at the dark center square, that has the minimum correlation value at the centerline pixel. The kernel orientation in this example is 135°. For a particular centerline pixel, the number of squares (66 for the example shown) within the strip kernel with the lowest correlation value and also within crack edges (see dashed lines in the figure) represents the correlation value. Consequently, the thickness for the indicated centerline pixel is 66/(1 1 χ cos 45°) 8.5 pixels. This thickness can be scaled to obtain the thickness in unit length.

Perspective Error

[0073] The previously-described methodologies are valid if the camera orientation is perpendicular to the plane of the object under inspection. If this plane is not perpendicular to the camera orientation (i.e., the projection surface and the object plane are not parallel), a perspective error will occur. FIG. 1 1 shows three views (a)-(c) illustrating the effect of perspective error for imaging.

[0074] In order to overcome the perspective error, the camera orientation vector and the normal vector of the object plane are needed. The camera orientation vector can be retrieved using SfM (as described above), and the normal plane can be computed by fitting a plane to the reconstructed 3D points, seen in the corresponding view, by excluding outliers using the RANSAC algorithm. For each centerline pixel, the number of pixels that are aligned with the corresponding thickness orientation can be counted in the horizontal and/or vertical directions. Next, the perspective error compensation for each component can be computed as:

X _x (EQ. 6)

cos«

[0075] where A _x is the perspective-free component of the crack thickness (for each centerline pixel), λ _χ is the measured crack thickness, a _x is the angle between the camera orientation vector and the fitted plane's normal vector in the x direction, and x represents either the horizontal or vertical directions. For each centerline pixel, the resultant of the two perspective- free components is the crack thickness.

[0076] A user can interactively select a portion of a crack, and the proposed system will average the crack thicknesses for that region. This will improve the robustness of the system in the presence of noise.

Experimental Results

[0077] In order to evaluate the performance of thickness quantification methodologies described above, an experiment was performed as follows: synthetic cracks with thicknesses of 0.4, 0.6, 0.8, 1 .0, 1 .2, and 1 .4 mm were drawn by a human operator using AutoCAD®, and printed using a 600 dpi HP LaserJet printer. Eighteen images with different camera poses were captured from the printed crack-like patterns to form six image sets. These images were in color, and they contained regular image noise. The crack edges were tapered in the captured images. Each image set consisted of three views, where the distance between two of the camera centers was known. The images were captured by a Canon PowerShot SX20 IS with a resolution of 2592 x 1944 pixels. For each image set, the SfM problem was solved and the camera-object distance was retrieved (as explained in Section 2.3). The working distances in this experiment varied between 725 mm to 1 ,760 mm. First, the cracks were extracted by a crack detection methodology described above. More than 10,000 measurements for each of the above thicknesses were carried out. A total of 70,721 thickness estimations were performed. To increase the robustness of the proposed thickness quantification system, thicknesses within a 5 x 5 neighborhood of each centerline were averaged. Statistical analysis of the collected data confirmed the effectiveness of the methodologies under review.

[0078] There can be many sources of error when quantifying a crack thickness using the above procedure, including bundle adjustment errors, scaling errors, crack orientation errors, and pixel representation errors (i.e., the number of pixels representing a thickness); however, the results of this experimental study indicate that the errors are quite reasonable, and they are amenable to improvement. Due to some rare irregularities in an extracted pattern, a small portion of the thinned image might not represent the exact centerline, which causes errors too. Averaging the neighboring thickness values may help get eliminate such outliers.

[0079] In order to illustrate the capabilities, of crack quantification methodologies according to the invention, a real crack quantification experiment was performed as follows. Five images were taken from a concrete surface. The image acquisition system was identical to the one that was used in the first experiment. These images are shown in FIG. 12 (a), (b), (c), (d), and (e). FIG. 12 (f) shows the reconstructed scene and recovered camera poses. The camera distance between the two side views (i.e., FIG. 12 (a) and (e)) was 1600 mm.

[0080] Here, FIG. 12 (c) is an example used to quantify cracks. The retrieved working distance and focal length for this view were 966 mm and 15759 pixels, respectively. The working distance varied from 800 mm to 1400 mm.

[0081] In order to further evaluate the performance of the exemplary crack quantification methodologies, fifteen crack thicknesses were computed. As mentioned earlier, in practice there is no quantitative approach to quantify cracks in concrete structures. The following approach was used to obtain the ground truth about the computed thicknesses. First, a known length was attached to the region under inspection. Then, an image was captured where the view plane was parallel to the scene plane. The scale was computed as the ratio between the known length and number of pixels representing it. Finally, a thickness was determined by multiplying the number of pixels representing the thickness (which was counted manually) by the computed scale.

[0082] Crack thicknesses were computed using the quantification methodologies described for FIGS. 7-10. The results of the methodology of FIGS. 9-10 were seen to be closer to the ground truth with respect to the methodology of FIGS. 7-8. Both of the approaches were able to quantify real cracks with a reasonable accuracy. Furthermore, in most cases, the methodologies were seen to quantify the thickness slightly greater than its actual thickness, which is desirable (i.e., conservative) for crack monitoring applications. The maximum differences between the results from the first and the second methodologies, for this experiment, and the ground truth values were found to be 0.1 1 mm and 0.08 mm, respectively.

[0083] FIG. 13 illustrates a high-level block diagram of a system 1300 suitable for processing data in accordance with the present invention. System 1300 includes a processing portion 1310 that includes an input/output block 1312 that can receive input data, e.g., from an outside source, and produce an output of output data as shown. A memory unit 1314 is connected by a suitable bus or link to I/O block 1312. Memory unit 13 4 can include or be composed of any suitable type of memory and/or memory structure. A processing system 1316 is connected to the memory unit 1314 and/or I/O block 1312. The processing system 1316 can include or be composed of one or more of any suitable type of processor, e.g., a central processing unit (CPU), an arithmetic processing unit (APU), a graphics processing unit (GPU), and the like. In some embodiments, system 1300 can also include a data acquisition system (1320) that functions to acquire data and provide such to the rest of the system 1300, e.g., via I/O block 1312. In exemplary embodiments, DAQ 1320 can include one or more cameras suitable for capturing one or more images of a scene of interest, e.g., one that include a structure that is suspected of having cracks. In some embodiments, system 1300 can receive a plurality of images as an input, process the images for crack detections in accordance with the present disclosure, and produce one or more multiscale crack maps as an output. In other or similar embodiments, system 1300 can receive one or more crack maps as an input, process such for crack quantification in accordance with the present disclosure, and produce one or more quantified crack maps indicating the quantified thickness for the crack(s).

Exemplary Embodiments:

[0084] While methodologies for both crack detection and crack quantification are generally described above in the context of working independently from one another, exemplary embodiments of the present invention utilize both crack quantification and crack detection as described herein. An example of such a methodology, described as an algorithm, is as follows:

Algorithm

Input: n images of a scene and the camera distance between

two of the views.

For each view:

1 . Establish the working distance and camera parameters by

solving the SfM problem and scaling the reconstructed

scene.

A. Crack Detection 2. Establish the appropriate structuring element based on the working distance and the focal length of the view, as well, as the crack thickness of interest;

3. Segment the potential crack patterns by applying the described morphological operation in (3) on the image;

4. Compute and assign appropriate features to each segmented pattern;

5. Classify cracks from non-crack patterns using a trained classifier (NN or SVM);

6. Repeat steps 2 through 5 for different crack thicknesses of interest and generate the multi-scale crack map as the union of all extracted crack pixels.

Output: the multi-scale crack map

B. Crack Quantification

7. Extract the centerline of each extracted crack using the morphological thinning operation;

8. Find the tangential orientation for each centerline pixel by correlating different orientational kernels with the binary crack map;

9. Estimate the thickness orientation as the perpendicular orientation to the tangent at each centerline pixel;

10. For each centerline pixel, compute the crack pixels that are aligned with the thickness orientation;

1 1 . Compensate for the perspective error by aligning the view plane with the object plane;

12. Compute the thickness in unit length by multiplying the measured thickness in pixels by the ratio between the working distance and the focal length; 13. Average the measured thicknesses in a small neighborhood to improve the robustness of the quantification

system.

Output: the crack thickness values

[0085] Accordingly, various benefits and advantages may be achieved through use of aspects and embodiments of the present invention.

[0086] Unless otherwise indicated, the method, techniques, and methodologies of crack detection and/or quantification that have been discussed herein are implemented with a computer system configured to perform the functions that have been described herein for the component. Each computer system includes one or more processors, tangible memories (e.g., random access memories (RAMs), read-only memories (ROMs), and/or programmable read only memories (PROMS)), tangible storage devices (e.g. , hard disk drives, CD/DVD drives, and/or flash memories), system buses, video processing components, network communication components, input/output ports, and/or user interface devices (e.g., keyboards, pointing devices, displays, microphones, sound reproduction systems, and/or touch screens).

[0087] Each computer system for implementing the methodologies of crack detection and/or quantification may be or include a desktop computer or a portable computer, such as a laptop computer, a notebook computer, a tablet computer, a PDA, a smartphone, or part of a larger system, such a vehicle, appliance, and/or telephone system.

[0088] A single computer system may be shared or networked for implementing the subject technology described herein.

[0089] Each computer system for implementing the subject technology may include one or more computers at the same or different locations. When at different locations, the computers may be configured to communicate with one another through a wired and/or wireless network communication system.

[0090] Each computer system may include software (e.g., one or more operating systems, device drivers, application programs, and/or communication programs). When software is included, the software includes programming instructions and may include associated data and libraries. When included, the programming instructions are configured to implement one or more algorithms that implement one or more of the functions of the computer system, as recited herein. The description of each function that is performed by each computer system also constitutes a description of the algorithm(s) that performs that function.

[0091] The software may be stored on or in one or more non-transitory, tangible storage devices, such as one or more hard disk drives, CDs, DVDs, and/or flash memories. The software may be in source code and/or object code format. Associated data may be stored in any type of volatile and/or non-volatile memory. The software may be loaded into a non-transitory memory and executed by one or more processors.

[0092] The components, steps, features, objects, benefits, and advantages that have been discussed are merely illustrative. None of them, nor the discussions relating to them, are intended to limit the scope of protection in any way. Numerous other embodiments are also contemplated. These include embodiments that have fewer, additional, and/or different components, steps, features, objects, benefits, and advantages. These also include embodiments in which the components and/or steps are arranged and/or ordered differently.

[0093] For example, while detection and quantification methodologies have been described herein in the context of cracks (e.g., in structures), these methodologies may be effective for other pattern analysis purposes, e.g., texture analysis. Moreover, while embodiments above are described as utilizing the SfM problem to derive a reconstructed 3D structure to obtain image acquisition parameters or geometry of a scene; as was noted previously, the geometric relation between image acquisition parameters (e.g., working distance from camera to object, focal length, etc.) and imaged object(s) can be obtained by other suitable means, e.g., with manual or user- obtained measurements such as with a laser range-finder or other measuring apparatus such as measuring tape. Thus, these other methods and their equivalents may be used with crack detection and/or crack quantification according to the present invention.

[0094] Unless otherwise stated, all measurements, values, ratings, positions, magnitudes, sizes, and other specifications that are set forth in this specification, including in the claims that follow, are approximate, not exact. They are intended to have a reasonable range that is consistent with the functions to which they relate and with what is customary in the art to which they pertain.

[0095] Any and all articles, patents, patent applications, and other publications that have been cited in this disclosure are incorporated herein by reference in their respective entirety.

[0096] The phrase "means for" when used in a claim is intended to and should be interpreted to embrace the corresponding structures and materials that have been described and their equivalents. Similarly, the phrase "step for" when used in a claim is intended to and should be interpreted to embrace the corresponding acts that have been described and their equivalents. The absence of these phrases from a claim means that the claim is not intended to and should not be interpreted to be limited to these corresponding structures, materials, or acts, or to their equivalents.

[0097] The scope of protection is limited solely by the claims that now follow. That scope is intended and should be interpreted to be as broad as is consistent with the ordinary meaning of the language that is used in the claims when interpreted in light of this specification and the prosecution history that follows, except where specific meanings have been set forth, and to encompass all structural and functional equivalents.

[0098] Relational terms such as "first" and "second" and the like may be used solely to distinguish one entity or action from another, without necessarily requiring or implying any actual relationship or order between them. The terms "comprises," "comprising," and any other variation thereof when used in connection with a list of elements in the specification or claims are intended to indicate that the list is not exclusive and that other elements may be included. Similarly, an element preceded by an "a" or an "an" does not, without further constraints, preclude the existence of additional elements of the identical type.

[0099] None of the claims are intended to embrace subject matter that fails to satisfy the requirement of Sections 101 , 102, or 103 of the Patent Act, nor should they be interpreted in such a way. Any unintended coverage of such subject matter is hereby disclaimed. Except as just stated in this paragraph, nothing that has been stated or illustrated is intended or should be interpreted to cause a dedication of any component, step, feature, object, benefit, advantage, or equivalent to the public, regardless of whether it is or is not recited in the claims.

[00100] The abstract is provided to help the reader quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, various features in the foregoing detailed description are grouped together in various embodiments to streamline the disclosure. This method of disclosure should not be interpreted as requiring claimed embodiments to require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the detailed description, with each claim standing on its own as separately claimed subject matter.

RELATED INFORMATION CONCERNING MULTI-IMAGE STITCHING AND SCENE RECONSTRUCTION FOR EVALUATING DEFECT EVOLUTION IN

STRUCTURES

SUMMARY

It is well-recognized that civil infrastructure monitoring approaches that rely on visual approaches will continue to be an important methodology for condition assessment of such systems. Current inspection standards for structures such as bridges require an inspector to travel to a target structure site and visually assess the structure's condition. A less time-consuming and inexpensive alternative to current visual monitoring methods is to use a system that could inspect structures remotely and also more frequently. This article presents and evaluates the underlying technical elements for the development of an integrated inspection software tool that is based on the use of inexpensive digital cameras. For this purpose, digital cameras are appropriately mounted on a structure (e.g., a bridge) and can zoom or rotate in three directions (similar to traffic cameras). They are remotely controlled by an inspector, which allows the visual assessment of the structure's condition by looking at images captured by the cameras. By not having to travel to the structure's site, other issues related to safety considerations and traffic detouring are consequently bypassed. The proposed system gives an inspector the ability to compare the current (visual) situation of a structure with its former condition. If an inspector notices a defect in the current view, he/she can request a reconstruction of the same view using images that were previously captured and automatically stored in a database. Furthermore, by generating databases that consist of periodically captured images of a structure, the proposed system allows an inspector to evaluate the evolution of changes by simultaneously comparing the structure's condition at different time periods. The essential components of the proposed virtual image reconstruction system are: keypoint detection, keypoint matching, image selection, outlier exclusion, bundle adjustment, composition, and cropping. Several illustrative examples are presented in this article to demonstrate the capabilities, as well as the limitations, of the proposed vision-based inspection procedure.

KEYWORDS

structural health monitoring, inspection tool, computer vision, scene reconstruction, image stitching

INTRODUCTION

[00101] Civil infrastructure system assets represent a significant fraction of the global assets and in the US are estimated to be worth $20 trillion. These systems are subject to deterioration due to excessive usage, overloading, and aging materials, as well as insufficient maintenance and inspection deficiencies. Bridges constitute one of the major civil infrastructure systems in the US. According to the National Bridge Inventory (NBI), more than 10,400 bridges are categorized as structurally deficient. ¹ There is an urgent need to develop effective approaches for the inspection and evaluation of these bridges. In addition, periodical inspections and maintenance of bridges will prolong their service life. ²

[00102] The National Bridge Inspection Program was established to inspect all highway bridges on the federal aid system by the Federal Highway Act of 1968. This mandatory inspection program was expanded to include all public bridges by Congress. ³ According to the American Association of State Highway and Transportation Officials (AASHTO) Manual for Condition Evaluation of Bridges, there are five categories of inspections: initial inspections, routine inspections, in-depth inspections, damage inspections, and special inspections. ⁴

[00103] Initial inspection is the first inspection of a new bridge. It ascertains the baseline for potential problem areas and estimates the structure inventory. Routine inspection is a periodic inspection of the bridge to identify changes with regards to the previous inspection. All the requirements of the National Bridge Inventory Standards (NBIS) must be satisfied in routine inspections. These requirements dictate the inspector's qualifications and the frequency of inspections. In-depth inspection is a close-up inspection of a few structural members in which the defects cannot be detected by a routine inspection. In-depth inspections are less frequent than routine inspections. Sophisticated Nondestructive Evaluation (NDE) methods might be used for in-depth inspections to help the inspector detect deficiencies. Damage inspection is carried out in response to damage caused by human actions or environmental conditions. Special inspection is the monitoring of a known defect. ⁴

[00104] Even though many NDE techniques, such as liquid penetrate tests, ultrasonic testing, radiographic testing, magnetic flux leakage (magnetic flow), Eddy current, acoustic emission, electrochemical potential and resistance measuring, and infrared thermography have been developed for the inspection of bridge structures, ² visual inspection is the predominant method used for the inspection of bridges. In many cases, other NDE techniques are compared with visual inspection results. ⁴ Visual inspection is a labor-intensive task that must be carried out at least bi-annually in many cases. ⁵

[00105] Recently, more effort has been dedicated to the improvement of NDEs that are useful for visual inspection. Mizuno et al. ⁶ developed an interactive support system that can enable inspectors to reach a decision at the bridge site using a wearable computer and a mobile communication device. Such a support system provides enhanced technical knowledge to the inspector. Choaset and Henning ⁷ developed a remotely controlled serpentine robot that can be used for visual inspection of bridges using a mounted sensor suite. The serpentine robot is flexible enough to access all the points of a bridge structure. These techniques have yet to be adopted in current inspection processes. Gonzalez-Aguilera and Gomez-Lahoz ⁸ developed a pipeline for dimensional analysis of bridge structures from an image. Kim et al. ⁹ conducted a preliminary study to show the feasibility of autonomous monitoring of bridge construction progress based on image analysis. Jahanshahi et al: ⁶ surveyed and evaluated several image-based approaches for automatic defect detection of bridge structures. None of the mentioned studies provide a pipeline for an inspector to visually assess the defect evolution in a structure.

[00106] The visual inspection of structures is a subjective process that depends on the inspector's experience and focus. Furthermore, inspectors who feel comfortable with height and lift spend more time finishing their inspection and are more likely to locate defects. ³ Difficulties in accessing some parts of a bridge hinders the transmission of knowledge and

experience from an inspector to other inspectors. Consequently, improving the skills and experiences of inspectors will take much time and effort using current visual inspection practices. ⁶ [00107] A systematic approach that provides inspectors with the ability to inspect a structure remotely by controlling cameras at a bridge site can overcome the above shortcomings and avoid the costs of traffic detouring during the inspection. Cameras can be conveniently mounted on a structure (similar to the Department of Transportation traffic cameras), and in the case of bridges, the cameras can be mounted on bridge columns Even though the cameras may be constrained in regard to translation, they can easily rotate in two or three directions. This can give inspectors the ability to assess a relatively large area covered by the camera. Determining the optimal number of cameras and their positions are interesting problems that will not be discussed here. The main purpose of this study is to enable inspectors to accurately and conveniently compare the structure's current condition with its former condition. In this study, a database of images captured by a camera is constructed automatically. If the inspector notices a defect in the current view, he or she can request the reconstruction of that view from the previously captured images. In this way, the inspector can look at the current view and the reconstructed view simultaneously. Since the reconstructed view is based on images that are in the database and it virtually has the same camera pose of the current view, the inspector can easily compare the current condition of the structure with its previous condition and evaluate the evolution of defects. Figure A shows a simplified schematic hardware configuration of the proposed inspection system.

[00108] A description of the inspection problem to be solved is given in the section 'Multi-image stitching and scene reconstruction'. Several components of the automatic multi-image stitching (i.e., image registration) are described in this section. Section 'Keypoint detection' introduces a keypoint detection technique for image registration purposes. Section 'Initial keypoint matching' describes keypoint matching between multiple images. Image selection and outlier exclusion are discussed in section 'Image selection and outlier exclusion'. Bundle Adjustment (BA) is introduced in section 'Bundle

adjustment'. Composition and blending of multiple images in one reference view are briefly reviewed in sections 'Composition' and 'Blending and exposure compensation', respectively. Experimental results are discussed and evaluated in section 'Experimental results and discussion'. The last section includes a summary and suggested future work. See also Figure A.

[00109] Figure A: Schematic hardware configuration of the image-based inspection system.

MULTI-IMAGE STITCHING AND SCENE RECONSTRUCTION

[00110] Image stitching algorithms are among the most widely used algorithms in computer vision. ¹¹ In this study, the proposed procedure has similarities to panoramic reconstruction of a view (i.e., in both cases the camera is constrained in regard to translation). Hence, this study benefited from the available extensive research literature for panorama image stitching. ^{1 1 13} Panorama image stitching has several commercial

applications; ¹³ however, to the best of the author's knowledge, the assessment of defect evolution in structures has not been among these applications.

[00111] PhotoStitch is an image stitching software that is bundled with Canon digital cameras. This software requires an initialization, such as a horizontal or vertical sweep. ¹⁴ AutoStitch is a panoramic image stitching software that automatically recognizes panoramas in an unordered set of images. AutoStitch transfers and stitches images in a spherical coordinate system. ¹⁴ Commercial photo stitching products Autopano Pro, Serif

PanoramaPlus, and Calico are all based on AutoStitch. These commercial products have more advanced stitching capabilities than AutoStitch.

Microsoft Image Editor is also a free advanced panoramic image stitcher developed by Microsoft Research. Panorama Tools is a set of programs and libraries for re-projecting and blending multiple images to construct panoramas. Panorama Tools is the core engine for many panorama

Graphical User Interface (GUI) front-ends, such as PTgui and hugin.

[00112] There are generally two types of image alignment and stitching algorithms: direct and feature-based. Even though direct stitching benefits from using all of the image data and can yield highly accurate registrations, feature-based stitching is faster, more robust, and has the ability to automatically detect overlapping relationships among a set of unordered images. ¹¹ ' ¹⁴ The latter characteristic of feature-based algorithms is ideal for the purpose of this study. Therefore, the proposed system in this study is essentially a feature-based image stitching algorithm.

[00113] In order to reconstruct a view from a large collection of captured images in the database and project it based on the current camera pose, the images should be selected automatically from the database. For this purpose, automatic 'keypoints' should be detected. In the next step, images that have greater number of matching keypoints with the current view should be identified. A procedure to select the images from the database is introduced below. The next step is to eliminate the outlier matching keypoints. Then, the camera poses for each of the selected views and the current view are computed. This is the BA problem. The stitching of the selected images will take place after this step. Finally, the postprocessing will take place to give a smooth comparable image. In this section, the above components are introduced and discussed. Figure B shows a schematic overview of the proposed image stitching procedure described above.

KEYPOINT DETECTION

[001 14] In keypoint detection, control points such as distinctive objects, edges, topographies, points, line intersections, and corners are detected. A comparison of common keypoint detection algorithms can be found in reference. ¹⁰

[001 15] Scale-Invariant Feature Transform (SIFT) ¹⁵ is a popular choice for keypoint detection. SIFT keypoints are invariant to changes in scale and rotation, and partially invariant to changes in 3D viewpoint and illumination. The SIFT operator is also highly discriminative and robust to significant amounts of image noise. Keypoints are identified by finding local extrema in a scale-space representation of the image. For each extremum point, SIFT then computes a gradient orientation histogram over a region around the point producing a 128-element descriptor vector. INITIAL KEYPOINT MATCHING

[00116] At this stage, the detected keypoints from the current-view image are matched with the detected keypoints of the database images. The matching keypoints are used as the criterion for similarity comparison between the current-view image and the database images. An initial estimate is necessary to identify the correspondences. Each SIFT keypoint has a 128- element descriptor vector assigned to it. The Euclidean distances between each keypoint's descriptor vector in the reference (current view) image and any of the keypoint descriptor vectors in the input image (any of the database images) are computed. An effective matching strategy was introduced by Lowe based on comparing the distance of the closest neighbor to that of the second-closest neighbor.15 In this study, all matches in which the distance ratio is greater than 0.6 were rejected. By doing this, about 95% of the false matches are discarded. Further details related to this approach may be found. ¹⁵ High false match elimination rate guarantees the functionality of the initial keypoint matching as a resemblance criterion between images. Figure C shows the matching of 42 SIFT keypoints in two overlapping images using the above technique. There are few outliers among these matching

keypoints. See Figure B.

[00117] Figure B. Schematic overview of the proposed image stitching procedure. First, automatic keypoints are detected in all the images.

Keypoint are then matched between the current view and each database image. Database images that have greater number of matching keypoints with the current view and can appropriately reconstruct the scene are selected. Next, keypoints between all the selected images (including the current view) are matched and the outliers are eliminated. Then, the BA problem is solved. The selected images are composed and blended. Finally, the reconstructed view is cropped and compared with the current view.

IMAGE SELECTION AND OUTLIER EXCLUSION

[00118] At this stage of the data processing, the images in the database that have overlaps with the current-view image are selected. One approach is to select a fixed number of images that have the greatest number of matching keypoints (i.e., overlap or resemblance) with the current-view image; however, in some cases a scene can be reconstructed by fewer images which reduces the registration error. In order to select the optimum number of images, the following procedure is proposed.

[001 19] All the images that have a number of initial matching keypoints greater than a threshold were selected. Images which have more than 40 matches with the current-view image were selected. In order to improve the correspondence estimation, outliers (defined as incorrect matching

keypoints) are identified and excluded. RANdom SAmple Consensus

(RANSAC) is used to compute homography between two images as well as to find outliers. Now, the image that has the greatest number of matching keypoints with the current-view image is transformed onto the current-view image (using the estimated homography by RANSAC) to find its projection boundaries on the current-view image. Then, the current-view image is updated by setting the pixel values in the projection region to zero (i.e., that projection region will be eliminated from the current-view image). The above procedure is repeated using the remaining images and the updated current- view image until the updated current-view image turns into a black scene (which means the selected images cover the whole current-view image). If, after one iteration, none of the remaining selected images have any matching keypoints with the updated current-view image, the latter one is updated by stretching the remaining regions by 10% in the horizontal and vertical directions. This iteration continues until the updated images turn into a black scene. Figure C shows the matching SIFT keypoints between two

overlapping images. The RANSAC algorithm identified eight outlier matches.

BUNDLE ADJUSTMENT

[00120] BA is usually the last step of multi-view structure and motion reconstruction procedures that are feature-based. ¹⁶ BA aims to optimize 3D structure and viewing parameters (e.g., camera pose and intrinsic

calibration) simultaneously, from a set of geometrically matched keypoints, from multiple views. In fact, BA is a large sparse geometric optimization problem in which the parameters consist of camera poses and calibrations, as well as 3D keypoint coordinates. ¹⁷ The Levenberg-Marquardt (LM) algorithm is an iterative minimization method that has been used to minimize the reprojection error of the BA problem. When the LM algorithm is used to minimize the reprojection error, the solution of the normal equations has the computational complexity of (\ ^ri ) j _n ^' the number of parameters.

Consequently, as the number of parameters increases, the minimization will be costly and , at some point, not feasible. One way to overcome the high cost of the LM algorithm is to use sparse methods. ¹² Lourakis and Argyros ¹⁶ provided details of how to efficiently solve the BA problem based on the sparse structure of the Jacobian matrix used in the LM algorithm. Their modified implementation of this algorithm is used to solve the BA problem in this study. See Figure C.

[00121] Figure C. Matching SIFT keypoints in two overlapping images (matched keypoints are connected by matching lines). Red (dark) matching lines show the outliers identified by RANSAC.

COMPOSITION

[00122] In the problem under discussion, the camera has four degrees of freedom (three for the camera orientations and one for the camera focal length). If the projection of the / ^' th point on the ^' th and /th view is defined by x and Xjj' , respectively, since the camera center is stationary, the relation between xy and xy, is a homography: i = K _j R/ Rj K ^{~ 1} .

[00123] Note that x and xy, are homogeneous image positions

(Kij—wy [H,y v,y if where L jj and vy are the pixel positions corresponding to the / ^" th 3D point in the y ^' th view). This means that, if the unknown camera parameters are estimated, one can transform the images from one plane to the other one through the above homography. [00124] The selected images are all transformed onto the plane of the current-view image and stitched using the homographies between each selected image and the current-view image. The composition surface is flat. Consequently, straight lines remain straight, which is important for inspection purposes. Finally, the reconstructed scene is cropped and can then be compared to the current-view image.

BLENDING AND EXPOSURE COMPENSATION

[00125] After stitching the images together, some image edges are still visible. This effect is usually due to exposure differences, vignetting

(reduction of the image intensity at the periphery of the image), radial distortion, or mis-registration errors. ¹⁴ Due to mis-registration or radial distortion, linear blending of overlapped images may blur the overlapping regions. In the problem under discussion, the preservation of the high- frequency components (e.g., cracks) are of interest. It is challenging to smooth low-frequency exposure variances of an image without blurring its high-frequency components. A solution to this problem is to use a technique that blends low-frequency components over a larger spatial region and high- frequency components over a smaller region. For this purpose, the Laplacian pyramid blending ¹⁸ technique is used. Figure D(a) shows a linear blending of images, whereas Figure D(b) shows the Laplacian pyramid blending of the same images. The visual artifacts due to radial distortion and misregistrations (i.e. , the 'ghosting effect' at the gusset plate and the column, and blurriness of the diagonal member) are eliminated in the latter one. In this study, six pyramid levels are used.

[00126] In order to reduce the exposure differences between the stitching images and the current-view image, the following error function is minimized e w∞{x>yl<xiii(x,y) - /«(χ, ,ν)] ² ,

[00127] where ά, is the exposure compensation coefficient for image /, l(x, y) is the brightness value of the pixel located at (x, y) in image / ^' , / _cv ( , y) is the brightness of the current-view image, and l/V _C i(x _> y) identifies the overlapping region between the current-view image and image /. The weighting function W _CV (x, y) is 1 in the overlapping region and 0 otherwise.

[00128] By minimizing the error in Equation (2), the exposure compensation coefficients are computed as:

∑,.∑, W _cyli x, y _cv (x,y)

[00129] Each stitching image is then multiplied by its corresponding exposure coefficient prior to blending. See Figure D.

[00130] Figure D. (a) Image stitching using a linear image blending, (b) image stitching using the Laplacian pyramid blending. The visual artifacts (i.e., the 'ghosting effect' at the gusset plate and the column, and blurriness of the diagonal member) in image (a), due to radial distortion and misregistration, are eliminated in image (b).

[00131] Note that, in order to gain some robustness against mis-registration errors, the average brightness values in the overlapping regions in Equation (3) have been used.

EXPERIMENTAL RESULTS AND DISCUSSION

[00132] Figure E shows an image database consisting of four image sets: 24 images of a truss system, 24 images of a full-scale structural model, 24 images of a typical hospital support structure, and 32 images of a

magnetorheological (MR) damper with its attached structures. The average camera-object distance for image databases in Figure E(a), (b), (c), and (d) are 2, 7, 2.5, and 1 .5 m, respectively. This quantity can be increased by using a more powerful lens. Each of these images has at least 50% overlap with its neighboring images. All of the images are saved in the database without any specific order (the images in Figure E are presented in an order to give the reader the sense about the overlapping regions); however, indexing the images can enhance the search speed for image selection. The resolution is 640 x 480 pixels for each image. All the images are captured by a Canon PowerShot A610 digital camera. Note that four different image sets are saved in a single database to show the robustness of the image selection algorithm in the presence of the outlier images. The SIFT keypoints are detected and saved in a file for each of the database images. In this way, there is no need to recompute the keypoints for the database images while reconstructing each scene. See Figure E.

[00133] Figure E. An image database consisting of: (a) 24 images of a truss system, (b) 24 images of a full-scale structural model, (c) 24 images of a typical hospital ceiling support structure, and (d) 32 images of an MR damper with its attached structures.

[00134] Figure F(a) shows a current-view image of the truss system shown in Figure E(a). A yellow tape is attached to the truss gusset plate in this image (this tape is not present in any of the database images). Figure F(b), (c), and (d) are the (autonomously) selected images from the database to reconstruct the scene. Figure G shows the reconstructed scene and the contribution of the selected images in Figure F.

[00135] On a AMD Athlon II X4 (2.6 GHz) processor, it takes 37 s for the proposed system to detect SIFT keypoints in the current-view image, find the matching keypoints between the current-view image and all the images in the database (104 images), select matching images, solve the BA problem, blend the selected images and crop the reconstructed scene. BA take less than a second of the whole computation time (because the sparse BA algorithm is efficiently implemented in C++). Note that no parallel processing is used in this process. As the number of images in the database increases, the search time will also increase. For the current example, if just the images in Figure 5(a) were saved in the database, the computation time would decrease by about 10 s. Furthermore, higher resolution images lead to a greater number of keypoints, which leads to higher computational cost.

Except for the BA algorithm, which is implemented in C++ , the rest of the algorithms are implemented in MATLAB. For faster performance (i.e., online processing), all the algorithms should be efficiently implemented in C++(or an equivalent computer language). In this study, our goal is to provide the proof of concept for the proposed inspection system. In all our experiments, the BA took the least computation time. Most of the computation time was consumed for searching images in the database. Fortunately, this process can be parallelized. See Figures F and G.

[00136] Figure F. (a) Current-view image of a truss system, (b), (c), and (d) are three image matches autonomously selected from the database. Note that a yellow tape is attached to the truss gusset plate in the current-view image. This tape is absent in the images from the database.

[00137] Figure G. The reconstructed scene and the contribution of the selected images in Figure F.

[00138] The current-view image and the reconstructed scene (after blending and exposure compensation) are shown in Figure H(a) and (b), respectively. One can recognize the yellow tape (shown by a red circle) in the current-view image while it is absent in the reconstructed scene (i.e., this synthetic change was not present in a prior inspection of the structure).

[00139] Figure l(a) shows the scene reconstruction and the contribution of five selected images from the database (Figure E). Figure l(b) is the current- view image captured relatively far away from the structure. Figure l(c) and (d) show the reconstructed scene using a linear blending versus the

Laplacian pyramid blending. Exposure differences of stitched images have led to a poor reconstruction result in Figure l(c) with respect to Figure l(d). Figure l(e) shows the reconstructed scene using the Laplacian pyramid blending and the proposed exposure compensation technique. One can compare the lower left side of Figure l(b), (d), and (e) to evaluate the effect of the proposed exposure compensation approach. It is obvious that Figure l(e) has more resemblance (i.e., less exposure difference) with Figure l(b) (the current-view image). Furthermore, there is an aluminum element in the reconstructed scene (Figure 1(e)) which is absent in the current-view image (Figure 1(b)).

[00140] Figure J presents an example where an inspector encounters a suspicious condition in the current-view image. By using the proposed system, he or she can visually compare the current condition of the structure with its previous status to identify probable changes. Figure J(a) shows the scene reconstruction and the contribution of four selected images from the database (Figure E) that contains 104 images. Figure J(b) is the current-view image of a typical hospital support structure. Figure J(c) is the final reconstructed scene. One can recognize that a ceiling tile is missing in the current-view image by comparing Figure J(b) and (c).

[00141] Figure K(a) shows the scene reconstruction and the contribution of 12 selected images from the database. Figure K(b) is the current-view image of an MR damper. Note that this image is zoomed-out relative to the images in the database (Figure E) and taken in a different lighting condition to show the invariance of the proposed system with respect to image exposure (the image is captured by using flash). Figure K(c) shows the reconstructed scene using the Laplacian pyramid blending and the proposed exposure

compensation technique. One can recognize that two bolts (shown by red circles in Figure K(c)) are missing in the current-view image by comparing Figure K(b) and (c). This is a good example of the practical capacities of the proposed methodology.

[00142] If an inspector zooms-in to have a closer look at the center of Figure K(b) (Figure L(a)), the proposed system finds the matching image in the database (Figure L(b)) and reconstructs the scene (Figure L(c)).

[00143] If overlapping images are captured periodically and saved in separate databases, then the evolution of changes can be efficiently tracked through time. Figure M shows three sets of image databases captured from a structural system at different time periods t _{1 t} t ₂ , and f ₃ where < t ₂ < h.

Figure N shows the change evolution of this structural system. Figure N(a), (b), and (c) are reconstructed from the images in Figure M(a), (b), and (c), respectively. Figure N(a) shows a bolted beam-column connection at inspection time . Figure N(b) shows that the nut is disappeared at

inspection time t ₂ . Figure N(c) shows that the bolt is disappeared and the beam has been displaced at inspection time f ₃ . Figure N(d) shows the current view of the connection where the beam has been displaced even more. COMPARISON OF IMAGE STITCHING ALGORITHMS

[00144] In order to check the registration performance of the proposed system, the registration Root Mean Square (RMS) error is computed. Ten real image sets with different image resolutions and different number of stitching images (varying from 2 to 13) are used. The RMS errors in this study are compared with the RMS errors from a well-known automatic stitcher: AutoStitch.19 Figure O confirms that the registration errors of this study and that of AutoStitch are close. It is worth mentioning that AutoStitch optimizes the camera parameters (motion) by minimizing the homography errors between stitching images, while in this study the proposed system minimizes the reprojection error by optimizing the camera parameters and the 3D coordinates of matching keypoints (motion and structure)

simultaneously. The physics of the problem is better preserved using the latter approach. In many panoramic image stitching algorithms, including AutoStitch, the composition surface is cylindrical or spherical whereas in this study this surface is flat. By using the flat composition surface, straight lines remain straight which is important for inspection purposes.

MINIMUM MEASURABLE DEFECT

[00145] Based on the inspection guidelines, appropriate cameras should be used for defect detection purposes. Here, a general introduction about the different parameters that affect the detection capabilities of a camera and their relation is presented. In order to accurately measure a feature in an image, at least two pixels which represent the feature have to be detected. See Figures H, I, J, K, L, M, N, and O.

[00146] Figure H. (a) Current-view image, (b) the reconstructed scene after blending, exposure compensation, and cropping. The yellow tape (shown by red circle) on the truss gusset plate in image (a) is absent in image (b).

[00147] Figure I. (a) The scene reconstruction and the contribution of five selected images from the database (Figure E), (b) current-view image of a full-scale structural model, (c) scene reconstruction using a linear blending, (d) scene reconstruction using the Laplacian pyramid blending, and (e) scene reconstruction using the Laplacian pyramid blending and exposure

compensation. There is an aluminum element in image (e) which is absent in image (b).

[00148] Figure J. (a) The reconstruction and the contribution of three selected images from the database (Figure E), (b) current-view image of a typical hospital support structure, and (c) scene reconstruction using the Laplacian pyramid blending and the exposure compensation. Note that a ceiling tile is missing in image (a).

[00149] Figure K. (a) The reconstruction and the contribution of 12 selected images from the database (Figure E), (b) current-view image of an MR damper (zoomed out), and (c) scene reconstruction using the Laplacian pyramid blending and exposure compensation. Note that two missing bolts in image (b) are shown by red circles in image (c). The lighting condition is different in images (a) and (b).

[00150] Figure L. (a) Current-view image of an MR damper (zoomed in), (b) the selected image from the database (Figure (E)), and (c) reconstructed scene. Note that there are two missing bolts is image (a) in comparison with image (c).

[00151] Figure M. Three image databases of a structural system captured at different time periods, (a), (b). and (c) images of a structural system captured at time periods t|. t ₂ . and t ₃ . respectively (t| < t ₂ < t ₃ ).

[00152] Figure N. Change evolution in a structural system: (a), (b), and (c) scene reconstructions of a structural system (beam-column connection) at time periods ti, t ₂ , and t ₃ , respectively ( < t ₂ < t ₃ ). (d) Current-view image of the same structural system.

[00153] Figure O. Comparison of RMS errors using this study and

AutoStitch. Ten real image sets with different image resolutions and different number of stitching images are used to evaluate the registration errors.

[00154] Below, a formula that gives the smallest measurable feature in an image is given: where SF is the smallest measurable feature, FL is the camera focal length, WD is the working distance (distance between an object and the camera), SS is the camera sensor size, and SR is the camera sensor resolution. Note that this formula does not consider lens distortion and the type of defect detection algorithm. This formula helps to select the appropriate image acquisition system for a given working distance and the smallest measurable defect.

[00155] For instance, for a working distance of 3000 mm and the usage of a Canon PowerShot A610 digital camera (where the maximum focal length is 29.2 mm, the sensor size is 7.2 mm and 5 3 mm in horizontal and vertical directions, and the maximum sensor resolution is 2592 x 1944 pixels) the minimum measurable feature is:

3000 m 1.2 mm

St =—— x————— x 2 = 0.57 mm.

293 mm 2^)2 pixel

SUMMARY AND FUTURE WORK

[00156] Among the possible techniques for inspecting civil infrastructure, the use of optical instrumentation that relies on image processing is a less time-consuming and inexpensive alternative to current monitoring methods. Visual inspection is the predominant method for bridge inspections. The visual inspection of structures is a subjective measure that relies heavily on the inspector's experience and focus (attention to detail). Furthermore, inspectors who do not have fear of heights and feel comfortable with height and lift spend more time finishing their inspection and are more likely to locate defects. Difficulties accessing some parts of a bridge adversely affect the transmission of knowledge and experience from an inspector to other inspectors. The integration of visual inspection results and the optical instrumentation measurements gives the inspector the chance to inspect the structure remotely by controlling cameras at the bridge site. This approach resolves the above difficulties and avoids costs of traffic detouring during the inspection. Cameras can be appropriately mounted on the structure. Although the cameras are constrained by translation (i.e., attached to a fixed location), they can rotate in three directions. The inspector thus has the appropriate tools to inspect different parts of the structure from different views.

[00157] The main purpose of this study is to give the inspector the ability to compare the current situation of the structure with the results of previous inspections. In order to reach this goal, a database of images captured by a camera is constructed automatically. When the inspector notices a defect in the current view, he or she can request the reconstruction of the same view from the images captured previously. In this way, the inspector can evaluate the growth of a defect of interest.

[00158] In order to stitch the images that have been previously captured from different views and reconstruct an image from them that has the same view as the newly captured image, the database must be autonomously searched to select images relevant to the new image. For this purpose, automatic keypoints should be detected. In the next step, the images that have the greatest number of matching keypoints with the current view are identified. The following step is to eliminate the outlier matching keypoints and find the optimum number of matches that reconstruct the current view. Then, the camera parameters for each of the selected views as well as the 3D coordinates of the matching keypoints are computed. This is the BA problem. The stitching and blending of the selected images will take place after this stage. Eventually, the reconstructed scene is cropped so as to be compared to the current view. If overlapping images are captured periodically and saved in separate databases, then the evolution of changes can be tracked through time by multiple reconstruction of a scene from images captured at different time intervals. Furthermore, this study confirms the feasibility of a system that can autonomously reconstruct scenes from previously captured images and provide a reference scene for an inspector while he or she visually inspects the structure.

[00159] In this study, several experimental examples are provided. Four different sets of images (total of 104 images) are saved in a single database to show the robustness of the proposed system in the presence of outlier images. Furthermore, three sets of images captured from a structural system at different time periods are presented to show how the proposed system facilitates the tracking of change evolution in a scene. The reconstruction examples in this study include zoom-in, zoom-out, different lighting

conditions, missing parts, displacement, and change evolution that show the capabilities of the proposed system for different scenarios.

[00160] Since many structures, such as bridges and tall buildings, continuously oscillate, the captured images are subject to motion blur (i.e., the apparent streaking caused by rapidly moving objects). Image stabilizing approaches can be used to prevent blurring caused by the minute shaking of a camera lens. Restoration techniques can be used for motion blur caused by movements of the object of interest or intense movements of the camera. Although there are several proposed algorithms for restoring motion blur, selecting an appropriate algorithm for the proposed application in this study needs further research. Furthermore, motion blur depends on parameters such as shutter speed, focal length, and motion frequency. By considering the effects of these parameters, it is possible to reduce motion blur by selecting the appropriate image acquisition system.

[00161] Under different weather conditions, the contrast and color of images are altered. On the other hand, the proposed study requires robust detection of keypoints. Fortunately, SIFT keypoints are highly robust to image noise and partially invariant to illumination changes. Consequently, except for some severe weather conditions, where the light is intensively scattered by the atmosphere, the proposed study can appropriately reconstruct the virtual scene (Figure K). In the case of very extreme weather conditions, contrast restoration techniques can be used to remove weather effects from images. ²⁰ Furthermore, in order to keep illumination variations as low as possible, the use of light-emitting diode (LED) is recommended.

[00162] The correction of radial distortion is not considered in this study. Radial distortion can be modeled using low-order polynomials. Furthermore, selection of blending weights based on the sharpness of the captured images is more of interest for inspection purposes whereas in this study, the closeness of a given pixel to the center of the selected images is used to assign blending weights. Eventually, implementing all of the discussed algorithms in a computer language such as C or C++ will dramatically decrease the computation time and will hasten the online usage of the proposed system. In this study, in order to initially match the keypoints an exact f-nearest neighbors approach is used. The usage of a k-d tree to find the approximate nearest neighbors will significantly improve the speed of the proposed system. ²¹

ACKNOWLEDGMENT

[00163] This study was supported in part by grants from US National Science Foundation. The authors would like to thank Zahra Tehrani for her critical reading of the manuscript.

REFERENCES - END NOTES

1. Chong KP, Carino NJ and Washer G. Health monitoring of civil infrastructures.

Smart Mater Struct 2003; 12(3): 483-493.

2. McCrea A, Chamberlain D and Navon R. Automated inspection and restoration of steel bridges - A critical review of methods and enabling technologies.

Autom Constr 2002; 11 (4): 351-373.

3. Graybeal BA, Phares BM, Rolander DD, Moore M and Washer G. Visual

inspection of highway bridges. J Nondestr Eval 2002; 21(3): 67-83.

4. Moore M, Phares B, Graybeal B, Rolander D and Washer G. Reliability of Visual Inspection for Highway Bridges, Volume I: Final Report. US Department of Transportation, Federal Highway Administration; 2001.

URL:<http://www.tfhrc.gov/hnr20/nde/01020.htm> (accessed May 2008). . Chang PC, Flatau A and Liu SC. Review paper: health monitoring of civil infrastructure. Struct Health Monit 2003; 2(3): 257-267. 6. Mizuno Y, Abe M, Fujino Y and Abe M. Development of interactive support system for visual inspection of bridges. Proc SPIE Int Soc Opt Eng 2001 ; 4337: 155-166.

7. Choset H and Henning W. A follow-the-leader approach to serpentine robot motion planning. J Aerosp Eng April 1999; 12(2): 65-73.

8. Gonzalez-Aguilera D and Gomez-Lahoz J. Dimensional analysis of bridges from a single image. J Comput Civil Eng 2009; 23(6): 319-329.

9. Kim C, Kim H and Ju Y. Bridge construction progress monitoring using image analysis. In: Proceedings of the 26th International Symposium on Automation and Robotics in Construction (ISARC 2009), Austin, Texas, USA, June 2009, pp.101-104.

10. Jahanshahi MR, Kelly JS, Masri SF and Sukhatme GS. A survey and

evaluation of promising approaches for automatic image-based defect detection of bridge structures. Struct Infrastruct Eng 2009; 5(6): 455-486.

H .Szeliski R. Image alignment and stitching: a tutorial. Found Trends Comput Graphics Vision 2006; 2(1 ): 1 -104.

12. Hartley R and Zisserman A. Multiple view geometry in computer vision.

Cambridge, UK: Cambridge University Press, 2000, ISBN 0-521-62304-9.

13. Brown M and Lowe DG. Recognising Panoramas. In: Proceedings of the 9th International Conference on Computer Vision ( ICCV2003), Nice, France, 2003, pp.1218-1225.

14. Brown MA. Mult-image matching using invatiant features. Vancouver, British Columbia, Canada: The University of British Columbia, 2005.

15. Lowe DG. Distinctive image features from scale-invariant keypoints. Int J

Comput Vision 2004; 60(2): 91 -110. 16. Lourakis MIA and Argyros AA. The Design and Implementation of a Generic Sparse Bundle Adjustment Software Package Based on the Levenberg- Marquardt Algorithm. Vol. 340, http://www.ics.forth.gr/-lourakis/ sba, Heraklion, Crete, Greece: Institute of Computer Science - FORTH, 2004 (accessed December 2009).

17. Triggs B, McLauchlan PF, Hartley Rl and Fitzgibbon AW. Bundle adjustment - a modern synthesis. In: Vision algorithms.* theory and practice. International Workshop on Vision Algorithms, Berlin, Germany, 2000, pp.298-372.

18. Burt PJ and Adelson EH. A multiresolution spline with application to image mosaics. ACM Trans Graphics October 1983; 2(4): 217-236.

19. Brown M and Lowe D. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision 2007; 74(1 ): 59-73.

20. Narasimhan SG and Nayar SK. Contrast restoration of weather degraded

images. IEEE Trans Pattern Anal Mach Intel! 2003; 25(1 ): 713-724.

21. Beis JS and Lowe DG. Shape indexing using approximate nearest-neighbour search in high-dimensional spaces. In: Conference on Computer Vision and Pattern Recognition. Puerto Rico, 1997, pp.1000-1006.

[00164] To the extent that there is any inconsistency in this "Related

Information" section with the material before it, the material before it should govern.