SYSTEM AND METHOD FOR DETECTING TARGETS IN MARITIME SURVEILLANCE APPLICATIONS

Field of the Invention The present invention relates to the field of image processing and methodologies to search and find possible targets in an image, especially on infrared imaging systems used in maritime surveillance applications.

Background of the Invention

It is known that there are methods and models to determine the position of an object (or equivalently a "target") in a still image or in a video frame. Such methods are used in maritime surveillance systems in which there is a sensor acquiring an infrared image of the scene under consideration and generally these images are in gray scale. This image consists of a two dimensional array of pixels which represent the radiation in an infrared band at these locations. Currently, there are systems to determine a target's location using a statistical approach, for example by calculating the standard deviation of pixel values of the neighbouring pixels around a pixel and successively decreasing this neighbouring area until a target is detected. Using such an approach, only the targets whose size smaller than the neighbouring area can be detected. Also, due to problems such as the possible noise from the sensor, texture of the target or unexpected temperature changes in the case of infrared vision, detecting a target becomes much more complicated using this kind of approaches.

Another method which is currently used to detect a target uses the image gradient values of the initial image but this method is much more susceptible to image noise. A multi-scale template matching method is proposed instead of these methods. In an infrared image, generally, the target is expected to be brighter than its neighbourhood and this bright zone on the target is generally diffuses to the neighbouring area in a gradient manner. In theory, this brightness distribution is expected to form a local Gaussian distribution as depicted in Figure 2, where the base of the 3D graph represents the pixels of a local area in the image and the perpendicular axis represents the intensity at that point. When this local area is correlated with a predetermined filter, a correlation value is found using related formula. There are existing methods using matched filters to detect a target in an image. The images should also be downsampled and re-analyzed to be able to detect different sized targets using a fixed sized filter. In some of the currently used methods wherein a Gaussian filter is used to correlate a target, a downscaling step is either not introduced or only the peak values are found. An example to such a method involves finding a correlation between a two dimensional block in a two dimensional image with a matched filter and finding the peak values throughout the image as possible target candidates. In some applications, finding only the peak values may result in incorrectly detected targets since some regions of the image may not be appropriate to search a target. In maritime surveillance applications for example, target candidates appearing near the horizon are much more important and accordingly different threshold values must be determined for different regions of the scene under consideration instead of finding all peak values. In addition, currently used methods use complex matched filters to find targets of a determined shape, which slows down the process and are not efficient.

To sum up, the current methods are not offering a reliable and efficient way of detecting a target in maritime surveillance applications with minimum possible false detections. The clutter in the image and the intensity difference caused by the distance makes the known methods error-prone and a method in which different thresholds can be set for different regions is required. The United States patent document US20110001823, an application in the state of the art, discloses a method for detecting target location and size, using pixel standard deviation where the window size is sought by varying window size by one pixel at each iteration.

The United States patent document US7430303, an application in the state of the art, discloses a method for detecting a target in images using image gradient values. The United States patent document US2004151343, an application in the state of the art, discloses a method which employs an elliptical Laplacian pyramid based matched filter to find candidate targets in the image data.

Summary of the Invention

The first objective of the present invention is to provide a reliable and efficient methodology to detect the possible targets in an image for a maritime surveillance application. Another objective of the present invention is to provide a methodology to define different regions of the image with different sensitivities for target detection.

Detailed Description of the Invention A system and method realized to fulfil the objective of the present invention is illustrated in the accompanying figures, in which:

Figure 1 is the schematic view of the preferred embodiment system.

Figure 2 is the 3D graph of a Gaussian distribution.

Figure 3 is a conceptual drawing of a representative threshold image.

Figure 4 is the flowchart of the preferred method of the present invention. The components illustrated in the figures are individually numbered where the numbers refer to the following:

1. System for detecting a target in maritime surveillance applications

2. Image sensor

3. Image processing unit

F. Threshold image

H. High threshold region

L. Low threshold region

M. Medium threshold region

A method for detecting a target in maritime surveillance applications (100) fundamentally comprises the following steps,

- receiving a pixel image (101),

row normalization of the received image (102),

generating an image pyramid of the received image for at least two scales; by first passing the lower scale image from a Gaussian filter and then downsampling it, lowest scale being the original image (103),

correlating an image block around each pixel of each scale of received image with a predetermined Gaussian filter (template), finding a normalized cross correlation value for each pixel of each scale and generating a respective correlation image for each scale of the pyramid

(104),

creating a threshold image with the same dimensions as the received image indicating a threshold value for each pixel (105),

generating an image pyramid for the threshold image with the same number of levels as the received image's pyramid by first passing the lower level's image from a Gaussian filter and then downsampling it

(106), comparing each level of pyramid of the correlation image with the respective level of pyramid image of the threshold image and finding pixels for which the correlation image has larger value than the threshold image (107),

- transferring positions of each pixel found in the levels of pyramid to the original image's level and listing all the pixels in the original image's level as the candidate targets (108).

First, an image of the scene under consideration is received (101). In the case of an infrared camera system, this will be a two dimensional grey scale pixel image. Each pixel represents the radiation in infrared band for that location of the scene under consideration. The main aim of this target detection system is to detect the potential targets and for an infrared system, a potential target is locally brighter than its neighborhood and can be modeled with a Gaussian as discussed above. Due to the reasons listed below, Gaussian assumption might not be valid;

In the vicinity of the horizon, the sky is brighter than the sea. This instantaneous change impairs the smoothness of Gaussian.

Since the intensity of each pixel depends on the distance to the sensor, the average intensity of each line of the image is different and a gradual change may be present, assuming the horizon line is perfectly horizontal with respect to the image coordinates. Even though this effect may be neglected at high resolution correlation calculations, at higher pyramid levels it becomes a problem.

Considering these problems, to make the method stable and successful, a normalization preprocessing step is introduced, where an average intensity value is found for every row and it is subtracted from each pixel of that row (102). As a result of this row normalization step, the average intensity or brightness of each row is equalized and the above mentioned problems are eliminated. For this preprocessing step, it is crucial to have the perfectly stabilized horizon line with respect to the image coordinate system and therefore a previous image stabilization step, which adjusts the slope of the horizon line with respect to the image coordinate system, is needed before row normalization of the image in a preferred embodiment.

Then, an image pyramid of the row normalized image is generated for a predetermined number of levels (103). This step is carried out to make sure that targets of different sizes are detected since the matched filter's size is not changing. As higher pyramid levels are generated, larger targets start to become smaller and smaller. Small targets are detected in lower levels whereas larger ones are detected in higher levels of the pyramid. In a preferred configuration of the present inventive method, at least one additional level is generated including the original image. Pyramid generation is done by first passing the initial received image from a Gaussian filter and then downsampling it by a factor of 2. If the initial image is said to be the zeroth level, then the filtered and downsampled version of it becomes the first level and the second level will be obtained by filtering and downsampling of this previous level image. The number of levels is determined according to the maximum size of potential targets. The kernel (w) of the Gaussian filter used for pyramid generation in a preferred configuration is; w = [1/16 4/16 6/16 4/16 1/16]

And the five by five Gaussian filter (IIBLUR) used for pyramid formation in a preferred configuration, derived from the kernel is

1/256 4/256 6/256 4/ 256 1/ 256

4/ 256 16/ 256 24/ 256 16/256 4/256 h BLUR 6/ 256 24/ 256 36/256 24/256 6/ 256

4/ 256 16/ 256 24/ 256 16/256 4/256

1/256 4/256 6/256 4/ 256 1/ 256 In the following step, a region around each pixel of each pyramid level of row normalized image is correlated with a predetermined Gaussian filter (104). In a preferred configuration, a seven by seven Gaussian filter (h _G -r) is used.

0 0 0 1 0 0 0

0 2 7 11 7 2 0

0 7 30 50 30 7 0

1 11 50 80 50 11 1 /512

0 7 30 50 30 7 0

0 2 7 11 7 2 0

0 0 0 1 0 0 0

This implies that in this preferred configuration, 7x7 regions around each pixel of the level images are correlated by this Gaussian filter by coinciding the midpoint of the filter with the pixel under consideration and calculating the normalized cross correlation between these using the following equation;

∑∑a _m - ave(Q)* (F _m - ave(F))

m n

l∑∑ C - ave(I)) ² ∑∑ (F^ - ave(F)) ²

V m n m n where I represents the image block around a pixel to be correlated and F represents the Gaussian filter; lowercase m and n represents the pixel at the m* row and η ^Λ column of the subscripted matrix and ave(F) and ave(I) represents the average values of the filter and image block, respectively. As it is apparent, this formula can be applied to any square filter with any size and the correlation value is found between -1 and 1. To be able to have a midpoint on the filter, it is square and has odd number of rows in this preferred configuration. At pixel positions where there is a good matching between the template and actual image, the correlation value is closer to 1. The pixels that have high correlation values are regarded as location of candidate targets. After the calculation of normalized cross correlation values for each pixel, a correlation image is obtained. By repeating this process for each level of the pyramid, correlation images of different sizes are obtained. In order to compare with the correlation values, a threshold image with the same dimensions as the received image is generated which indicates a threshold correlation value for each pixel position (105).

Since some critical targets are mostly close to the horizon for maritime surveillance applications, a low threshold region (L) is generated in the threshold image (F) around the horizon with the lowest threshold values. The other two regions above and below this low threshold region are adjusted to have higher threshold values than this low threshold region (L). Below the low threshold region, a high threshold region (H) having the highest values is introduced in order to prevent from clutter in the sea. Above the low threshold region, a medium threshold region (M) having threshold values in between the two other regions is introduced to be more sensitive to targets in the sky. (Figure 2)

The threshold image is preferentially generated by a horizon detection algorithm in a preferred configuration, which intends to detect the horizon in a received image and determine all thresholds accordingly. In order to utilize threshold image at different scales, an image pyramid of the threshold image with the same number of levels as the received image's pyramid is generated using the same filter and same downsampling ratio. (106)

Having all the necessary images in the image pyramid of the initial image and a respective threshold image for each of them, each image in the image pyramid of the correlation image is compared with the respective image in the image pyramid of the threshold image. The pixels whose threshold image values are larger than the correlation image are found in every levels of the pyramid in step (107). These pixels indicate the position of potential targets. Finally, targets detected at different levels of the image pyramid must be related to the original image. For this purpose, positions of each pixel found in higher pyramid levels are transferred to the original image's coordinates using the simple relation

(2 ^k m - 2 ^k_1 + ½, 2 ^k n - 2 ^k_1 + ½)

Here, lowercase m and n represent coordinates of a pixel in the k level of the pyramid and the above relation converges to the original coordinates (m,n) for level zero (k=0). Then all the found targets are combined and listed in the original image's coordinate system as the candidate targets (108).

Ultimately, the inventive method (100) provides a way of preferentially comparing correlation values with a threshold and utilizes multi-scale template matching. The generation of the threshold image is independent from this invention but in fact, the capability of assigning different thresholds and row normalization makes the method (100) a valuable technique to detect targets. Utilization of horizon line is a very important aspect of the invention since a threshold image is constructed by assigning higher priority to the targets in the vicinity of the horizon. In order to detect the targets of different sizes, an image pyramid is constructed and a template matching based detection scheme is applied for all scales of the pyramid.

A system for detecting targets in maritime surveillance applications (1) fundamentally comprises;

- at least one image sensor (2) to acquire at least one electronic pixel image of the scene under consideration when necessary,

- at least one image processing unit (3) configured to receive at least one image and implement the method for detecting a target in maritime surveillance applications (100) using this image and output targets found by the method (100) if there is any. In a preferred embodiment of the present invention, image sensor (2) is a sensor configured to acquire an image with a contrast between a possible target and its background. Preferentially image sensor (2) is an infrared vision camera which is able to differentiate temperature differences in the scene.

In a preferred embodiment of the present invention, image processing unit (3) is configured to receive at least one image from image sensor (2). Anyone proficient in the image processing field should understand that this system (1) and method (100) is applied on a sequence of images and targets can be continuously monitored. In another embodiment of the present inventive system (1), image processing unit (3) is configured to receive at least one image from a recorded or live video stream.

In conclusion, a reliable and efficient methodology to detect the targets of different sizes in an image is proposed for maritime surveillance applications.

Within the scope of these basic concepts, it is possible to develop a wide variety of embodiments of the inventive "system and method for detecting targets in maritime surveillance applications" (1), (100). The invention cannot be limited to the examples described herein; it is essentially according to the claims.