A MOVING OBJECT DETECTION METHOD

Field of the Invention

The present invention relates to moving object detection methods for surveillance systems.

Background of the Invention

Moving object detection application is critical for both airborne and land-based surveillance systems with a fixed and a rotating imaging sensor. The aim of such systems is to report moving objects to an operator for awareness against possible threats, or to track multiple moving targets automatically.

Moving object detection registers a sequence of images acquired by an imaging sensor and extracts possible moving regions by taking difference of this geometrically registered image with respect to selected reference. Image residuals (difference with respect to the geometrically registered reference) of an aligned sequence include both moving and non-moving objects. The major reasons of false alarms are parallax effect and contrast changes among consecutive images.

Parallax effect occurs on objects with varying depths with respect to the camera used for surveillance. Buildings and trees are the major source of false alarms due to parallax. Such static structures produce optical flow fields consistently since their depth deviation causes an optical misalignment when warped by the estimated global geometric transformation (perspective, or affine).

False alarms also occur due to misalignment of high contrast image structures. Gradient based suppression of residuals dampens such false alarms. False alarms manifest itself by misalignment of thin line structures among consecutive images such as road sides, roof edges.

The United States patent document numbered US2008278584 discloses a moving object detection apparatus and method by using optical flow analysis. The apparatus includes four modules of image capturing, image aligning, pixel matching, and moving object detection. Plural images are successively inputted under a camera. Based on neighbouring images, frame relationship on the neighbouring images is estimated. With the frame relationship, a set of warping parameter is further estimated. Based on the wrapping parameter, the background areas of the neighbouring images are aligned to obtain an aligned previous image. After the alignment, a corresponding motion vector for each pixel on the neighbouring images is traced. By analysing all the information generated from the optical flow, registration of the images with respect to the reference can be carried out for a moving object detection application. Optical flow technique used in US2008278584 is utilized for registration purposes.

The United States patent document numbered US2011116682 discloses an object detection method and an object detection system, suitable for detecting moving object information of a video stream having a plurality of images, are provided. The method performs a moving object foreground detection on each of the images, so as to obtain a first foreground detection image comprising a plurality of moving objects. The method also performs a texture object foreground detection on each of the images, so as to obtain a second foreground detection image comprising a plurality of texture objects. The moving objects in the first foreground detection image and the texture objects in the second foreground detection image are selected and filtered, and then the remaining moving objects or texture objects after the filtering are output as real moving object information. The United States patent document numbered US2009052740 discloses a moving object detecting device measures a congestion degree of a space and utilizes the congestion degree for tracking. In performing the tracking, a direction measured by a laser range sensor is heavily weighted when the congestion degree is low. When the congestion degree is high, a sensor fusion is performed by heavily weighting a direction measured by an image processing on a captured image to obtain a moving object estimating direction, and obtains a distance by the laser range sensor in the moving object estimating direction.

The United States patent document numbered US20150030202 discloses a method is provided for an intelligent video processing system based on object detection. The method includes receiving an input video sequence corresponding to a video program, obtaining a plurality of frames of the input video sequence, and obtaining a computational constraint and a temporal rate constraint. The method also includes determining one or more regions of interest (ROIs) of the plurality of frames based on the computational constraint and temporal rate constraint, and selecting a desired set of frames from the plurality of frames based on the ROIs such that the desired set of frames substantially represent a view path of the plurality of frames. Further, the method includes detecting object occurrences from the desired set of frames based on the selected desired set of frames such that a computational cost and a number of frames for detecting the object occurrences are under the computational constraint and temporal rate constraint.

Existing methods may not produce satisfactory results in terms of false alarms. Thus, further methodology is necessary to improve moving object detection performance in terms of false alarms. Optical flow based moving object detections was investigated earlier, where flow vectors were only utilized for registration purposes. However, this usage of optical flow does not necessarily reduce false alarms. Summary of the Invention

The objective of the present invention is to realize a moving object detection method which reduces the false alarms resulting from parallax effect and contrast changes.

Another objective of the present invention is to realize a moving object detection method which utilizes optical flow to suppress residuals, without any relation to registration, in accumulative manner using the camera motion.

Detailed Description of the Invention

A method realized to fulfill the objective of the present invention is illustrated in the accompanying figures, in which:

Figure 1 is the flowchart of the method.

Figure 2 is a single image taken from a video sequence

Figure 3 is Residual history image (RHI) accumulated through time of a video sequence

Figure 4 is optical flow vectors

Figure 5 is optical flow magnitudes

Figure 6 is suppressed residual image The steps and parts illustrated in the figures are individually numbered where the numbers refer to the following:

100. Moving object detection method

101. Storing acquired images in an image storage unit (1010),

102. Registering stored images onto a reference image and taking difference between the stored images and the reference image,

103. Accumulating residuals using motion history imaging of differences, 104. Calculating optical flows of an acquired image with respect to the reference image,

105. Projecting optical flow vectors onto the direction of camera motion,

106. Suppressing accumulated residuals calculated in step 103, by using the stored optical flow projections,

107. Extracting blobs within suppressed residual image,

108. Declaring moving targets by using a joint deferred decision logic tracker. 1010. Image storage unit

1020. Residual storage unit

1021. Transformations storage unit

1040. Flow history storage unit

1070. Connected component storage unit

RHI. Residual history image

FHI. Flow history image

SRI. Suppressed residual image

In the method (100), acquired images during surveillance are recorded to an image storage unit (1010), in step 101. In the preferred embodiment of the invention, the image storage unit (1010) is a circular buffer.

Images in the image storage unit (1010) are geometrically registered onto a current reference image in step 102. Geometric registration is the processing block which aligns an input image onto a given reference spatially. Geometric registration parameters of the images, which are stored in the image storage unit (1010), are stored in a geometric transformations storage unit (1021). Difference between registered images and the reference frame are also calculated in step 102 and sum of the residuals is stored in residual storage unit (1020). In the preferred embodiment of the invention, the residual storage unit (1020), and geometric transformation storage unit are circular buffers. Residuals are defined in detail in following paragraphs. Residuals are accumulated in step 103 and result is written to the Residual History Image (RHI). Here, pixels which do not include any motion are penalized as given in [Collins, R. 2006, Moving Object Localization in Thermal Imagery by Forward-backward MHI] ;

Here P_ _x denotes the geometric projection computed from time (τ -1) to (τ). "d ^' is the temporal decay to penalize pixels with no motion. D(x,y,x,k) is residual at location (x,y) at time instance (τ) with a time gap of (k) (i.e. residual is computed between image τ and τ+k). Residual is the difference of an image acquired at time τ and geometrically registered image that is acquired at time τ +k, where "k" may be positive or negative. "T" is the difference threshold to declare that there is putative motion among consecutive images. P _+k I(x, y,T+k) is the geometric projection that aligns image acquired at time 1 onto image acquired at time T + k D(x, y, T,k) = I(x, y, T) -P _+k I(x, y, T+k) . /is the image acquired, whose time instance is denoted by 1 , and a pixel coordinate is represented by x,y. H _F (x, y, r ) is the pixel value of a Motion history image (MHI) at pixel (x,y) computed at time instance ( 1).

x, y, T, i) < T

Here MHI equation in [Collins, R. 2006, Moving Object Localization in Thermal Imagery by Forward-backward MHI] is slightly modified to incorporate multiple input images to be used for computing residual of several images captured at different times. W _{ is the weight assigned to each residual for MHI computation. A linearly decreasing weighting function is used in the preferred embodiment of the invention. The reference image is selected from the image storage unit (1010). There is an inherent tradeoff over the selection of reference image index. Selecting the reference image index as the most current index produces less false alarms, however has high probability of misdetecting the slowly moving targets. Selecting the reference image acquired at distant time instances enables catching slowly moving targets at the expense of extreme false alarm due to parallax. In the preferred embodiment of the invention, reference image index has to be selected depending on application requirements. Using every image in MHI as proposed overcomes the presented challenges. Optical flow is calculated for the most recently acquired image with respect to a previous image with a specified time delay in step 104 and result is stored in Flow History Storage Unit (1040). An error constant is needed to calculate optical flows (u & v)

Where

£ _b = E _x u + E _y v + E _t (brightness constancy equation) is the rate of brightness change

and

£ _c ² = (u— uf + (v— v) ² is the measure of departure from smoothness in the velocity field, u and v are the final averaged optical flow magnitudes. In one of the preferred embodiments of the invention u and v can be calculated with Horn- Schunck methods. However, computation of u and v is not limited to mentioned methods here.

^and ^ are the image gradients in horizontal and vertical dimensions, respectively. £ _¾ is the temporal image difference with respect to the previous frame.

Total error constant to be minimized to compute optical flows (u & v) is calculated as follows;

ε ¹ = jj [ ² e _c ² + ε )dxdy

1 Optical flows are projected to the direction of camera motion in step 105 and these projected flow vectors (2D vector arrays) are stored in Flow history storage unit (1040) after being warped by the inverse of the estimated camera motion. This way, a pixel in Flow history image (FHI) obtained after the process in step 105, accumulates flows corresponding to the same 3D coordinate throughout time. Flow history image (FHI) holds the optical flow vector (u,v) for each pixel of the image. Optical flow vector of each pixel is alpha-filtered temporally and stored on Flow history storage unit (1040) for temporal consistency. Note that optical flow vector tuple (u,v) at each spatial location is projected onto the direction of the camera motion using ( P _ _x ). OC denotes the temporal weighting for each optical flow component projected to the camera direction.

FHI _u (x, y, T) = FHI _u (x, y, τ - 1) * (1 - a) + p;_ _lU (x, y) * a

FHI _V (x, y, T) = FHI _v (x, y, τ - 1) * (1 - a) + P^vix, y) * a

Residual history image (RHI) obtained after the process in step 103, is suppressed using Flow history image (FHI) in step 106. Residuals are penalized by the suppression operation depending on the flow history magnitude. Higher optical flow magnitude results in further suppression of residual in exponential manner. In a preferred embodiment, residuals can be reduced exponentially. Suppressed residual is stored in Supressed Residual History Image (SRI). Suppressed residuals are used in Motion History Image ( H _f (X, γ, τ) ) computation. Blobs within this Motion History Image are extracted using connected component analysis in step 107. Connected component analysis labels binary pixels which have neighborhood as a component and enumerates each component with a unique number. Results of step 107 are stored in Connected Component Storage Unit (1070). Centroids of extracted blobs are given to deferred decision logic tracking module to declare moving targets in step 108. Deferred decision logic analyzes motion of each blob throughout time and assigns scores to blobs depending on the expected kinematic. Blobs with consistent motion have higher scores, and thus declared as a moving object. Within the scope of these basic concepts, it is possible to develop a wide variety of embodiments of the inventive "moving object detection method (100)". The invention cannot be limited to the examples described herein; it is essentially according to the claims.