Background of the Invention

[0002] Object counting is an important part of video surveillance and video analysis system. Object counting is best described as automatic process to calculate number of object in the scene or passing through a monitored area or line.

[0003] It is still unsolved issue in object counting where the objects are often appears with different sizes and shapes. The most significant challenge in object counting is the occlusion level of the objects when the counting is performed. With the detected motion blob, the actual number of object is easily determined by known average object size information if the objects in the motion blobs are not occluded. However, the process of determining of number of object has become a non-trivial operation if the objects in the motion blob are occluded to each other. If the occlusion is very high, the actual number of object will be underestimated. Otherwise, the actual number of object will be overestimated if the objects are very loose to each others.

[0004] Therefore, a solution is needed to determine the objects in the motion blob as the objects generally are occluded each others during the counting process is executed.

Summary of the Invention

[0005] Therefore, one objective of the present invention is to provide a system and method for accurate object counting.

[0006] One aspect of the present invention provides a method of object counting, wherein the method is performed in a computer-executable electronic medium. In one embodiment, the method comprises initializing an area of interest, wherein an area where the number of objects that move into/out this area is to be counted is defined; dividing a captured scene of the area of interest into a grid of blocks; determining the average size information of an object for each block of the grid of blocks; learning occlusion rates of motion blobs in the captured scene so that classifiers are optimized based on the occlusion rates of the motion blobs; detecting objects in motion blobs in the captured scene; tracking the detected objects in the motion blobs so as to give the detected moving blobs a consistent label throughout the appearance of the blobs in the scene ; and counting objects in the tracked motion blobs by analyzing the tracked motion blobs based on their occlusion rates, where the occlusion rates of the tracked motion blobs is determined with the optimized classifier; whereby upon finishing counting will the number of objects be output to a displaying device that is electronically connected with the computer-executable electronic medium.

[0007] in another embodiment of the method, in the step of determining the average size information of an object for each block of the grid of blocks, the average size information is recorded for block grid that has been travelled by the object during registration; and for any individual block grid where the average size information of an object is not calculated as the object does not travel to that block, the average size information of an object is automatically estimated based on the average size information of other individual block grids that have been updated during the travel of the object.

[0008] In another embodiment of the method, the step of learning the occlusion rate comprises determining classes of motion blobs according to their occlusion rate and arrangement type; extracting features for each determined class of motion blob; and optimizing classifier with the extracted features. In a further embodiment of the method, the step of determining classes of motion blobs according to their occlusion rate and arrangement type comprises analyzing the dimension of the motion blobs to determine the arrangement types; and analyzing the overlapping ratio of the objects in the motion blobs to determine the occlusion rate for each arrangement type.

[0009] In another embodiment of the method, the step of extracting features for each determined class of motion blob comprises extracting the silhouette feature of the objects area as the foreground information; extracting the silhouette feature of the non- objects area as the background information; extracting the internal features of the objects area including edge information, texture and shape information of the motion blob; and storing the extracted features in a storage medium.

[0010] In another embodiment of the method, the step of counting objects in motion blobs comprises extracting the bounding box information of the tracked motion blob; and deciding on whether the bounding box information is equal to a noise ratio; if yes, the number of objects for the motion blob image is assigned to zero; if no, a further decision is made as to whether the bounding box information is equivalent to a single object; if yes, the number of objects for the motion blob is assigned to one; if no, a yet another decision step is made to as to whether the bounding box information is larger than a single object; if yes, the number of count for the motion blob is determined based on the occlusion rate. In a further embodiment of the method, the step of counting the number of objects in the motion blob based on occlusion rate comprises extracting the features of the motion blob; selecting the optimized classifier and determining the value of the occlusion rate; updating the average size information of the object that is determined during the registration process as a new value based on the occlusion rate; and determining the number of object from the blob image by dividing the blob image accordingly.

[0011] Another aspect of the present invention provides an object counting system.

In one embodiment, the system comprises a capturing device for capturing images of an area of interest; a computer-executable electronic medium for storage and embedded algorithm, wherein the computer-executable electronic medium is electronically connected to the capturing device for receiving the captured images; and a displaying device electronically connected to the computer-executable electronic medium for displaying the output from the computer-executable electronic medium; wherein the embedded algorithm comprises a method of object counting, said method comprising initializing an area of interest, wherein an area where the number of objects that move into/out this area is to be counted is defined; dividing a captured scene of the area of interest into a grid of blocks; determining the average size information of an object for each block of the grid of blocks; learning occlusion rates of motion blobs in the captured scene so that classifiers are optimized based on the occlusion rates of the motion blobs; detecting objects in motion blobs in the captured scene; tracking the detected objects in the motion blobs so as to give the detected moving blobs a consistent label throughout the appearance of the blobs in the scene ; and counting objects in the tracked motion blobs by analyzing the tracked motion blobs based on their occlusion rates, where the occlusion rates of the tracked motion blobs is determined with the optimized classifier; whereby upon finishing counting will the number of objects be output to a displaying device.

[0012] The objectives and advantages of the invention will become apparent from the following detailed description of preferred embodiments thereof in connection with the accompanying drawings.

Brief Description of the Drawings [0013] Preferred embodiments according to the present invention will now be described with reference to the Figures, in which like reference numerals denote like elements.

[0014] FIG 1 is a flowchart illustrating the method of object counting in accordance with one embodiment of the present invention.

[0015] FIG 2 is a flowchart of learning the occlusion rate in accordance with one embodiment of the present invention.

[0016] FIG 3 illustrates the classes of motion blobs with different arrangement types and occlusion rates.

[0017] FIG 4 is a flowchart of determining classes of motion blobs that have different occlusion rate and arrangement type 41 in accordance with one embodiment of the present invention.

[00 8] FIG 5 is a flowchart of determining arrangement type in accordance with one embodiment of the present invention.

[0019] FIG 6 is a flowchart of determining the occlusion rate for each arrangement type in accordance with one embodiment of the present invention.

[0020] FIG 7 is a flowchart showing the determination of the occlusion rate based on the overlapping area ratio as compared to predefined threshold values.

[0021] FIG 8 is a flowchart of extracting features for each determined class of motion blob in accordance with one embodiment of the present invention.

[0022] FIG 9 depicts the two features that represent the foreground and background information of the motion blob. [0023] FIG 10 is a flowchart of optimizing classifier in accordance with one embodiment of the present invention.

[0024] FIG 1 1 is a flowchart illustrating the process of counting objects in motion blobs in accordance with one embodiment of the present invention.

[0025] FIG 12 is a flowchart illustrating the process of counting the number of objects in the motion blob based on occlusion rate in accordance with one embodiment of the present invention.

[0026] FIG 13 is a flowchart illustrating the process of updating the average size information in accordance with one embodiment of the present invention.

Detailed Description of the Invention

[0027] The present invention may be understood more readily by reference to the following detailed description of certain embodiments of the invention.

[0028] Throughout this application, where publications are referenced, the disclosures of these publications are hereby incorporated by reference, in their entireties, into this application in order to more fully describe the state of art to which this invention pertains.

[0029] One aspect of the present invention provides a method for determining the number of objects in the motion blobs where the objects are overlapping with each other during counting process. The method of determining number of object of the present invention provides a method to learn the average size information of the object based on the occlusion rate. This is done at the online mode.

[0030] Now referring to FIG 1, there is provided a flowchart illustrating the method of object counting in accordance with one embodiment of the present invention. The method is performed by a computer-executable electronic medium such as microprocessors, computers, PDA, or the like. The method of object counting is used to determine the number of objects in a monitored area, where the objects in the monitored area are captured by any suitable capturing devices such as camera. Briefly, the method of object counting comprises the operations of registration and online counting, where the registration comprises initializing area of interest 10, dividing the captured scene of the area of interest into grid 20, determining the average size information for an object 30, and learning the occlusion rate 40; when the registration is completed 50, the online counting commences, where the online counting comprises detecting objects 60, tracking objects 70, and counting objects 80. Upon finishing counting will the number of objects be output to any suitable displaying devices.

[0031] The step of initializing area of interest 10 defines an area where the number of objects that move into/out this area will be counted. The area of interest may be formed by a region or line.

[0032] The step of dividing the captured scene of the area of interest into grid 20 divides the captured scene into a grid of blocks with fixed or non-fixed dimension.

[0033] The step of determining the average size information for an object 30 determines the average size information of an object for each individual blocks, where the average size information is recorded for block grid that has been travelled by the object during registration. The centroid location of the object indicates the individual block grid where the average size information of the particular object is calculated. For any individual block grid where the average size information of an object is not calculated as the object does not travel to that block, the average size information of an object is automatically estimated based on the average size information of other individual block grids that have been updated during the travel of the object. The estimation can be done using any known interpolation methods including nearest neighborhood, bilinear and bicubic.

[0034] The step of learning the occlusion rate 40 learns the occlusion rate for detected motion blobs. Briefly, based on the collected samples of motion blobs with different occlusion rates, classifiers are optimized to learn the patterns of the motion blobs to indicate various occlusion rates.

[0035] Now referring to FIG 2, there is provided a flowchart of learning the occlusion rate 40 in accordance with one embodiment of the present invention. As shown in FIG 2, the step of learning the occlusion rate 40 comprises determining classes of motion blobs that have different occlusion rate and arrangement type 41, extracting features for each determined class of motion blob 42, and optimizing classifier with the extracted features 43.

[0036] The step of determining classes of motion blobs that have different occlusion rate and arrangement type 41 determines the classes of motion blobs based on occlusion rates and arrangement types. These classes of motion blobs are important as a source to the optimization of classifier. The arrangement types can be flexibly designated considering the counting accuracy and computing complexity. One exemplary designation of the arrangement types includes vertical, horizontal and diagonal. The same for the occlusion rates; they can also be flexibly categorized. One exemplary categorization of the occlusion rates includes three rates: 0%, 25% and 50% occlusion rates. It is apparent that the designation of arrangement types and the categorization of occlusion rates are used to reduce the computing complexity without significantly compromising the counting accuracy. FIG 3 illustrates the classes of motion blobs with different arrangement types (only vertical and horizontal arrangement types shown) and occlusion rates (only 0%, 25% and 50% occlusion rates shown).

[0037] Now referring to FIG 4, there is provided a flowchart of determining classes of motion blobs that have different occlusion rate and arrangement type 41 in accordance with one embodiment of the present invention. Briefly, the classes of motion blobs are determined based on their arrangement types and occlusion rates, where the arrangement types are determined by analyzing the dimension of the motion blobs 411, and the occlusion rate for each arrangement type is determined by analyzing the overlapping ratio of the objects in the motion blobs 412.

[0038] Now referring to FIG 5, there is provided a flowchart of determining arrangement type 411 in accordance with one embodiment of the present invention. The dimension of a motion blob is defined by its height and width, and a standard height and width for a motion blob is predefined for determining the arrangement type. To determine the arrangement type of a motion blob, its height and width are analyzed, and then the height and width of the motion blob are compared with the standard height and width to determine the arrangement type of the motion blob. FIG 5 shows the determination of arrangement types of motion blobs with three arrangement types including vertical, horizontal and diagonal. First, a decision is made as to whether the height of the motion blob is larger than the standard height and the width of the motion blob is less than or equal to the standard width 4111; if yes, the motion blob is assigned as a type of vertical arrangement 4112. If no in step 4111, a further decision is made as to whether the width of the motion blob is larger than the standard width and the height of the motion blob is less than or equal to the standard height 4113; if yes, the motion blob is assigned as a type of horizontal arrangement 4114; if no, the motion blob is assigned as a type of diagonal arrangement 4115.

[0039] Now referring to FIG 6, there is provided a flowchart of determining the occlusion rate for each arrangement type 412 in accordance with one embodiment of the present invention. First, the boundary of each of the object in the motion blob is determined 4121; then calculates the overlapping area ratio between the boundaries of each of the objects and the area of the motion blob 4122; in other words, the ratio between the overlapping areas between each objects within the motion blob to the area of the motion blob; and finally the occlusion rate is determined based on the overlapping area ratio as compared to predefined threshold values 4123.

[0040] Now referring to FIG 7, there is provided a flowchart showing the determination of the occlusion rate based on the overlapping area ratio as compared to predefined threshold values. In practice, the occlusion rates and predefined threshold values can be flexibly designated. For illustration, two threshold values (71=10% and T2=50%) and three occlusion rates (0%, 25% and 50%) are used herein. The determination of the occlusion rates starts with a decision as to whether the overlapping area ratio is less than Tl 41231; if yes, the occlusion rate for the motion blob is assigned as 0% 41232. If no in step 41231, a further decision is made as to whether the overlapping area ratio is less than T2 41233; if yes, the occlusion rate of the motion blob is assigned as 25% 41234; and if no, the occlusion rate of the motion blob is assigned as 50% 41235. Furthermore, the assigned occlusion rate is just a reference value; it can be other suitable values.

[0041] The step of extracting features for each determined class of motion blob 42 extracts the features from each image in all the determined classes of motion blobs. The extracted features encompass the information of the foreground and background of the blob image.

[0042] Now referring to FIG 8, there is provided a flowchart of extracting features for each determined class of motion blob 42 in accordance with one embodiment of the present invention. First, the silhouette of the objects area is extracted as the foreground information 421. Second, the silhouette of the non-objects area is extracted as the background information 422. Third, extracted are the features including edge information, texture and shape information from the motion blob, which represent the internal information of the objects area 423. Finally, the extracted features are stored in the storage medium 424. FIG 9 depicts the two features that represent the foreground and background information of the motion blob.

[0043] With the availability of the classes of motion blobs and the extracted features, the classifier is optimized 43. The optimization can be done with the known optimization methods including backpropagation.

[0044] Now referring to FIG 10, there is provided a flowchart of optimizing classifier 43 in accordance with one embodiment of the present invention. The optimization first extracts a set of features from the storage medium 431. The second step involves updating the weights of the classifier base on the feature values 432. A decision step is to decide whether the extraction index reaches the end of the storage medium 433; if no, the extraction index is increased 434, and the operation goes back to step 431 until all the set of features are extracted. If yes in step 433, the fitness value of the classifier is calculated 435. Then, a further decision step is to decide whether the fitness value is larger than a predefined threshold value 436; if yes, the classifier is computed base on the updated weights 438, and the method proceeds to whether the registration is complete. If no in step 436, the extraction index is re-initialized to the beginning of the storage medium 437 and the process goes back to step 431 until the fitness value is larger than a predefined threshold value.

[0045] The step of detecting objects 60 detects the moving blobs in the scene using for example the known foreground detection techniques including frame differencing method.

[0046] The step of tracking objects 70 gives the detected moving blobs a consistent label throughout the appearance of the blobs in the scene.

[0047] The step of counting objects 80 calculates the number of objects by analyzing the tracked motion blobs based on the calculated occlusion rate. The occlusion rate of the tracked motion blobs is determined from the motion blob with the optimized classifier.

[0048] Now referring to FIG 1 1 , there is provided a flowchart illustrating the process of counting objects in motion blobs 80 in accordance with one embodiment of the present invention. In the present invention, the number of objects is determined by the bounding box information of the motion blobs. The process of counting objects in motion blobs 80 first extracts the bounding box information of the tracked motion blob 81; then decide on whether the bounding box information is equal to the noise ratio 82; if yes, the number of objects for the motion blob image is assigned to zero 83. If no in step 82, a further decision is made as to whether the bounding box information is equivalent to a single object 84; if yes, the number of objects for the motion blob is assigned to one 85. If no in step 84, a yet another decision step is made to as to whether the bounding box information is larger than a single object 86; if yes, the number of count for the motion blob is determined based on the occlusion rate 87; If no, number of count is set to zero 89. The process of counting the number of objects in the motion blob based on occlusion rate is described in detail hereinbelow.

[0049] Now referring to FIG 12, there is provided a flowchart illustrating the process of counting the number of objects in the motion blob based on occlusion rate 87 in accordance with one embodiment of the present invention. First, the features of the motion blob which are similar to the ones used in the process of registration are extracted 871. With the extracted features, the optimized classifier is selected and subsequently the value of the occlusion rate is determined 872. The average size information of the object that is determined during the registration process is updated as a new value based on the occlusion rate 873. The detail of the update is highlighted in FIG 9. With the updated average size information, the number of object from the blob image is determined by dividing the blob image accordingly 874.

[0050] Now referring to FIG 13, there is provided a flowchart illustrating the process of updating the average size information 873 in accordance with one embodiment of the present invention. In the present invention, the determined occlusion rate value is evaluated. If the occlusion rate is equivalent to 25 8731, then the average size information of the object is reduced by 25% 8732. If the occlusion rate value is equivalent to 50 8734, then the average size information of the object is reduced by 50% 8735.

[0051 J The present invention also provides an object counting system, where the system comprises an image capturing device for capturing images of a monitored area, an electronic processor electronically connected to the capturing device for processing the images and performing the object counting using the method described above, and a displaying device electronically connected to the electronic processor for displaying the results of object counting. The capturing device can be any suitable ones including cameras. The electronic processor can be any suitable ones as long as it has the computer- executable medium for storing data and embedding algorithm for performing the method. The displaying device can be any suitable ones including flat screens.

[0052] While the present invention has been described with reference to particular embodiments, it will be understood that the embodiments are illustrative and that the invention scope is not so limited. Alternative embodiments of the present invention will become apparent to those having ordinary skill in the ait to which the present invention pertains. Such alternate embodiments are considered to be encompassed within the scope of the present invention. Accordingly, the scope of the present invention is defined by the appended claims and is supported by the foregoing description.