FIELD OF INVENTION

The present invention relates to a system and method for monitoring human behaviour. More particularly, the present invention relates to a system and method for monitoring human behaviour based on video surveillance.

BACKGROUND OF THE INVENTION

Nowadays, many confined areas are used to isolate people such as detention cells, prisons, and old folks’ centres. Sometimes, certain people are isolated in a confined area that surrounded by a grill against their will and causes them to act aggressively. Conventionally, a number of surveillance cameras are installed in these confined areas to monitor the behaviour of the occupant so that any aggressive behaviour can be detected. Normally, the confined area has a poor lighting condition which may affect the monitoring process due to low-quality images captured by the installed camera.

An example of a system and method for monitoring human behaviour is disclosed in a European Patent Application No. EP 1459272 A1. The patent relates to a system for detecting suspicious behaviour. The system observes the behaviour of a monitored subject by using captured images from an observation unit, preferably a video camera. The system also includes infrared heat profiles of persons, images of actual people, and behaviours of persons. The system then recognises whether the observed behaviour is associated with suspicious behaviour. The observation of motion related to behaviour is compared against a database of behavioural patterns in a pattern recognition module.

Another example of a method for monitoring human behaviour is disclosed in a China Patent No. CN 107194317 A. The patent relates to a method for detecting violence behaviour. The method comprises the steps of obtaining picture frames in a video streaming, detecting human bodies in the picture frames and framing out the detected human bodies. The method also carries out clustering analysis on an optical flow feature points to obtain moving objects in the picture frames. Furthermore, the human bodies detection is carried out to determine whether violence incidents occur or not. Although there are many systems and methods for monitoring human behaviour, most of the systems and methods do not monitor a specific area which differentiates the outside and inside of the confined area. This condition may lead to misrepresentation for determining whether the object is an outsider or insider. In addition, the images obtained from the installed cameras may be substandard when low lighting condition occurs. Consequently, the systems and methods cannot properly analyse the images due to some deficiency such as blurry image or low contrast image. Therefore, there is a need for a system and method for monitoring human behaviour which addresses the mentioned problems.

SUMMARY OF INVENTION

A system (100) for monitoring human behaviour is provided in the present invention. The system (100) comprises an image processing module (103) which includes an object detection unit (502) configured to detect any moving object in the image frames, an event detection unit (504) configured to detect at least one predefined event in the image frames, a lighting adaptation unit (501) configured to detect low lighting condition in the image frames, and an object tracking unit (503) configured to perform object tracking in the image frames. The lighting adaptation unit (501) is connected to the object detection unit (502) while the object tracking unit (503) is connected to the object detection unit (502) and the event detection unit (504).

Preferably, the system (100) further comprises a video acquisition module (102) configured for acquiring video from at least one camera and converting the video into a plurality of image frames.

Preferably, the system (100) further comprises a monitoring module (104) configured for displaying where the detected event occurs and alerting an operator if any aggressive behaviour is detected.

A method for monitoring human behaviour is also provided in the present invention. The method includes the steps of adapting a lighting condition in a plurality of image frames by determining whether there is a low light condition by a lighting adaptation unit (501), detecting a moving object in the plurality of image frames by an object detection unit (502), and tracking the detected object in the plurality of image frames and determining whether the detected object is an outsider or insider of a confined area by an object tracking unit (503).

Preferably, prior to adapting the lighting condition in the image frames includes a step of converting a video acquired from at least one camera into a sequence of image frames by a video acquisition module (102).

Preferably, the method further includes a step of displaying the detected area and alerting an operator if any aggressive behaviour is detected by the monitoring module (104).

Preferably, the step of adapting the lighting condition in the plurality of image frames includes detecting a lighting condition in the plurality of image frames, accumulating number of image frames where a low light condition is detected, lowering a threshold of the image frames to cater a low contrast condition in the low lighting image frames and normalising the plurality of image frames to maximise the contrast between the object and the background of the image frames.

Preferably, the step of detecting lighting condition in the plurality of image frames includes converting the image frames to a hue-saturation-value colour model, extracting hue channel image for each image frame based on a hue channel information, quantising the hue channel information to n number of bins, constructing a hue channel histogram, computing a p-tile thresholding to the histogram to obtain a threshold value and confirming status of the lighting condition of the image frames based on the threshold value.

Preferably, the step of tracking detected moving object in the image frames includes performing object tracking in one of the plurality of image frames, analysing whether the object is a new or a previous tracked object, computing an object area overlapping with a grill area if the object is new, setting the object as an outsider if the object area overlapped 100% with the grill area (20) or setting the object as an insider if the object area overlapped with the grill area (20) is less than 100%, checking the status of the object whether the object is an outsider if the analysed object is not new, computing the object area overlapping with a non-grill area (30) if the object is an outsider, accumulating number of frames that fulfils the overlapping criteria if the object is an outsider and if the object area overlapped with the non-grill area (30) is more than 50%, and setting the object as an insider if the number of fulfilled frames is more than f frames, wherein f \s a predefined number.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

FIG. 1 illustrates a block diagram of a system (100) for monitoring human behaviour according to an embodiment of the present invention.

FIG. 2 illustrates a flowchart of a method for monitoring human behaviour according to an embodiment of the present invention.

FIG. 3 illustrates a flowchart of sub-steps for adapting lighting condition according to the method shown in FIG. 2.

FIG. 4 illustrates a flowchart of sub-steps for detecting the lighting mode in the image frames according to the method shown in FIG. 3.

FIG. 5 illustrates a flowchart of sub-steps for tracking the detected moving object in the image frames according to the method shown in FIG. 2.

FIG. 6 (a) illustrates an example of a blob area of detected object in an image frame according to an embodiment of the present invention.

FIG. 6 (b) illustrates a grill area in an image frame according to an embodiment of the present invention.

FIG. 6 (c) illustrates a non-grill area in an image frame according to an embodiment of the present invention.

FIG. 6 (d) illustrates an overlapped area between the blob area and grill area in an image frame according to an embodiment of the present invention. DESCRIPTION OF THE PREFERRED EMBODIMENT

A preferred embodiment of the present invention will be described hereinbelow with reference to the accompanying drawings. In the following description, well-known functions or constructions are not described in detail since they would obscure the description with unnecessary detail.

Initial reference is made to FIG. 1 which illustrates a block diagram of a system (100) for monitoring human behaviour according to an embodiment of the present invention. The system (100) analyses a sequence of images to monitor human behaviour in an area of interest of a confined area that is surrounded by a grill. The human behaviour refers to at least one predefined event detected in image frames. Preferably, the camera is installed in one end of the grill area (20) to allow the system (100) to monitor inside and outside of a grill area (20). Additionally, the system (100) determines whether a moving object detected in the image frames is an outsider or insider. The insider refers to an occupant within the confined area such as a detainee while the outsider refers to a person outside the confined area such as a warden or a guard.

The system (100) comprises a video acquisition module (102), an image processing module (103), and a monitoring module (104). The system (100) is connected to at least one camera to acquire a stream of video from the camera, wherein the camera is installed in the confined area such as detention cell or prison.

Although it is described herein that one camera is installed in the confined area, it may be understood that there may be a plurality of cameras installed in the confined area. Although it is described herein that the system (100) monitors human behaviour in one confined area, it may be understood that the system (100) may monitor human behaviour in a plurality of confined area simultaneously.

The video acquisition module (102) is connected to the camera to acquire video from the camera. The video acquisition module (102) is configured to convert the video into a plurality of image frames. The video acquisition module (102) is further connected to the image processing module (103) to send the image frames to the image processing module (103) for detecting process. The image processing module (103) comprises a lighting adaptation unit (501), an object detection unit (502), an object tracking unit (503), and an event detection unit

(504).

The lighting adaptation unit (501) is configured to detect a low lighting condition in the image frames. The lighting adaptation unit (501) has an ability to maximise the visibility of any moving object in the image frames during different lighting condition, either day or night mode by changing contrast between an object and background of the object detection. The lighting adaptation unit (501) is connected to the object detection unit (502).

The object detection unit (502) is configured to detect any moving object in the image frames, whereby only moving object is analysed further for event detection. The object detection unit (502) is connected to the lighting adaptation unit (501) so that the moving object is able to be detected in different lighting condition. In particular, the lighting condition of the image frames obtained is utilised to adjust a threshold used for differentiating the moving object from the background. The object detection unit (502) is connected to the lighting adaptation unit (501) and the object tracking unit (503).

The object tracking unit (503) is configured to perform object tracking in the image frames, whereby the same object is assigned as a tracker label across subsequent image frames. The object tracking unit (503) is also configured to differentiate between the outsider and the insider based on the grill that separates the outside and inside of the confined area. The object tracking unit (503) is connected to the object detection unit (502) and the event detection unit (504).

The event detection unit (504) is configured to detect at least one predefined event in the image frames. The predefined event refers to a predetermined event such as climbing, loitering, tampering, and aggressive event. The event detection unit (504) performs the event detection based on a tracking information obtained from the object tracking unit (503), wherein the tracking information includes object states such as object location, object label, object trajectories in consecutive frames, and object appearance. The monitoring module (104) is connected to the image processing module (103). The monitoring module (104) is configured to display where the detected event occurs around the grill area (20). Additionally, the monitoring module (104) is configured to alert an operator if any aggressive behaviour is detected, wherein the operator is a person responsible to operate and monitorthe system (100). The operator may also be a guard or warden of the confined area.

FIG. 2 illustrates a flowchart of a method for monitoring human behaviour according to an embodiment of the present invention. Initially, the video acquisition module (102) acquires the video from the camera and converts the video into a sequence of image frames as in step 1000.

Once the video is converted into the sequence of image frames, the lighting adaptation unit (501) adapts the lighting condition in each image frame by determining whether there is a low lighting condition as in step 2000. If a low lighting condition is detected, the lighting adaptation unit (501) changes the contrast between the object and the background to ensure that any object is able to be monitored under the low lighting condition. This is because, in the low lighting condition, the image frames appear in grayscale, thus, the existence of the object cannot be seen clearly. The substeps for adapting the lighting condition in the image frames are further explained in relation to FIG. 3.

After the lighting condition is adapted, the object detection unit (502) detects the moving object in the image frames as in step 3000. The object detection process able to adapt to the different lighting condition and different image view in the day and night mode by utilising background subtraction technique. The background subtraction technique calculates the foreground mask, hence, separating the moving object and the background. The moving object is differentiated from the background by analysing the changes in each image pixel.

In step 4000, the object tracking unit (503) performs object tracking and determines whether the moving object is an outsider or insider. The tracker label assigned to each of the objects is utilised to differentiate between the outsider and the insider. If the detected object is an outsider, the object is not further analysed. The sub- steps for tracking the detected object in the image frames are further explained in relation to FIG. 5.

Once the object tracking is performed, the event detection unit (504) detects a predefined event in the image frames based on the tracking information obtained from the object tracking unit (503) as in step 5000.

Finally, the monitoring module (104) displays the area where the detected event occurs and alerts the operator if any aggressive behaviour is detected as in step 6000.

Reference is now made to FIG. 3 which illustrates a flowchart of the sub-steps for adapting the lighting condition in image frames as in step 2000 of the method of FIG. 2. Initially, the lighting condition in the image frames is detected as in step 2100 whether it is low or not.

If there is no low lighting condition detected, step 3000 of the method of FIG. 2 is proceeded whereby the object detection unit (502) detects the moving object in the image frames as is.

If a low lighting condition is detected in the image frames as in decision 2200, then the night mode is assumed to be activated as in step 2300. Once the night mode is assumed to be activated, a number of frames detected with the low lighting condition are accumulated as in step 2400.

If the number of frames detected with the low lighting condition is less than n number of frames, the step 3000 of the method of FIG. 2 is proceeded whereby the object detection unit (502) detects the moving object in the image frames as is. If the number of frames detected with the low lighting condition is more than n number of frames as in decision 2500, then the night mode is confirmed to be activated. After the night mode is confirmed to be activated, the threshold for object detection is lowered to cater to a low contrast condition in the low lighting image as in step 2600. Thereon, the image frames are normalised to maximise the contrast between the object and background of the image as in step 2700. The step 3000 of the method of FIG. 2 is then proceeded whereby the object detection unit (502) detects the moving object in the normalised image frames.

FIG. 4 illustrates a flowchart of sub-steps for detecting the lighting mode in the image frames as in step 2100 of the method of FIG. 3. Initially, the image frames are converted to a hue-saturation-value, HSV, colour model as in step 2110, wherein the image frames are originally in a red-green-blue, RGB, colour model. Thereon, a hue channel information is analysed. The hue channel information represents colour tint of pixels that has a value ranging from 0 to a certain degree such as 120 for green and 240 for blue. Thereon, a hue channel image is extracted from each image frame based on the hue channel information as in step 2120.

Once the hue channel image is extracted, the hue channel information is quantised into n number of bins as in step 2130. In step 2140, a hue channel histogram is constructed.

In step 2150, a p-tile thresholding is computed to the histogram, wherein the p- tile thresholding assumes the object in the image frames are brighter than the background and occupy a fixed percentage of the image area. Likewise, if low light condition is happening in the frame, dark pixels will occupy most of the image area and the histogram will be concentrated to the left. Hence, the threshold value is obtained based on the computed p-tile thresholding of the histogram by setting p equals to certain percentage (e.g. more than 70%) . If the threshold obtained is less than or equal to dark threshold (e.g. three) as in decision 2160, the image is deemed to be in the low light condition. Otherwise, the status of the low light condition is false or likely to be the day mode.

Reference is now made to FIG. 5 which illustrates a flowchart of the sub-steps for tracking the detected moving object in the image frames as in step 4000 of the method of FIG. 2. Initially, the object tracking is performed in the image frames as in step 4050.

Next, the object is analysed whether the object is new or a previously tracked object as in step 4100. Particularly, in order to distinguish whether the object is new or not, the presence of the object in an image frame is compared to all previously tracked objects based on the tracker labels assigned to the previously tracked objects. If the object in the image frame is similar in appearance to any of the previously tracked objects and/or location of the object is near to any of the previously tracked objects, the object is assumed to be one of the previously tracked objects and the object is indicated as not new.

If the object is new as in decision 4150, object area overlapping with the grill area (20) is computed as in step 4200. Hence, if the object area is overlapped approximately 100% with the grill area (20) as in decision 4250, the object status is set as an outsider and the object is not analysed further. Preferably the overlapping value may range from 80% to 100%. The reason for classifying the outsider as 100% overlapped between the blob area (10) and the grill area (20) is due to the viewpoint of the camera. Each camera is typically located at a top corner of the confined space and thus, each camera captures a top perspective of the interior of the confined area. This causes the outsider to only be captured by the camera in 100% overlapping with the grill area. To detect if the object area is overlapped with the grill area (20), the object is compared against the grill area (20) in the image frames. If the object appears to be within the grill area (20), the overlapping value refers to the number of pixels that represents the grill area (20) and the object location at the same time. FIG. 6 (a) illustrates an example of a blob area (10) that refers to the presence of the object in an image frame, FIG. 6 (b) illustrates an example of the grill area (20) in an image frame, while FIG. 6 (c) illustrates an example of a non-grill area (30) which is a complement of the grill area (20) in an image frame. Referring to FIG. 6 (d), an image frame shows that the blob area (10) is overlapped 100% with the grill area (20) and thereby, the overlapping area between the blob area (10) and the non-grill area (30) is 0%.

If the object area does not overlap 100% with the grill area (20) as in decision 4250, the object status is set as an insider and step 5000 of the method of FIG. 2 is proceeded whereby the event detection unit (504) detects the predefined event in the image frames.

Referring back to decision 4150, if the analysed object is not new, the status of the object is checked whetherthe object is an outsider as in decision 4300. If the object is an insider, the step of 5000 of the method of FIG. 2 is proceeded whereby the event detection unit (504) detects the predefined event in the image frames. If the object is an outsider as in decision 4300, the object area overlapping with the non-grill area (30) is computed as in step 4350. Thereafter, if the object area overlapped with the non-grill area (30) is more than 50% as in decision 4400, the number of frames that fulfil the overlapping criteria is accumulated as in step 4450. If the number of fulfilled frames is more than f frames as in decision 4500, the object is set as an insider and step 5000 of the method of FIG. 2 is proceeded whereby the event detection unit (504) detects a predefined event in the image frames, wherein f \s a predefined number. However, if the fulfilled frames are less than f frames, then step 5000 of the method of FIG. 2 is proceeded whereby the event detection unit (504) detects the predefined event in the image frames.

Referring back to decision 4400, if the object area overlapped with the non-grill area (30) is less than 50%, the method ends. Thus, the image frames would not proceed to the next step where the image frames that fulfil the overlapping criteria are accumulated.

While embodiments of the invention have been illustrated and described, it is not intended that these embodiments illustrate and describe all possible forms of the invention. Rather, the words used in the specifications are words of description rather than limitation and various changes may be made without departing from the scope of the invention.