Login| Sign Up| Help| Contact|

Patent Searching and Data


Title:
CAMERA TRACKING METHOD
Document Type and Number:
WIPO Patent Application WO/2022/146389
Kind Code:
A1
Abstract:
The invention relates to a camera tracking method providing human being detection, tracking, human activity detection and prediction, emergency/abnormal case detection from cameras in all areas where cameras can be installed. Invention particularly relates to a method providing automatic recognition of video or stream contents shot by user instantly by use of artificial intelligence algorithm on IoT - edge, cloud or on-premises server, instant notification of detected abnormalities instantly and automatically by SMS - call and e-mail and analysing thereof by use of specified parameters and periodically automatic reporting thereof.

Inventors:
TEMUR NAZLI (TR)
Application Number:
PCT/TR2021/051572
Publication Date:
July 07, 2022
Filing Date:
December 29, 2021
Export Citation:
Click for automatic bibliography generation   Help
Assignee:
XENA VISION YAZILIM SAVUNMA ANONIM SIRKETI (TR)
International Classes:
G06T7/00; G06K9/62; G06N20/00; G06T7/20; G06V40/10; G08B21/02; H04N5/00
Foreign References:
CN109819208A2019-05-28
US20200349241A12020-11-05
US20090222388A12009-09-03
US20070121999A12007-05-31
US20200349347A12020-11-05
US20140372348A12014-12-18
US20110235925A12011-09-29
Attorney, Agent or Firm:
DESTEK PATENT, INC. (TR)
Download PDF:
Claims:
CLAIMS A camera tracking method providing human being detection, tracking, human activity detection and prediction, emergency/abnormal case detection from cameras (7) in all areas where cameras (7) can be installed and characterized by comprising process steps of;

• Receiving videos as input from cameras (7) via wireless/wire communication means (1001 ),

• Receiving live stream images from cameras (7) via wireless/wire communication means (1002),

• Arrangement of inputs received in steps 1002 or 1001 as frame by image arrangement unit (2) (edge),

• Logging images read in process step 1003 by image arrangement unit (2) with IP and camera number to be given into web software (12) as input containing information of the camera (7) from which read image is received (1004),

• Assignment of addresses in a manner enabling receiving videos from any camera (7) via camera connection unit (6) under process steps of 1001 or 1002 as input simultaneously and receiving via wireless/wire communication means (1005),

• Assignment of addresses in a manner enabling receiving any live streams from cameras (7) via camera connection unit (6) under process steps of 1001 or 1002 as input simultaneously and receiving via wireless/wire communication means (1006),

• Reading of input received under process steps 1002 or 1001 as frame by server (1 ) (1007),

• Logging images read in process step 1007 by server (1 ) with IP and camera number to be given into Web Software (12) as input containing information of the camera (7) from which read image is received (1008),

• Receiving any videos from cameras (7) as input simultaneously with process steps of 1001 , 1002, 1005 or 1007 via wireless/wire communication means (1009),

• Receiving any live streams from cameras (7) as input simultaneously with process steps of 1001 , 1002, 1005 or 1007 via wireless/wire communication means (1010),

• Transmission of input from process steps of 1009 or 1010 to cloud server (3) and reading of data by said cloud server (3) (1011 ),

• Logging images read in process steps 1010 or 1009 by cloud server (3) with IP and camera number to be given into Web Software (12) as input containing information of the camera (7) from which read image is received (1012),

9 • Sequencing in mixed order the image from cameras (7) in a given number by web software (12) in multithreaded form in a manner to be processed separately for each camera (7) (1013),

• Conduct of human detection by deep learning in each sequenced image (1014),

• In case images contain sensitive data for people detected in process step 1014, face detection by deep learning, bed detection and blur, noise and masking with threshold value which is parameter selected by server (1 ) conducting first reading, image arrangement unit (2) and cloud server (3) for the sections (1015),

• In case of detection of human in images, skipping to process step 1017, otherwise, skipping to the very beginning and running either one of steps 1001 ,1002,1005,1007,1009,1011 (1016),

• Recording data in restricting frame in picture format when in masked form in process step 1015 (1017),

• Initiation of human monitoring for image coming from each camera (7) by deep learning (1018),

• In case of failure in human tracking, checking of other people by process step 1016, starting of running of process steps 1020 and 1025 at the same time (1019),

• Comparison of tracked person to other persons and detection of other camera views where s/he is (1020),

• Estimation of route detection of tracked person (1021),

• In case frame data of the person tracked in process step 1020 match other cameras (7), combining route data in each camera with rotes in other cameras on basis of camera convergence and visualization thereof (1022),

• Grouping route details of tracked person in each camera (1023),

• If found, reporting different walking patterns and all routes (1024),

• In case of human tracking, estimation of staying of person (1025),

• Controlling whether or not staying exists, if not discovered, continuation of tracking human (1026),

• In case of discovery of staying, starting track of staying for each tracked person (1027),

• Controlling whether or not stay track is successful (1028),

• Testing whether or not staying is violence for each discovered staying (1029) ,

• Decision on whether or not there is abnormality, only in case of estimation exceeds a certain percentage rate of safe rate (1030),

• In case of detection of safety rate above a certain percentage and classification as normal, marking it as normal in image (1031), • In case of classification of safe above a certain percentage of and abnormal, re-estimation of abnormality by controlling a specified past time period (1032),

• Estimation of staying change rate and acceleration between previous pose and current pose for each frame as long as human pose tracking can be made (1033),

• Testing pose coordinates in sequenced form and with speed and acceleration rates coming in mixing form (1034)

• Assessment of whether or not abnormal (1035),

• Checking whether abnormality probability value is abnormal or normal (1036),

• In case abnormality probability value is above a specified threshold value, assessment as abnormal (1037),

• In case normality probability value is above a specified threshold value, assessment as normal (1038),

• Comparison of value obtained in process step 1037 to value obtained in process step 1035 for final assessment (1039),

• Assessment of two outputs (1040),

• In case both outputs are labelled abnormal, production of alarm by algorithm software (15), otherwise, realization of process step 1031 (1041), and/or

• Automatic sending of SMS by GSM module (4) (1042), and/or

• Automatic mail sending by server (1) and cloud server (3) (1043), and/or

• Automatic emergency call by GSM module (4) (1044), and/or

• Conduct of reporting and current and past display on monitor (5) (1045) and/or

• Control by algorithm software (13) whether or not a specified time is exceeded) (1046) and/or

• In case of exceeding specified hour sending report by mail (1047). The method according to claim 1 and characterized by said safety rate in process steps 1030, 1031 , 1032 being 50 per cent. The method according to claim 1 and characterized by Bayesian hmm being activated for past 3 sec, 10 fps in process step 1032 and estimation whether or not causal abnormal being made. The method according to claim 1 and characterized by LSTM (Long short-term memory) being used in process step 1034.

11

5. The method according to claim 1 and characterized by Hidden Markov Model (HMM) being used in process step 1035.

6. The method according to claim 1 and characterized by LSTM output being used for last assessment in process step 1037.

7. The method according to claim 1 and characterized by Bayesian HMM result being compared to LSTM output in process step 1039. 8. The method according to claim 1 and characterized by specified hour in process steps

1046 and 1047 being 12 hours.

9. The method according to claim 1 and characterized by said cameras (7) being IP Camera (8), Phone Camera (9), USB Camera (10) and/or CCTV Camera (11).

12

Description:
Camera Tracking method

The Field of the Invention

The invention relates to a camera tracking method providing human being detection, tracking, human activity recognition and prediction, emergency/abnormal event/activity detection from multiple cameras in all areas where cameras can be installed/placed.

The invention particularly relates to a method providing automatic real-time recognition of video or stream contents captured by multiple cameras instantly by use of artificial intelligence algorithm on iot - edge, cloud or on-premise server, providing instant notification of detected abnormalities instantly and automatically via SMS - call -alarm and e-mail and analysing thereof by use of specified parameters and providing periodically automatic reporting thereof.

Background of the Invention

Today several cameras are used in various domain areas. Some of the cameras are for security reasons while some are used in mobile devices and various devices where proof I evidence based video capture is intended. In general, continued watching of video streams that are received from security cameras by specified personnel is performed. However, difficulties in detection and tracking of emergency instances when emergency cases occur are experienced in such systems. Early intervention to abnormal cases saves life. To be enabled to do so, it is essential to be able to detect the emergency-abnormality at the exact point of occurrence. Existing video recording systems are not capable to do so. The number of simultaneous videos/streams that can be watched by security officers examining incidents from monitors is limited and watching thereof at same rate of attention is difficult.

Upon technical searches, the application numbered CN104484574A discloses in its abstract as “Invention discloses training correction system with real time human body motion auditing based on quaternion. System uses depth indexing video collection device used to collect a trainee’s depth video data, a human body motion analysis and processing module used to analyse and process trainee’s main common data, an electro-cardio data collection device used for collection and human body status analysis and processing module used to obtain data such as nervousness degree, tiredness degree and emotional changes of trainee by means of electro-cardio data, heart rate, breathing frequency and heart beat rate, breathing frequency of trainee.

As seen the system relates to a real time human body motion auditing training correction system but does not disclose an embodiment capable to solve above mentioned disadvantages.

As a result, due to above described disadvantages and inadequacy of existing solutions it has been necessary to make development in the related art.

Purpose of the Invention

Differently from the existing related art, the invention aims to disclose an embodiment having different technical features providing a new solution.

Primary purpose of the invention is to provide automatic detection of instant human activities and recognition thereof from multiple camera views of tens of thousands cameras at the same time and achieve result at the same accuracy. Also, video can be taken from IP - Analogue camera or Mobile Devices and can be processed at edge, server or cloud. When instantly processed videos/streams event detection / activity recognition outcomes are associated with cases such as abnormal case / emergencies containing attack - violence, falling, fainting, heart attack, epilepsy crises, an alarm is produced and sent in SMS, call and e-mail format.

System and method disclosed under the invention instantly compute cases such as human detection, human tracking, crowd detection, route detection estimation, entrance-exit detection, different walking patterns for advanced statistical analysis and periodically report them. Each person can be detected in different cameras by use of body view and routes between different cameras can be detected and analytical and statistical reports can be produced. Among the studies, the method applied to detect human activity is a complex activity detection method. The activities detected in this method are the ones that do not have a certain pattern, sequence, number and way of occurrence cannot be defined by human. To detect such activities, following steps are followed respectively.

1 . Detection of person and tracking in sequenced frames,

2. Detection of person’s position, detection as 25 coordinate keypoints.

3. For computing a human activity instance, person’s keypoint positions is training for different number of classes by use of Light GBM. For example, for violence recognition, 370.000 keypoint position combination taken from various activity (for example: violence, nonviolence, average violence) and behaviour pattern signature created.

4. For each person, training by use of LSTM to cover combination of person posture changes in sequence, change angle and acceleration of each key point in such combinations and behaviour signature obtained from previous 3rd step and convergence of number of samples in sequence on video required for discovery of minimum period of activity,

5. Discovery of probability of occurrence of each posture sample based on past realization statistics by use of Bayesian Hidden Markov Model,

6. Production of alarm in case where abnormal status label occurs for both results for suppressing wrong detection by assessment of result obtained in 4 together with probability approach in 5,

7. Assessment of activity for each frame and display on video,

8. In case of abnormal status of activity, production of alarm and sending sms or making emergency call or sending e-mail based on nature of alarm status information-, Here depending on vital risk of abnormal situation and continuation thereof, respectively, e-mail, SMS and emergency call steps will be followed.

System produces reports covering statistical data at certain time intervals and e-mails them. By help of these steps, generalisation of complex human activity not having a certain pattern is made and discovery thereof is achieved.

The structural and characteristics features of the invention and all advantages will be understood better in detailed descriptions with the figures given below and with reference to the figures, and therefore, the assessment should be made taking into account the said figures and detailed explanations.

Brief Description of the Drawings

Figure 1 is an illustrative view of components of method disclosed under the invention. Figure 2 is an illustrative view of components of method disclosed under the invention. Figure 3 is a schematic view of the method of the invention.

The drawings are not necessarily to be scaled and the details not necessary for understanding the present invention might have been neglected. In addition, the components which are equivalent to great extent at least or have equivalent functions at least have been assigned the same number.

Description of Part References

1 . Server

2. Display arrangement unit

3. Cloud Server

4. GSM Module

5. Monitor

6. Camera connection unit

7. Camera

8. IP Camera

9. Phone Camera

10. USB Camera

1 1 . CCTV Camera

12. Web Software

13. Algorithm Software

X. Used for Parallel/simultaneously continuing work steps.

Detailed Description of the Invention

In this detailed description, the preferred embodiments of the invention have been described in a manner not forming any restrictive effect and only for purpose of better understanding of the matter.

Invention relates to a camera tracking and method providing human being detection, tracking, human activity detection and prediction, emergency/abnormal case detection from cameras (7) in all areas where cameras (7) can be installed. In the system of the invention server (1 ) is the main medium where algorithm runs. Human detection, human tracking, abnormal situation and emergency situation detection, alarm production, e-mail sending, report production works are all made at the server (1 ).

Display arrangement unit (2) provides blurring, improvement, if needed encoding/encrypting of sensitive data from images received from cameras and sending them to server (1 ).

Cloud server (3) provides detection of incident/ activities without needing a remote installation to the camera system and without putting a physical device. It provides processing, backup and storing data in cloud. Human detection, human tracking, abnormal case and emergency case detection, alarm production, email sending, report production processes are realized at cloud server (3) alternative to server (1 ).

GSM module (4) provides sending SMS and production of instant notifications in emergency/abnormal activity case. Monitor (5) provides prompt display of captured incident in processed image. Performance of more than one camera connection and power supply is provided by camera connection unit (6). Images are obtained from cameras (7). The cameras (7) can be security camera, IP camera (8), phone camera (9), USB camera (10), CCTV camera (i

Web software (12) provides display of more than one camera image to client by web portal and auditing access data. Algorithm software (13) provides making algorithm functional on image.

Process steps performed by the method of invention are as follows:

• Receiving videos as input from cameras (7) via wireless/wire communication means (1001 ),

• Receiving live stream images from cameras (7) via wireless/wire communication means (1002),

• Arrangement of inputs received in steps 1002 or 1001 as frame by image arrangement unit (2) (edge) (1003),

• Logging images read in process step 1003 by image arrangement unit (2) with IP and camera number to be given into web software (12) as input containing information of the camera (7) from which read image is received (1004), • Assignment of addresses in a manner enabling receiving videos from any camera (7) via camera connection unit (6) under process steps of 1001 or 1002 as input simultaneously and receiving via wireless/wire communication means (1005),

• Assignment of addresses in a manner enabling receiving any live streams from cameras (7) via camera connection unit (6) under process steps of 1001 or 1002 as input simultaneously and receiving via wireless/wire communication means (1006),

• Reading of input received under process steps 1002 or 1001 as frame by server (1 ) (1007),

• Logging images read in process step 1007 by server (1) with IP and camera number to be given into Web Software (12) as input containing information of the camera (7) from which read image is received (1008),

• receiving any videos from cameras (7) as input simultaneously with process steps of 1001 , 1002, 1005 or 1007 via wireless/wire communication means (1009),

• Receiving any live streams from cameras (7) as input simultaneously with process steps of 1001 , 1002, 1005 or 1007 via wireless/wire communication means (1010),

• Transmission of input from process steps of 1009 or 1010 to cloud server (3) and reading of data by said cloud server (3) (1011 ),

• Logging images read in process steps 1010 or 1009 by cloud server (3) with IP and camera number to be given into Web Software (12) as input containing information of the camera (7) from which read image is received (1012),

• Sequencing in mixed order the image from cameras (7) in a given number by web software (12) in multithreaded form in a manner to be processed separately for each camera (7) (1013),

• Conduct of human detection by deep learning in each sequenced image (1014),

• In case images contain sensitive data for people detected in process step 1014, face detection by deep learning, bed detection and blur, noise and masking with threshold value which is parameter selected by server (1 ) conducting first reading, image arrangement unit (2) and cloud server (3) for the sections (1015),

• In case of detection of human in images, skipping to process step 1017, otherwise, skipping to the very beginning (running either one of steps 1001 , 1002, 1005, 107, 1009, 1011) (1016),

• Recording data in restricting frame in picture format when in masked form in process step 1015 (1017),

• Initiation of human monitoring for image coming from each camera (7) by deep learning (1018), • In case of failure in human tracking, checking of other people by process step 1016, starting of running of process steps 1020 and 1025 at the same time (1019),

• Comparison of tracked persons to other persons and detection of other camera images where they are (1020), (Comparison is performed by histogram comparison and short term (10 FPS 10 sec.) multi-class appearance learning.)

• Conduct of route estimation of tracked person (1021 ) (route estimation is made with reference to centroid of past 10 FPS 10sec initial trajectory data is conducted. It is seated onto a curve by help of trajectory polynomial regression created by the centroid coordinates and thus route is estimated.)

• In case frame data of the person tracked in process step 1020 match other cameras (7), combining route data in each camera with rotes in other cameras on basis of camera convergence and visualization thereof (1022),

• Grouping rote data of persons tracked in each camera (1023) (grouping is made on basis of variance and curve shape, and different walking patterns are discovered by route variance and curve fitting.)

• If found, reporting different walking patterns and all routes (1024),

• In case of tracking people, estimation of posture (1025), (pose is computed by use of pose estimation algorithm for the person.)

• Controlling whether or not staying exists, if not discovered, continuation of tracking human (1026),

• In case of discovery of staying, starting track of staying for each tracked person (1027),

• Controlling whether or not stay track is successful (1028),

• Testing whether or not staying is violence for each discovered staying (1029) (provided with Light GBM model),

• Decision on whether or not there is abnormality, only in case of estimation exceeds a certain percentage rate of safe rate (preferably 50%) (1030),

• In case of detection of safety rate above a certain percentage (preferably 50%) and classification as normal, marking it as normal in image (1031 ) (marked-logged as normal for use in estimation at Bayesian HMM.)

• In case of classification of safe above a certain percentage (preferably 50%) of and abnormal, re-estimation of abnormality by controlling a specified past time period (1032), (Bayesian hmm is activated for past 3 sec, 10 fps and computing whether or not abnormal casually is conducted.)

• Estimation of staying change rate and acceleration between previous pose and current pose for each frame as long as human pose tracking can be made (1033), • Testing pose coordinates in sequenced form and with speed and acceleration rates coming in mixing form (1034) by means of (LSTM (Long short-term memory),

• Assessment of whether or not abnormal (1035) by (Hidden Markov Model (HMM)

• Checking whether abnormality probability value is abnormal or normal (1036),

• In case abnormality probability rate is above a specified threshold value, assessment as abnormal (1037) (for final assessment, it is compared to LSTM output.),

• In case normality probability value is above a specified threshold value, assessment as normal (1038),

• Comparison of value obtained in process step 1037 to value obtained in process step 1035 for final assessment (1039) (in other words, Bayesian HMM result is compared to LSTM output.)

• Assessment of two outputs (1040),

• In case both outputs are labelled abnormal, production of alarm by algorithm software (15), otherwise, realization of process step 1031 (1041 ), and/or

• Automatic sending of SMS by GSM module (4) (1042), and/or

• Automatic mail sending by server (1 ) and cloud server (3) (1043), and/or

• Automatic emergency call by GSM module (4) (1044), and/or

• Conduct of reporting and current and past display on monitor (5) (1045) and/or

• Control by algorithm software (13) whether or not a specified time (preferably 12 hours) is exceeded) (1046) and/or

• In case of exceeding specified hour (preferably 12 hours) sending report by mail (1047).