[0001] An artificial intelligence and machine vision-based drone to assess relief and rescue conditions in disasters

Technical Field

[0002] The technical field of this invention relates to the equipment of the crisis situation analysis and life detector systems in natural and anthropogenic disasters (floods, earthquakes, wars, etc.) where drones are used to search for the missing and the injured people.

Background Art

[0003] A patent titled INDOOR DRONE HAVING FUNCTIONS FOR

DECOMPOSING FINE DUST, EXTINGUISHING FIRE, DRIVING THIEF OUT, AND PROTECTING ELDERLY LIVING ALONE AND METHOD OF FLYING DRONE has been registered in South Korea, which explanation of its function is as follows:

[0004] The present invention relates to an indoor drone having a function for decomposing fine dust and a method of flying the drone, and more specifically, to an air-purifying drone having the functions for decomposing fine dust, extinguishing the fire, detecting the human body, spraying detection gas for the eyes of people, driving a thief out, preventing fire, and protecting the elderly living alone and patients with dementia, a position prediction apparatus and a method thereof using the drone in an indoor space for monitoring a designated aviation path, and a method of flying the drone which deals with a symptom and transmits a signal to a relevant agency. In general, a terminal checking and receiving a position based on satellites has a problem that it cannot provide position information in an area with a weak satellite signal including the indoors, tunnels, underground parking lots, and city centers. Accordingly, the present invention aims to provide a drone having a sensor for recognizing flames and the human body, transmission/reception image devices, and functions for extinguishing fire and spraying gas, drone functions for decomposing fine dust and exchanging signals with the relevant agencies while taking first aid when a symptom occurs, and a method of flying the drone to allow the drone to automatically return, to be automatically charged, to automatically land and take off, and to take first aid for a symptom.

[0005] This invention focus on ambient air impurities, fire extinguishing, human body detection, and fire prevention. This drone while carrying out its mission, does not receive information from other drones, and the environment situation analysis is different from the claimed method.

Technical Problem

[0006] At present, casualty search systems are designed and implemented by companies that are superior in detecting and identifying casualties caught in debris and avalanches. These systems are brought by search teams near disaster sites and their results are utilized to investigate rescue targets.

[0007] Such systems have drawbacks including the presence of a rescuer at the disaster site, identification of the rescuer's vital and movement signs instead of the injured by the system, and the rescue team's confusion, long search operation time, a large number of users required by the system, and delay in the search process in the early moments of the disaster until the announcement of the area situation by the specialized search and rescue team.

[0008] Hence, the development of detection systems for the injured and missing people caught under debris is very efficient in tracking and identifying the position of the injured. Rescuers in the early moments of disasters such as earthquakes cannot assess all the earthquake-affected areas due to the insecurity of the crisis area and the looseness of the debris.

[0009] Another issue with casualty search systems is the need for several rescuers to set up and use them in disaster areas. An example of this system is the Delsar life detector, which works with 4 users and 6 users. The presence of the rescuer in the disaster area causes errors in the injured detection operations. Other issues include damage to relief equipment, which can disrupt the situation analysis process. [0010] In case the relief and rescue are done quickly in the early moments and the search operation is carried out at a high speed, the percentage of casualties will be greatly diminished.

[0011] In this invention, we seek to design and set up an artificial intelligence-based drone to interact (conversation in the form of questions and answers) (NLP natural language processing in PYTHON with NLTK platform) with humans based machine vision system (image processing) in building environments (residential, administrative, sports, schools, entertainment,) to solve the above issues.

Solution to Problem

[0012] The mentioned invention (figure no. 1), according to the remote control operation in the shortest possible time, analyzes the condition of the injured (figure no. 3) in the debris, because the drone was already embedded in this environment (residential, office, sports, schools, recreation,).

[0013] Given the fact that it is possible to drone be damaged in the event of a disaster, a debris and damage-resistant shield (Figure No. 4) has been designed in which the drone is kept, and this shield itself has a control and positioning system. (Figure No. 5, parts 5, pressure sensors, 6 piezoelectric sensor, 7 module, 8 battery, 9 color screen, 10 battery charger socket, 11 gyroscope sensor, 12 sound market module, 13 storage compartment for electronic components ). Along with several other drones that are present in the complex or building, which interact with each other through the wifi system in the drone, and in case of loss of data transmission lines or internet, the robots communicate with each other through their central wifi system. As a result, it provides all the information related to the injured in real time to the rescuers, based on which they prioritize the areas with high casualties.

[0014] The occurrence of an accident or disaster has signs including loud noise, rising ambient temperature, huge smoke, strong vibration, etc. If any of these are sensed by this system, the files are automatically activated and the status is checked and reports are prepared and sent. Also, considering that the system is connected to emergency centers such as fire stations, emergency and police, the start of the activity and the preparation of reports can be ordered by each of them. [0015] The images of the areas covered by the drone are also directly available to the rescuers (police, fire station, emergency center) who should check the existing situation before entering the area and if an area needs special equipment, they should carry it with them. They can communicate with the injured or people in the environment using a camera (Figure No. 2, Part No. 2) and programming through NLP natural language processing in Python with the NLTK platform.

[0016] Given the integration of two artificial intelligence based on image processing through the camera and sound processing through the speaker, it can extract images related to each person and at the same time analyze the person's voice on the person's image. This causes that when rescuers rescue a person, his/her voice and image will no longer be sent to rescue centers (police, fire station, emergency) as persons in the disaster area and need of help, thereby reducing the time of information analysis. Then it is possible to find out how many people have been rescued from disaster and how many more people should be searched for.

[0017] The casualty search system consists of two general parts. The first part is the drone and the equipment mounted on the drone, and the other part is the equipment related to artificial intelligence and machine vision (Figure No. 2, part 3), which uses these two artificial intelligence systems based on YOLO algorithms for machine vision and NLP algorithms for audio processing. Hence, the operators can interact and talk to the victims as well as their members in the disaster area.

[0018] The equipment related to the UAV system is actually the systems related to the detection of injured people who trapped in the debris, and this information should be sent to the centers in real-time.

[0019] The equipment mounted on the drones should be as light as possible to increase the duration of the maneuvering of the drones. In addition, to prevent a collision or possible damage of the drone to the injured, a guard or shield for a drone is used (Figure No. 2, Parts 1-8, 2-8 and 3-8) which is printed with a soft printer that has a soft body.

Advantageous Effects of Invention [0020] · Connecting to several other robots that can interact with the drone.

[0021] · Sending data and information in a specialized and selective manner for each department in question.

[0022] · Presence at the place before the disaster.

[0023] · Quickly checking the situation of the disaster site and sending information.

[0024] · Analysis and removal of the image and voice sent by rescued persons.

[0025] · Given the built-in artificial intelligence, it has the self-learning potential in every maneuver.

[0026] · Gathering information and learning to predict what will happen in future events.

[0027] · The ability to change different programs and commands according to the needs of each organization.

Brief Description of Drawings

[0028] Figure 1 : 3D view of the drone (unmanned), along with the main view (frontal, top, side).

[0029] Figure 2: Exploded view with all the parts used in the construction and design of the civilian unmanned drone.

[0030] Figure No. 3: Schematic view of the operation of the civilian unmanned drone in order to identify the victims.

[0031] Figure No. 4: 3D view of the drone shield inside the building to prevent possible damage to the drone in emergency conditions.

[0032] Figure No. 5: Exploded view of the drone and its shield along with the information transmission system from the drone shield.

[0033] Figure No. 6: Explanation of system function by flowchart:

Description of Embodiments

[0034] · Part No. 1 , drone.

[0035] · Part No. 2, camera to record images.

[0036] · Part No. 3, the electronic board of the drone with all the electronic parts. [0037] · Part No. 4, coreless motors for drone flight.

[0038] · Part No. 5, the upper part of the drone.

[0039] · Part No. 6, UAV propellers.

[0040] · Part No. 7, the support base, on which the drone is mounted.

[0041] · Part No. 1-8, the lower part or guard of drone.

[0042] · Part No. 2-8, accessories for connecting the lower and upper parts of the drone guard.

[0043] · Part No. 3-8, the upper part of the shield or guard of the drone.

[0044] Figure No. 3: Schematic view of the operation of the civilian unmanned drone in order to identify the victims.

[0045] Figure No. 4: 3D view of the drone shield inside the building to prevent possible damage to the drone in emergency conditions.

[0046] Figure No. 5: Exploded view of the drone and its shield along with the information transmission system from the drone shield.

[0047] · Part No. 1, impact and debris-resistant shield.

[0048] · Part No. 2, drone.

[0049] · Part No. 3, drone support base.

[0050] · Part No. 4, drone guard to prevent possible damage to the drone.

[0051] · Part No. 5, pressure sensor for the pressure applied to the shield.

[0052] · Part No. 6, a piezoelectric sensor in order to identify the vibration entering the shield.

[0053] · Part No. 7, Esp32 programming board in order to send or receive information on the drone shield.

[0054] · Part No. 8, battery.

[0055] · Part No. 9, color screen to display the desired information.

[0056] · Part No. 10, charger socket for charging the battery. [0057] · Part No. 11 , gyroscope sensor to determine the position and angle of the shield.

[0058] · Part No. 12, sound module or buzzer for alarm or warning.

[0059] · Part No. 13, electronic parts shield.

[0060] Explanation of system function by flowchart:

[0061 ] A: After the occurrence of signs of a disaster, given the fact that the drone is already at the installation site, it will reach the disaster area in the shortest time.

[0062] B: Given the activation of the machine vision and sound system, the drone starts sending information to three aid centers.

[0063] B-1 : Police stations.

[0064] B-2: Hospital centers.

[0065] B-3: fire stations.

[0066] C: Each relief and rescue center using this artificial intelligence algorithm can receive and analyze the information, in turn, helps in speeding up the rescue. All centers can connect to this system, but each one can only analyze its data.

[0067] D: Algorithms used to analyze all events:

[0068] D-1: Yolo

[0069] E: Algorithm applied for the programmed image processing system and based on that it scan the environment.

[0070] F: Object identification or classification: In object identification, a raw image is received and it is determined which category it belongs to.

[0071] Classification and Topology: In this case, we have an image where there is only one object in which we find the place of that object.

[0072] Positioning the object: In this case, the position of the object in the image is found.

[0073] G: Among the most important features of the environment identification system, the following can be mentioned: [0074] 12 million bounding boxes to categorize 1.7 million images in 500 different classes (collections, categories, topics).

[0075] Images with complex scenes that include several different objects; Average of 7 bounding boxes per image.

[0076] Very diverse images that contain different and distinct objects (according to each sector in question).

[0077] G-1 : Regarding the fire station, the system marks the required items and analyzes them:

[0078] · Identification and estimation of fire size

[0079] · Identification of wall and floor cracks

[0080] · Identification of collapsed places and debris on victims

[0081] · Estimation of smoke and its volume for a possible fire in the coming hours

[0082] · The amount of bending of the columns and beams in the building to estimate the time for the building to collapse

[0083] · Identification of area where collapse or disaster has occurred

[0084] *And .

[0085] G-2: Black signal: Scan through the camera to process the image [0086] Red signal: receiving and recording images from the site [0087] Green signal: sending data and information to the fire station operator [0088] G-3: Identification of probable thieves [0089] Identification of property in the area

[0090] Controlling the site and identifying the responsible persons to prevent any abuse

[0091] Identifying people and obtaining the information of each person in the area along with commuting

[0092] G-4: Black signal: Scan through the camera to process the image [0093] Red signal: receiving and recording images from the site [0094] Green signal: sending data and information to the police operator

[0095] G-5: Identification of people trapped under debris

[0096] The initial preparation for the upcoming conditions

[0097] Identification of injuries in people

[0098] Obtaining demographic statistics of affected persons and preparing appropriate equipment

[0099] Determining the injured organs of people

[0100] Obtaining vital signs of persons

[0101] G-6: Black signal: Scan through the camera to process the image

[0102] Red signal: receiving and recording images from the site

[0103] Green signal: sending data and information to the police operator

[0104] H: With every disaster and operation, artificial intelligence is able to reach general data by collecting information and storing it and based on these data and experiences, it begins to analyze and predict possible cases using transfer learning artificial intelligence

[0105] Finally, it started to suggest using tools in different situations and increasing the preparedness to deal with disaster

[0106] D -2 : NLP

[0107] I: The algorithm used for the audio processing system.

[0108] J: It is a way for computers to understand human language

[0109] It is considered one of the branches of artificial intelligence and helps computers and robots to understand human language by knowing how humans use language.

[0110] K: The stages of natural language processing in the system, which included the following

[0111 ] A person talks to a drone

[0112] The voice device of the drone records a person's voice [0113] The voice device of the drone converts the human voice into text and sends it to the competent authorities

[0114] Texts are processed and an appropriate text response is considered

[0115] The text response is converted into audio

[0116] The drone plays the answer audio file

[0117] L: Voice assistant and interactive conversation.

[0118] L-1 : syntactic analysis.

[0119] L-2: Semantic analysis.

[0120] L-3: noisy data

[0121] L-1 -1 : Syntactic analysis is related to the subject of language syntax.

[0122] Knowledge syntax is the study of the rules related to the arrangement of words in people's conversations, and it assesses the words in the sentence in a way that is meaningful to us so that in times of stress, people's expressions can be analyzed.

[0123] L-2-1 : To check the meanings of unclear words used by people or injured ones at the affected site.

[0124] L-3-1 : It contains a series of information from the site that is considered meaningless, but artificial intelligence must try to analyze the data to reach its final goal. The main functions are as follows:

[0125] The sound of injured people trapped under the debris to find possible casualties

[0126] Recognition of people's voices in crowded places

[0127] Analysis of injured speech in crowded places

[0128] Analyzing and identifying the sound of the building and construction materials at the time of the disaster and obtaining the probability of collapse or further damage

[0129] M: Analyzing and sending audio information to the relevant centers for the operator. [0130] N: combination and placement of these two parts facilitates providing aid and finding the injured for each related center. This greatly reduces the confusion of operators in the initial moments.

[0131] It also greatly increases the probability of finding people using artificial intelligence and learning in each algorithm operation. Sending the proposed algorithm to each of the expert teams according to the stored data of different similar disasters.

[0132] Improving the performance of artificial intelligence by doing more operations. Integrating the image processing system with the sound processing system to reach accurate and sensitive information.

[0133] Sending data and information from different departments to the centers that are closest and more similar to the subject of that organization so that they can form the interaction between different organizations, which is possible according to the classifications that occurred in the image processing department.

Industrial Applicability

[0134] The invention can be used in all crises and unpredictable disasters such as floods, earthquakes, etc., where the victims may be trapped and are not accessible. Each of the relief organizations can receive the information they need from this system based on their needs.

[0135] Given the pre-installation of the drone in the site such as commercial complexes, residential complexes, markets, schools, offices, etc., if one of the signs of a disaster occurs, the files are automatically activated, and check the situation and prepare and send a report.

[0136] The drone given the conditions and the environment starts to fly and starts to analyze and process the images and sounds of the site. Using artificial intelligence in the audio and video parts that are related to the audio and vision processing of the machine, it categorizes the information and sends it specifically to the relief organization.

[0137] In addition, according to the collection of data from the situation and previous disasters, it also sends suggestions for the existing conditions to the related organizations, enabling them to send rescuers with more preparation. [0138] In addition, given the presence of several drones, each one of them processes information together and interactively analyzes or scans the intended site, which makes it possible to find probable victims faster and help them as quickly as possible.