Login| Sign Up| Help| Contact|

Patent Searching and Data


Title:
SANITARY CONTROL SYSTEM
Document Type and Number:
WIPO Patent Application WO/2021/086235
Kind Code:
A1
Abstract:
The invention relates to the field of computing, more specifically to a sanitary control system based on video analytics, and can be used, for example, for sanitary control at food facilities. The essence of the invention is that the sanitary control system comprises an action recognition video camera, a user identification module, a hand movement detection module, a soap usage detection module, a central computing device, and a feedback module. In one embodiment, the claimed system can also comprise a face recognition video camera connected to the user identification module. Also, in another embodiment, the claimed system additionally comprises a disinfection detector. The action recognition video camera is connected to the input of the central computing device via the hand movement detection module and the soap usage detection module connected in parallel. The face recognition video camera is connected to another input of the central computing device via the user identification module with its output connected to the feedback module. The feedback module includes at least one of the following elements: a light-emitting diode indicator, an information display device, and/or a Telegram bot device.

Inventors:
BABAEV NAUM ALEKSANDROVICH (RU)
VARICH STANISLAV GEORGIEVICH (RU)
KHOREV EVGENIY IGOREVICH (RU)
KASHKINOV MATVEY ILYICH (RU)
GIRKIN VALERY IGOREVICH (RU)
KUDINOV ANTON NIKOLAEVICH (RU)
Application Number:
PCT/RU2020/050309
Publication Date:
May 06, 2021
Filing Date:
October 29, 2020
Export Citation:
Click for automatic bibliography generation   Help
Assignee:
LIMETED LIABILITY COMPANY CONNECTOME (RU)
International Classes:
G07C1/10; G06K9/00; G08B21/24
Domestic Patent References:
WO2014062755A12014-04-24
WO2006061550A12006-06-15
Foreign References:
US20070064986A12007-03-22
RU2678401C12019-01-28
US20150302769A12015-10-22
US6038331A2000-03-14
Attorney, Agent or Firm:
LEBEDEV, Vitaliy Viktorovich (RU)
Download PDF:
Claims:
CLAIMS

1. A sanitary control system comprising:

- an action recognition video camera,

- a user identification module,

- a hand movement detection module,

- a soap usage detection module,

- a central computing device,

- a feedback module; wherein the action recognition video camera is connected to the input of the central computing device via the hand movement detection module and the soap usage detection module connected in parallel, and the user identification module is connected to another input of the central computing device with its output connected to the feedback module, and the feedback module includes at least one of the following elements:

- a light-emitting diode indicator; and/or

- an information display device; and/or

- a Telegram bot module.

2. A system according to claim 1, wherein the system additionally comprises a face recognition video camera connected to the user identification module.

3. A system according to claim 1 or 2, wherein the system comprises a disinfection detector with its input connected to the action recognition video camera, and its output connected to the input of the central computing device.

4. A sanitary control system comprising:

- a face recognition video camera,

- an action recognition video camera,

- a user identification module,

- a hand movement detection module,

- a soap usage detection module,

- a central computing device,

- a feedback module; wherein the face recognition video camera is connected to the central computing device via the user identification module with its another input connected to the action recognition video camera via the hand movement detection module, the soap usage detection module, and the disinfection detector, all connected in parallel, and the output of the central computing device is connected to the feedback module containing at least one of the following units: - a light-emitting diode indicator; and/or

- an information display device; and/or

- a Telegram bot module.

Description:
SANITARY CONTROL SYSTEM

The invention relates to the field of computing, more specifically to a control system based on video analytics, and can be used, for example, for sanitary control at food facilities, in public catering, in medical institutions, in the pharmaceutical industry, etc. In general, the claimed invention relates to an access control and management system and is a set of hardware and software technical means used to monitor compliance with sanitary regulations in hand sanitation and provide secured access of personnel to areas with increased sanitary requirements or inform responsible personnel in case of non-compliance with sanitary regulations.

Background

Restricted area access control and management systems are well-known in the art. WO 2006061550 A1, IPC G07B 15/04, published on 15.06.2006 discloses an access management system comprising a barrier, such as a turnstile, for managing access to a predefined area for a user carrying a mobile device. To gain access to this area, certain functional features of the mobile device must be checked, such as the ringtone or camera/video functionality. In case if the functionality is disabled, the access through the turnstile and to the predefined area is allowed.

The closest analogue (prototype) of the claimed invention is a sanitary control system disclosed in RF Patent Publ. No. 2678401, IPC G07C 9/02, published on 28.01.2019. The disclosed sanitary control system, defined as the closest analogue, comprises at least one face recognition camera connected to an industrial computer, and a barrier device, wherein the system additionally comprises at least one action recognition camera connected to the industrial computer, and the industrial computer, by using a neural network running thereon, performs analysis of a video stream received from at least one action recognition camera, and the face recognition camera and action recognition camera cover a washstand with a soap dispenser, a hand sanitation unit, and the industrial computer is connected to the barrier device to control the locking and unlocking of the barrier device.

The disadvantages of the system, defined as the closest analogue, include the lack of functionality to inform responsible personnel in case of non-compliance with sanitary regulations using the system.

Moreover, another disadvantage of the system, defined as the closest analogue, is the absence of state storage in the industrial computer. Due to that, in the closest analogue, all recognizable actions are matched against actions stored in a database. Thus, the database load is extremely high, especially when the system is configured to work with a significant number of users (for example, in industrial conditions), as request processing performed by the database, writing data to it, as well as receiving responses from the database require significant time, causing a delay in the operation of the system as a whole.

In the closest analogue, the hand sanitizing actions (hand washing) recognized using neural network algorithms could not be broken down into steps, and the entire process of action recognition for hand sanitation was atomic (i.e., was an operation that is either performed entirely (from the beginning to end), or not at all), thereby causing the following problems:

1. Two users began washing their hands. The first user stopped sanitizing his/her hands before the time set by the regulations and left the area covered by the action recognition cameras. The second user continued sanitizing his/her hands in accordance with the regulations. In such conditions, the system acted arbitrary and attributed the event incorrectly.

2. In case of incorrect user interaction with the system when the hand sanitation action recognition algorithm was running, the system had to run the algorithm entirely anew instead of just a single incorrectly performed step, thereby increasing the computational system load.

Brief description

The claimed sanitation control system monitors hand sanitation based on factors such as duration of washing, sequence of movements, use of soap and/or disinfectants. The claimed system can also assess hand sanitation actions. The system can perform monitoring in real time, and also provide post-monitoring based on exported data. The data received from the system can be further analyzed to make decisions on personnel training, warnings, additional checks of products for contamination, etc.

The claimed invention is a sanitary control system, comprising a set of software and hardware technical means for monitoring of hand sanitation based on photo and video data analysis. The monitoring is the detection of hand sanitation by users, as well as collection and analysis of information about the users of the system who previously sanitized their hands. The hand sanitation monitoring can also be performed by other methods. Monitoring of interaction with the soap dispenser or another part of the system (for example, a dryer, a disinfectant dispenser) is also performed, for example based on photo and video data analysis or interaction with various sensors.

The technical problem solved by the claimed invention is to resolve the disadvantages of the prototype, including, in particular: the lack of functionality to inform responsible personnel in case of non-compliance with sanitary regulations using the system; the absence of state storage in the industrial computer; inability to break down the detection process into separate steps. The essence of the invention is that the claimed sanitary control system comprises an action recognition video camera, a user identification module, a hand movement detection module, a soap usage detection module, a central computing device, a feedback module, wherein the action recognition video camera is connected to the input of the central computing device via the hand movement detection module and the soap usage detection module connected in parallel, and the user identification module is connected to another input of the central computing device with its output connected to the feedback module, and the feedback module includes at least one of the following elements: a light-emitting diode (LED) indicator; an information display device; a Telegram hot module.

In one embodiment, the system additionally comprises a face recognition video camera connected to the user identification module.

In another embodiment, the claimed system comprises a disinfection detector with its input connected to the action recognition video camera, and its output connected to the input of the central computing device.

The claimed hand sanitation control system can additionally comprise:

- additional devices feeding a video stream into the central computing device, that can also be used for action recognition and/or face recognition and/or for monitoring of interaction with the soap dispenser or another part of the system;

- a tool for automatic object recognition, wherein data is read and/or written to a transceiver device (such as sensors, RFID, magnet cards, NFC chip) based on received signals;

- a hand dryer located at or nearby the hand sanitation device;

- UI for user interaction (for example, an acrylic disk with LED backlight, GUI display);

- controlled barrier devices.

The technical result provided by the claimed sanitary control system is an increase in the speed of the hygienic hand sanitizing action recognition process due to the attainability of hand sanitation movement type recognition. The claimed sanitation control system provides detection of the hand sanitation grade, such as: home grade (mechanical hand sanitation), hygienic grade (involving skin antiseptics), surgical grade (a sequence of hand sanitation manipulations with sterile gloves put on thereafter), EN-1500 (European hand sanitation standard, a general hand sanitation technique from the hygienic grade (sequential hand sanitation using soap and water followed by skin antiseptic) — and up to the surgical grade) etc.

The claimed sanitary control system, when compared to the system defined as the closest analogue, is more flexible and can be implemented and/or configured in various manners.

The claimed sanitary control system provides increased hand sanitation recognition (detection) accuracy due to the presence of a soap usage detection module running a neural network as a part of the claimed sanitary control system. The soap usage detection module eliminates the possibility of non-compliance with hand sanitary regulations. The signal from the action recognition video camera is fed to the input of the soap usage recognition module and it assesses the use of soap by detecting soap suds on the user's hands using a neural network algorithm. In another embodiment of the claimed system, in case if non-foamable detergents are used, the system detects the use of soap by detecting pressing on the dispenser with the detergent (soap). In yet another embodiment of the system, in case if contactless dispensers are used, the detection of the use of soap is performed by receiving a signal from the detergent dispenser that soap has been squeezed out.

Moreover, the claimed sanitary control system provides an up to 10 fold increase in performance when signals received from the face recognition video camera and action recognition video camera are coded and decoded. Also, to improve performance in the claimed system, high- performance protocols are used for data exchange between the modules included in the claimed system. The ZeroMQ protocol can be used as an example of a high-performance protocol in the claimed system.

Moreover, the claimed sanitary control system enforces compliance with sanitary regulations when multiple users (for example, facility staff) are present within the area covered by the action recognition video camera and the face recognition video camera. Users can freely move between hand sanitation posts (sinks with faucets etc.), and the claimed system will continue to detect the actions of each user individually.

The claimed sanitary control system also provides the ability to customize various types of user profiles. As such, it is possible to allow or disallow the claimed system to detect the face of a specific user or group of users, or enable/disable the ability to select a hand sanitation regulation for a specific user or group of users depending on the sanitary requirements for a specific user or a group of users.

Brief description of the drawings

Fig.1 shows a flow chart of the claimed sanitary control system. This figure shows the blocks, elements and modules that make up the claimed sanitary control system, that is: an action recognition video camera (2); a user identification module (3); a hand movement detection module (4); a soap usage detection module (5); a central computing device (7); a feedback module (8), including at least one of: a Telegram bot module (9), an information display device (10), and/or LED indicator (11). Fig.1 also shows a face recognition video camera (1) and disinfection detector included in an embodiment of the claimed system. Fig.2 shows computational algorithms of the elements and modules included in the claimed sanitary control system.

Embodiments of the invention

As shown at Fig1., the claimed sanitary control system includes the following units and modules: a face recognition video camera (1) (optionally), an action recognition video camera (2), a user identification module (3), a hand movement detection module (4), a soap usage detection module (5), a disinfection detector (6) (optionally), a central computing device (7), and a feedback module (8).

In turn, the feedback module (8) can include at least one of the following: a light-emitting diode (LED) indicator (11), an information display device (10), and a Telegram bot software module (9).

The face recognition video camera (1) is connected to the user identification module (3), which is, in turn, connected to the central computing device. The action recognition video camera (2) is connected to the central computing device (7) via the hand movement detection module (4), soap usage detection module (5), and disinfection detector (6), all connected in parallel, with its output connected to the feedback module (8).

The face recognition video camera (1) is a video camera mounted in such a way that it covers the user's face during identification, for example, at the level of the user's face, and is used to obtain an image of the user's face in real time.

The action recognition video camera (2) is a video camera that is mounted above the hand sanitation post (such as washbowl, sink with faucets, washstand and the like). This video camera is used for obtaining (detection) of an image of the user's hands and tracking thereof, detection of user interaction with the soap/disinfector dispenser, detection of the presence of soap suds on the user's hands, etc.

The detection and logic modules, that is the user identification module (3), the hand movement detection module (4), the soap usage detection module (5), and the disinfection detector (6), included in an embodiment of the claimed system, are controlled by software running on at least one computing device. As such, one computing device can run all of the detection and logic modules included in the claimed system. At the same time, in an embodiment of the claimed invention, each of the detection and logic modules can run on an individual computing device. The computing device can be presented as a ready-to-use device (such as Raspberry Pi, a widely used single board portable computer) or can be installed on top of existing hardware (such as an ordinary server), if such hardware meet the claimed sanitary control system's requirements. In addition, the detection and logic modules included in the claimed system can be implemented by the way of installation on a cloud server.

The user identification module (3) included in the claimed system provides recognition of the users' faces.

The hand movement detection module (4) included in the claimed system provides recognition of the users' hand movements.

The soap usage detection module (5) included in the claimed system detects whether the user of the system has used soap.

The disinfection detector (6), optionally included in the claimed system, detects whether the user of the system has used disinfectant.

The soap usage detection module (5) can be a suds detection module. In another embodiment of the claimed system the soap usage detection module (5) can be a module for detection of interaction with the dispenser. The latter can be implemented either by analyzing the video stream using a neural network, or by receiving a signal from the dispenser.

The details confirming the possibility for the user identification module (3), the hand movement detection module (4), the soap usage detection module (5), and the disinfection detector (6) to perform the corresponding aforementioned functions as parts of the claimed sanitary control system are shown as a computational algorithm flow chart at Fig.2.

In an embodiment, the user identification module (3) uses the Gstreamer multimedia framework to extract a fragment with the user's face image from the video stream received from the face recognition video camera (1), i.e. the image is subject to clipping (the "crop" function). The Single Shot MultiBox Detector neural network is used to perform face detection in the user identification module (3). Moreover, the user identification module (3) uses the ZeroMQ data exchange protocol and library.

In another embodiment, the user identification module (3) includes an RFID reader and the identification is performed by reading data from an RFID transponder previously distributed to each user of the system. Other user identification alternatives are also possible.

In an embodiment of the claimed sanitary control system, the hand movement detection module (4) also uses the Gstreamer multimedia framework to extract a fragment with the user's hands image from the video stream received from the action recognition video camera (2). To operate as a part of the claimed system, the hand movement detection module (4) uses machine learning techniques. The detection of the user's hands movements is performed by the module (4) using the TensorFlow machine learning library. The hand movement detection module (4) can also use the LongShort Term Memory recurrent neural network architecture. The hand movement detection module (4) can also use the Convolutional NeuralNetwork neural network architecture. Moreover, the hand movement detection module (4) uses the ZeroMQ data exchange protocol and library.

In an embodiment, the soap usage detection module (5) uses the Gstreamer multimedia framework. The soap usage detection can be performed by the module (5) using the TensorFlow machine learning software library. Moreover, the soap usage detection module (5) uses the ZeroMQ data exchange protocol. In an embodiment of the claimed system, it additionally comprises a disinfection detector (6). Clipping of the image received by the disinfection detector (6) from the action recognition video camera (2) can also by performed using the Gstreamer multimedia framework. To detect the usage of disinfector by the disinfection detector (6), the PyTorch library is used. Moreover, the disinfection detector (6) uses the ZeroMQ data exchange protocol.

The ZeroMQ data exchange protocol can be used for providing interaction between the user identification module (3), the hand movement detection module (4), the soap usage detection module (5), and the disinfection detector (6) and the central computing device (7).

The central computing device (7) is a computing device running software for collecting data from the detecting modules, and also processing the data sent to the feedback module (8).

The central computing device (7) can be implemented using hardware with the following parameters:

Operating system: Ubuntu 16.04.5 LTS x64 v. 4.15.0 or above;

Motherboard: Asus Maximus VIII Ranger;

CPU: Intel(R) Core(TM) i7-7700K @ 4,2 GHz;

Video card: Nvidia GeForce GTX 1080 Ti;

RAM: Corsair DDR-4 2666MHz 16Gb; to increase the performance of the central computing device, multiple RAM modules can be used;

Storage device: Samsung 860 EVO SSD 500 Gb and above, and Western Digital WD40EZRZ-00G 5400rpm 4 Gb and above.

The central computing device (7) interacts with the user identification module (3), the hand movement detection module (4), the soap usage detection module (5), and the disinfection detector (6) using a high-performance protocol (for example, the JSON-RPC remote call protocol) specifically providing high performance of the claimed sanitary control system. The protocol used for signal interchange between the central computing device (7) and the detection and logic modules (3), (4), (5), (6), supports notifications (i.e. the data sent to the central computing device does not necessarily require a response), as well as multiple calls.

The user feedback module (8) can include a Telegram hot module (9), which is a software module receiving data from the central computing device (7) on every event detected by the detection and logic modules (3), (4), (5), (6). For example, the information on an "Ivanov failed to wash his hands" event is received by the Telegram of Ivanov's supervisor using the Telegram bot.

The feedback module (8) can include an information display device (10) in the form of a display for displaying information to the user and receiving user feedback. The information display device (10) can be implemented as a screen (such as an LED display) mounted in such a way that the user could comfortably observe the screen. Usually, the information display device (10) is mounted at the level of the user's face. The screen displays tips and results of user interaction with the system.

The feedback module (8) can also include a light-emitting diode indicator (11) also providing user feedback. The light-emitting diode indicator (11) can be implemented in a form of color indicators, indicating the results of the user interaction with the system.

Thus, the feedback module (8) provides several user feedback options, based on a signal received from the central computing device: by sending a message using the Telegram bot module (9), by indicating a successful or failed event using the light-emitting diode indicator (11), or visual data representation upon a successful or failed event using the information display device (10).

The claimed sanitary control system can additionally provide the following functions:

- recording and/or storing data in the central computing device (7) or another data storage;

- automatic admission to restricted areas;

- notifying the user on successful or failed hand sanitation (using a display, LED indicator, or other indication means);

- notifying other users of the system on successful or failed hand sanitation by users;

- reading and/or writing data to an access control transceiver device of the user;

- other feedback, data storage and processing features.

User recognition and identification in the claimed sanitary control system is performed using facial recognition computer vision algorithms or PACS systems (RFID and the like).

Thus, all technical means and underlying software encompassed by the invention are developed and released by both domestic enterprises and leading foreign companies.

The interaction of the means provided by the invention is implemented in terms of various known processes in the field of computing. The process of manufacturing of all the devices making up the claimed sanitary control system can involve the use of standard stock industrial machinery, as well as known materials and parts.

As shown at Fig.2, the claimed sanitary control system works as follows. In an embodiment of the claimed sanitary control system, the user identification module (3) includes an RFID reader, and the identification is performed by reading data from an RFID transponder. According to this embodiment, the user data is recorded on the RFID transponder.

In another embodiment of the claimed system, when a user appears at the area covered by the face recognition video camera (1), the user identification module (3) performs detection of the face of this user after receiving a signal from the face recognition video camera (1). The user identification module (3) then analyses only the extracted fragment containing the face image of the user(s), using an automatic face detection algorithm (i.e. by clipping the image, that is, using the "crop" function), thus also increasing the performance of the claimed system. The user identification module (3) contains face image data of the personnel admitted to the zone secured using the claimed sanitary control system. Thus, as a result of the face recognition process, the face can be successfully recognized or not. In the case of a successful recognition of the user's face, the user identification module (3) sends identification data corresponding to the recognized face to the central computing device (7) which matches the received identification data against the actions of the respective user presented at the area covered by the face recognition video camera (1) and action recognition video camera (2). In the case of a failed recognition of the user's face, the user identification module (3) sends a message to the central computing device (7), which is then transmitted to the feedback module (8). The claimed sanitary control system can work with multiple cameras at the same time, and thus provide simultaneous face recognition and hand sanitation recognition for multiple users, presented at the area covered by the face recognition video camera (1) or one or multiple action recognition video cameras (2).

Further, the user of the system identified using the user identification module (3) proceeds with hand sanitation. In this case, the hand movement detection module (4) detects hand movements based on a signal from the action recognition video camera (2), making a conclusion on whether the user is sanitizing his/her hands or not. In the case if the hand movement detection module (4) finds that the user has not proceeded with sanitizing his/her hands, this module (4) falls back to the hand movement detection step. In the case if the hand movement detection module (4) finds that the user has proceeded with sanitizing his/her hands, this module (4) assesses the compliance of the hand movements with a pre-defined hand sanitation grade. In the case of compliance of the movements of the user's hands with the said hand sanitation grade, the hand movement detection module (4) proceeds to measurement of the duration of the hand sanitation. In the case of non-compliance of the movements of the user's hands with the said hand sanitation grade, the hand movement detection module (4) sends a signal to the central computing device (7) indicating a non-compliance with the hand sanitation requirements by a specific user. This signal is further transmitted from the central computing device (7) to the feedback module (8), indicating of a non-compliance with the hand sanitation requirements by a user identified using the user identification module (3).

At the same time, the soap usage detection module (5) detects the use of soap by the user sanitizing his/her hands using a signal received from the action recognition video camera (2). In an embodiment of the claimed system, the soap use detection is performed based on receipt of a signal from a detergent dispenser. In case if a use of soap by the user is detected, the soap usage detection module (5) sends a corresponding signal to the central computing device (7). If the soap usage detection module (5) fails to detect a use of soap by the user, this module (5) sends a signal indicating the non-use of soap by the respective user to the central computing device (7).

The disinfection detector (6), which is a part of the claimed system in one of its embodiments, functions in a similar way. The disinfection detector (6) receives a signal from the action recognition video camera (2). If a use of the disinfectant by the user is detected, the disinfection detector (6) sends a signal to the central computing device (7) indicating a use of the disinfectant by the user. In the case if no use of the disinfectant by the user is detected by the disinfection detector (6), the disinfection detector (6) sends a signal to the central computing device (7) indicating that the disinfectant was not used by the respective user. In an embodiment of the claimed system, the detection of a use of the disinfectant is performed by receiving a signal from the disinfectant dispenser.

Then the hand movement detection module (4) additionally measures the duration of the hand sanitation by the user. In the case if the duration of the hand sanitation by the user corresponds to the hand sanitation grade, the module (4) sends a signal to the central computing device (7) indicating that the hand sanitation by the user corresponds to the hand sanitation grade. In this case, the computing device (7) can send a corresponding signal to the feedback module (8). In the case if the duration of the hand sanitation by the user does not correspond to the hand sanitation grade, the module (4) sends a signal to the central computing device (7) indicating that the hand sanitation by the user does not correspond to the hand sanitation grade. In this case, the computing device (7) can send a signal to the feedback module (8) indicating that the hand sanitation by the user is non- compliant.

If all of the hand sanitation grade requirements are met by the user, the detection and logic modules (3), (4), (5), (6) send a signal to the central computing device (7) indicating that the user meets the requirements.

If at least one of the hand sanitation grade requirements is not met by the user, and if such an event is detected by at least one of the detection and logic modules (3), (4), (5), (6), such module(s) send(s) a signal to the central computing device (7) indicating that the user does not meet the hand sanitation grade requirements. The claimed sanitary control system can also be used at medical institutions and food facilities in accordance with, for example, Sanitary regulations and norms (SanPiN) 2.1.3.2630-10 "Sanitary and epidemiological requirements for organizations carrying out medical activities", as well as, for example, with other sanitary and epidemiological rales and norms such as SanPiN 2.3.4.551-96 "Food and refinery facilities (technological processes. Raw materials) Production of milk and dairy products Sanitary regulations and norms", SanPiN 3238-85 "Sanitary regulations for meat industry enterprises", SanPiN 2.3.4.050-96 "Food and refinery industrial facilities (technological processes, raw materials) Production and distribution of fish products Sanitary regulations and norms" and/or other sanitary and epidemiological regulations and norms and /or hygienic standards. The hand sanitation can also be performed in accordance with the European hand sanitation standard EN-1500 (or other standards for hand sanitation).