Login| Sign Up| Help| Contact|

Patent Searching and Data


Title:
METHOD AND SYSTEM FOR AUTOMATIC PASSENGER COUNTING IN PUBLIC TRANSPORT
Document Type and Number:
WIPO Patent Application WO/2018/156096
Kind Code:
A1
Abstract:
An invention relates to the field of control devices and, more specifically, to systems used for registering entry or exit of people, in particular, passengers in public transport by counting the number of people passing under an optical camera mounted above a passenger door of a vehicle. The invention also relates to improving the efficiency of the analysis of a video stream coming from optical cameras to a microcomputer, being part of the said system, using the developed method for automatic passenger counting in public transport.

Inventors:
KHAZIN MARAT BORISOVICH (UA)
SKORCHENKO SERGIY ANATOLIYOVICH (UA)
Application Number:
PCT/UA2017/000090
Publication Date:
August 30, 2018
Filing Date:
September 13, 2017
Export Citation:
Click for automatic bibliography generation   Help
Assignee:
KHAZIN MARAT BORISOVICH (UA)
International Classes:
G07C9/00; G06K9/00
Domestic Patent References:
WO2013128326A12013-09-06
Foreign References:
US6919804B12005-07-19
EP1100050A12001-05-16
EP0700017A21996-03-06
US20140241581A12014-08-28
US20080212099A12008-09-04
US6919804B12005-07-19
Other References:
ANTONIO ALBIOL; INMACULADA MORA; VALERY NARANJO: "Real-Time High Density People Counter Using Morphological Tools", IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, vol. 2, no. 4, December 2001 (2001-12-01), XP002777046
ANTONIO ALBIOL; INMACULADA MORA; VALERY NARANJO: "Real-Time High Density People Counter Using Morphological Tools", IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, vol. 2, no. 4, December 2001 (2001-12-01), XP011028400
Attorney, Agent or Firm:
BOROVYK, Petro (UA)
Download PDF:
Claims:
Claims

1. A method for automatic passenger counting in public transport comprising: obtaining a single-frame image from video streaming from at least one optical camera mounted above at least one doorway of a vehicle, saving a brightness value for pixels of an area on every image frame, selected by preset coordinates of boundaries, with a random access memory (RAM), determining the pixel brightness value of at least one control line inside the selected area with the doorway of the vehicle being opened, determining a value and a direction of an optical flow for at least one control line of the selected area in each image frame with the doorway of the vehicle being opened, generating a temporary image based on the stored pixel brightness values using at least one hardware-software complex, determining the maximum time interval that corresponds to no passenger passing through the doorway of the vehicle, identifying areas for passenger counting inside the selected area by generating an image from a totality of points inside the selected area with the maximum and minimum brightness values for each point in the selected area at a certain maximum time interval taking into account the mutual arrangement of points relative to the doorway of the vehicle, generating a grayscale image for processing, the said image comprises the temporary image based on the saved pixel brightness values and the image of the totality of points inside the selected area with the maximum and minimum brightness values, using pixel brightness values from the RAM, setting the maximum brightness value for the image thus generated, generating a new grayscale image for another time interval, which corresponds to no passenger passing through the doorway of the vehicle before obtaining an image with brightness values that match the set maximum value, binarization of the grayscale image for processing, identifying tolerances for image defects to separate the related areas vertically and/or horizontally, processing of the binary image thus generated by identifying the related areas and image defects for subsequent separation of the related areas vertically and/or horizontally relative to boundaries of the selected area until the defects corresponding to the predefined tolerances disappear, determining the direction of an optical flow for each separated area, and passenger counting by counting the separated areas in near real-time mode.

2. The method according to claim 1 wherein an area is selected in the obtained image of each frame by establishing boundaries of the area with the help of two lines located closer to the doorway and farther from the doorway of the vehicle and two lines that restrict the said lines at both sides.

3. The method according to claim 1 wherein an image is obtained from at least one optical camera as grayscale at a rate of 60 frames per second.

4. The method according to claim 1 wherein while selecting at least one preferably rectangular area, deviations are set out for coordinates of boundaries of the rectangle located closer to the doorway and farther from the doorway of the vehicle.

5. The method according to claim 1 wherein when obtaining a value and a direction of an optical flow, at least four control lines are used inside a rectangle, the said lines being parallel to boundaries of the rectangle located closer to the doorway and farther from the doorway of the vehicle.

6. The method according to claim 1 wherein a time interval at which no passenger passing through a doorway of a vehicle is recorded is used as a maximum time interval corresponding to minimal passenger traffic.

7. The method according to claim 1 wherein an image generated for processing is translated into a binary code followed by noise suppression.

8. The method according to claim 1 wherein image defects are identified for subsequent separation of related areas using a convex hull algorithm.

9. The method according to claim 1 wherein once the direction of motion of each separated area relative to a doorway of a vehicle has been identified, an image is subject to filtration by removing abnormally large and/or abnormally small related areas.

10. The method according to claim 1 wherein once the direction of motion of each separated area relative to a doorway of a vehicle has been identified, an image is subject to filtration by removing related areas having abnormally high and/or abnormally low speed.

11. A system for automatic passenger counting in public transport in accordance with the method disclosed in claim 1 comprising interconnected with data exchange channels: at least one hardware- software complex for obtaining and processing image data from at least one optical camera positioned on a vehicle, at least one optical camera mounted above at least one doorway of the vehicle, at least one means of receiving the signal from at least one navigation system, at least one means for wireless communication between the hardware-software complex and a remote server, a means for door position control at the doorway of the vehicle, a means for illumination of the doorway of the vehicle, a sensor of illumination level of the doorway of the vehicle, the said hardware- software complex comprises at least one microcomputer for passenger counting in the vehicle, where the hardware-software complex is installed, configured so to provide a possibility for a software-based analysis of an image obtained from video streaming and passenger counting by counting the separated related areas in near real-time mode, and a means for data storage equipped with a random access memory connected to the microcomputer.

12. The system according to claim 11 wherein a GPS reception module is used as a means for receiving a signal from at least one navigation system.

13. The system according to claim 11 wherein a wireless modem is used as a means for wireless communication between a hardware-software complex and a remote server.

14. The system according to claim 11 wherein a door opening sensor is used as a means for door position control.

15. The system according to claim 11 wherein the system further comprises a means for illuminating an area of a vehicle which image contains a video stream from an optical camera.

16. The system according to claim 11 wherein the system further comprises a microcontroller coupled to a microcomputer, a sensor of illumination level at a doorway of a vehicle and a means for illumination of the doorway of the vehicle.

17. The system according to claim 16 wherein a means for door position control is coupled to a microcontroller and a microcomputer.

18. The system according to claim 17 wherein a microcomputer is coupled to an optical camera and a means for receiving a signal from at least one navigation system and a means for wireless communication between a hardware-software complex and a remote server.

19. The system according to claim 1 wherein a hardware-software complex for obtaining and processing image data comprises a main unit where there are a microcomputer, an optical camera, a means for receiving a signal from at least one navigation system, a means for wireless communication between the hardware-software complex and a remote server, a means for door position control at a doorway of a vehicle and a means for illumination of the doorway of the vehicle, and additional units, coupled to the main one, where there are a microcomputer, an optical camera, a means for illumination of the doorway of the vehicle and a means for door position control at the doorway of the vehicle.

20. The system according to claim 1 wherein a hardware-software complex for obtaining and processing image data comprises a main unit where there are a microcomputer, a means for receiving a signal from at least one navigation system, and a means for wireless communication between the hardware-software complex and a remote server, and additional units, coupled to the main one, in the quantity that corresponds to that of doorways of the vehicle in which there are an optical camera, a means for illumination of the doorway of the vehicle and a means for door position control at the doorway of the vehicle.

21. The system according to claim 1 wherein a hardware-software complex for obtaining and processing image data comprises a main unit where there are a microcomputer, an optical camera, a means for receiving a signal from at least one navigation system, a means for wireless communication between the hardware-software complex and a remote server, a means for door position control at the doorway of the vehicle and a means for illumination of the vehicle doorway, and additional units, coupled to the main one, where there are an optical camera, a means for illumination of the doorway of the vehicle and a means for door position control at the doorway of the vehicle.

AMENDED CLAIMS

received by the International Bureau on 22 June 2018 (22.06.2018)

1. A method for automatic passenger counting in public transport comprising: obtaining a single-frame image from video streaming from at least one optical camera mounted above at least one doorway of a vehicle, saving with a random access memory (RAM) a brightness value for pixels of an area on an image of every frame, with the said area having preferably a rectangular form and which is selected on the obtained image of every frame with the help of two lines, one line being located closer to the doorway and the second line being farther from the doorway of the vehicle, and two lines that restrict the said lines at both sides, determining the pixel brightness value of at least one control line inside the selected area with the doorway of the vehicle being opened, determining a value and a direction of an optical flow for at least one control line of the selected area in each image frame with the doorway of the vehicle being opened, generating a temporary image of the selected area of every frame based on the stored pixel brightness values using at least one hardware-software complex, determining the maximum time interval that corresponds to no passenger passing through the doorway of the vehicle, generating an image from a totality of points inside the selected area with the maximum and minimum brightness values for each point in the selected area at a certain maximum time interval taking into account the mutual arrangement of points relative to the doorway of the vehicle, in order to identify areas for passenger counting inside the selected area and to generate a grayscale image for processing, the said image comprises the temporary image based on the saved pixel brightness values and the image of the totality of points inside the selected area with the maximum and minimum brightness values, using pixel brightness values from the RAM, setting the maximum brightness value for the image thus generated, generating a new grayscale image for another time interval, which corresponds to no passenger passing through the doorway of the vehicle before obtaining an image with brightness values that match the set maximum value, binarization of the grayscale image for processing, identifying tolerances for image defects to separate the related areas vertically and/or horizontally, processing of the binary image thus generated by identifying the related areas and image defects for subsequent separation of the related areas vertically and/or horizontally relative to boundaries of the selected area until the defects corresponding to the predefined tolerances disappear, determining the direction of an optical flow for each separated area, and passenger counting by counting the separated areas in real-time mode.

The method according to claim 1 wherein an image is obtained from at least one optical camera as grayscale at a rate of 60 frames per second.

The method according to claim 1 wherein while selecting at least one preferably rectangular area, deviations are set out for coordinates of boundaries of the rectangle located closer to the doorway and farther from the doorway of the vehicle.

The method according to claim 1 wherein when obtaining a value and a direction of an optical flow, at least four control lines are used inside a rectangle, the said lines being parallel to boundaries of the rectangle located closer to the doorway and farther from the doorway of the vehicle.

5. The method according to claim 1 wherein a time interval at which no passenger passing through a doorway of a vehicle is recorded is used as a maximum time interval corresponding to minimal passenger traffic.

6. The method according to claim 1 wherein an image generated for processing is translated into a binary code followed by noise suppression.

7. The method according to claim 1 wherein image defects are identified for subsequent separation of related areas using a convex hull algorithm.

8. The method according to claim 1 wherein once the direction of motion of each separated area relative to a doorway of a vehicle has been identified, an image is subject to filtration by removing abnormally large and/or abnormally small related areas.

9. The method according to claim 1 wherein once the direction of motion of each separated area relative to a doorway of a vehicle has been identified, an image is subject to filtration by removing related areas having abnormally high and/or abnormally low speed.

10. A system for automatic passenger counting in public transport in accordance with the method disclosed in claim 1 comprising interconnected with data exchange channels: at least one hardware-software complex for obtaining and processing image data from at least one optical camera positioned on a vehicle, at least one optical camera mounted above at least one doorway of the vehicle, at least one means of receiving the signal from at least one navigation system, at least one means for wireless communication between the hardware-software complex and a remote server, a means for door position control at the doorway of the vehicle, a means for illumination of the doorway of the vehicle, a sensor of illumination level of the doorway of the vehicle, the said hardware-software complex comprises at least one microcomputer for passenger counting in the vehicle, where the hardware- software complex is installed, configured so to provide a possibility for a software-based analysis of an image obtained from video streaming and passenger counting by counting the separated related areas in near real-time mode, and a means for data storage equipped with a random access memory connected to the microcomputer.

11. The system according to claim 11 wherein a GPS reception module is used as a means for receiving a signal from at least one navigation system.

12. The system according to claim 11 wherein a wireless modem is used as a means for wireless communication between a hardware-software complex and a remote server.

13. The system according to claim 11 wherein a door opening sensor is used as a means for door position control.

14. The system according to claim 11 wherein the system further comprises a means for illuminating an area of a vehicle which image contains a video stream from an optical camera.

15. The system according to claim 11 wherein the system further comprises a microcontroller coupled to a microcomputer, a sensor of illumination level at a doorway of a vehicle and a means for illumination of the doorway of the vehicle.

16. The system according to claim 16 wherein a means for door position control is coupled to a microcontroller and a microcomputer.

17. The system according to claim 17 wherein a microcomputer is coupled to an optical camera and a means for receiving a signal from at least one navigation system and a means for wireless communication between a hardware- software complex and a remote server.

18. The system according to claim 1 wherein a hardware-software complex for obtaining and processing image data comprises a main unit where there are a microcomputer, an optical camera, a means for receiving a signal from at least one navigation system, a means for wireless communication between the hardware-software complex and a remote server, a means for door position control at a doorway of a vehicle and a means for illumination of the doorway of the vehicle, and additional units, coupled to the main one, where there are a microcomputer, an optical camera, a means for illumination of the doorway of the vehicle and a means for door position control at the doorway of the vehicle.

19. The system according to claim 1 wherein a hardware-software complex for obtaining and processing image data comprises a main unit where there are a microcomputer, a means for receiving a signal from at least one navigation system, and a means for wireless communication between the hardware- software complex and a remote server, and additional units, coupled to the main one, in the quantity that corresponds to that of doorways of the vehicle in which there are an optical camera, a means for illumination of the doorway of the vehicle and a means for door position control at the doorway of the vehicle.

20. The system according to claim 1 wherein a hardware-software complex for obtaining and processing image data comprises a main unit where there are a microcomputer, an optical camera, a means for receiving a signal from at least one navigation system, a means for wireless communication between the hardware-software complex and a remote server, a means for door position control at the doorway of the vehicle and a means for illumination of the vehicle doorway, and additional units, coupled to the main one, where there are an optical camera, a means for illumination of the doorway of the vehicle and a means for door position control at the doorway of the vehicle.

Description:
Method and System for Automatic Passenger Counting in Public Transport

An invention relates to the field of control devices and, more specifically, to systems used for registering entry or exit of people, in particular, passengers in public transport by counting the number of people passing under an optical camera mounted above a passenger door of a vehicle. The invention also relates to improving the efficiency of the analysis of a video stream coming from optical cameras to a microcomputer, being part of the said system, using the developed method for automatic passenger counting in public transport.

Many similar methods and systems are known to the applicant including the following considered the closest prior art.

A prior art discloses the method and the system for identifying the presence of people in a room as disclosed by WO2013128326A1 published on September 06, 2013. The said method and system are designed to count people preferably in a room with cameras being positioned at a sufficiently high level to capture a video stream. Besides, the system must have several cameras. Motion sensors may be used instead of cameras. The method comprises the division of the room into squares wherein the direction of motion of objects is determined.

A disadvantage associated with this method and the system is the positioning of optical cameras at a height that exceeds that of a doorway in the vast majority of vehicles. In this case, the said method and system fail if, at certain time points, a frame obtained from video streaming is completely overlapped with a moving object. The other drawback is that it is necessary to use an image of the entire area of a room in order to count people.

The similar method and system are disclosed in US2014241581A1 published on August 28, 2014. The said method and system are also designed for automatic counting of physical persons with optical cameras being positioned at a sufficiently high level and, in this case, several optical cameras, a matrix image sensor and a motion sensor must be used.

The disadvantage associated with this system is its complexity due to the need to use multiple cameras at the same time and the need to use motion sensors. The disadvantage of the method is that a colorful image must be obtained for processing.

The prior art also discloses a method for counting people passing through a gate as disclosed in US2008212099A published on September 04, 2008. The method is intended to be used in immovable room and also requires a camera positioned at a sufficiently high level. Further, the said method is sensitive to changes in intensity and color temperature of illumination. To separate objects on an image and count them, a color histogram is used. This imposes increased requirements to the stability of color characteristics of an optical camera. Also, in order to count people it is necessary to completely process an image frame. The prior art also discloses a system for automatic passenger counting in public transport disclosed in US6919804B1 published on July 19, 2005. The system comprises ultrasonic or infra-red sensors mounted above a door of a vehicle, one per passenger's width. The sensors are separated from each other, so that each sensor captures its part of a step of the vehicle. The accuracy of counting is thus ensured only when passengers of more or less standard size are passing under the respective sensor. If a passenger stands under several sensors or passes from one sensor to another, the system mistakenly counts several passengers. In other words, the system is designed for "ideal" passengers of the same size, having the same trajectory of motion and being standard for the system. Therefore, the disadvantage of this system is that there is quite a high probability of error in counting due to the system sensitivity to passenger sizes and trajectory of motion and a width of a doorway of the vehicle.

A method for automatic passenger counting in public transport using morphological filtration of an image obtained and the related system as disclosed in [Antonio Albiol, Member, IEEE, Inmaculada Mora, and Valery Naranjo, Real- Time High Density People Counter Using Morphological Tools, IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL 2, NO. 4, December 2001] are considered the closest prior art. The said method comprises: obtaining an image from video streaming from at least one optical camera mounted above at least one doorway of a vehicle, selecting an area in the image thus obtained, obtaining a brightness value for pixels located inside the selected area, in particular, for several lines of the image, determining passenger counting areas inside the selected area, generating an image for processing, processing the image so generated by identifying the related areas and image defects for subsequent separation of the related areas vertically and/or horizontally relative to boundaries of the selected area followed by passenger counting by counting the separated related areas automatically in near real-time mode. The said method may be implemented by a system comprising interconnected with data exchange channels: at least one hardware-software complex for obtaining and processing image data from at least one optical camera mounted above at least one doorway of a vehicle and a microcomputer for passenger counting in the vehicle, wherein the hardware-software complex is installed.

The disadvantage associated with the said method and system is a possibility of quite a high error (20%) even under the "ideal" conditions, disclosed in the said information source, which are quite far from the real conditions of operation of a vehicle. Further, when working with the said method, image control lines, which are used to obtain maximum image contrast and to determine, on its basis, areas for counting (so-called "blobs", spots), may also change their positions. A change in positioning may be caused by looser fixation of an optical camera while the vehicle is moving. This requires ongoing manual adjustment of positioning of such lines, which results in error and requires constant monitoring of the system, especially in real-life conditions. Besides, the system fails to provide reliable passenger counting under the conditions of insufficient illumination, e.g. at night.

Neither method nor system identical to those claimed has been identified. The object of the invention was to develop the method for automatic passenger counting in public transport which provides better accuracy of counting, use of an image from one optical camera for processing, positioning of the optical camera at a height corresponding to that of a doorway of a vehicle, determination of the number of passengers irrespective of their size and trajectory of motion, and application of the method under any illumination.

The said object is achieved so that the method for automatic passenger counting in public transport, in accordance with the invention, comprises: obtaining a single-frame image from video streaming from at least one optical camera mounted above at least one doorway of a vehicle, saving a brightness value for pixels of an area on every image frame, selected by preset coordinates of boundaries, with a random access memory (RAM), determining the pixel brightness value of at least one control line inside the selected area with the doorway of the vehicle being opened, determining a value and a direction of an optical flow for at least one control line of the selected area in each image frame with the doorway of the vehicle being opened, generating a temporary image based on the stored pixel brightness values using at least one hardware-software complex, determining the maximum time interval that corresponds to no passenger passing through the doorway of the vehicle, identifying areas for passenger counting inside the selected area by generating an image from a totality of points inside the selected area with the maximum and minimum brightness values for each point in the selected area at a certain maximum time interval taking into account the mutual arra ngement of points relative to the doorway of the vehicle, generating a grayscale image for processing, the said image comprises the temporary image based on the saved pixel brightness values and the image of the totality of points inside the selected area with the maximum and minimum brightness values, using pixel brightness values from the RAM, setting the maximum brightness value for the image thus generated, generating a new grayscale image for another time interval, which corresponds to no passenger passing through the doorway of the vehicle before obtaining an image with brightness values that match the set maximum value, binarization of the grayscale image for processing, identifying tolerances for image defects to separate the related areas vertically and/or horizontally, processing of the binary image thus generated by identifying the related areas and image defects for subsequent separation of the related areas vertically and/or horizontally relative to boundaries of the selected area until the defects corresponding to the predefined tolerances disappear, determining the direction of an optical flow for each separated area, and passenger counting by counting the separated areas in near real-time mode.

According to one embodiment of the method, an area is selected in the obtained image of each frame by establishing boundaries of the area with the help of two lines located closer to the doorway (inside transport) and farther from the doorway(outside transport) of the vehicle and two lines that restrict the said lines at both sides. According to another embodiment of the method, an image is obtained from at least one optical camera as grayscale at a rate of 60 frames per second.

According to yet another embodiment of the method, while selecting at least one preferably rectangular area, deviations are set out for coordinates of boundaries of the rectangle located closer to the doorway and farther from the doorway of the vehicle.

According to yet another embodiment of the method, when obtaining a value and a direction of an optical flow, at least four control lines are used inside a rectangle, the said lines being parallel to boundaries of the rectangle located closer to the doorway and farther from the doorway of the vehicle.

According to yet another embodiment of the method, a time interval at which no passenger passing through a doorway of a vehicle is recorded is used as a maximum time interval corresponding to minimal passenger traffic.

According to yet another embodiment of the method, an image generated for processing is translated into a binary code followed by noise suppression.

According to yet another embodiment of the method, image defects are identified for subsequent separation of related areas using a convex hull algorithm.

According to yet another embodiment of the method, once the direction of motion of each separated area relative to a doorway of a vehicle has been identified, an image is subject to filtration by removing abnormally large and/or abnormally small related areas.

According to yet another embodiment of the method, once the direction of motion of each separated area relative to a doorway of a vehicle has been identified, an image is subject to filtration by removing related areas having abnormally high and/or abnormally low speed. The object of the invention is also to develop a system for automatic passenger counting in public transport that helps to reduce error in counting and, at the same time, to simplify the system, to operate the system under any illumination and to increase reliability of its use. An additional object is to create a system with increased versatility by ensuring module-based configu ration and combination of its main components depending on the intended use.

The said object is achieved in that the system for automatic passenger counting in public transport, in accordance with the method disclosed above and in accordance with the invention claimed herein, comprises interconnected with data exchange channels: at least one hardware-software complex for obtaining and processing image data from at least one optical camera positioned on a vehicle, at least one optical camera mounted above at least one doorway of the vehicle, at least one means of receiving the signal from at least one navigation system, at least one means for wireless communication between the hardware- software complex and a remote server, a means for door position control at the doorway of the vehicle, a means for illumination of the doorway of the vehicle, a sensor of illumination level of the doorway of the vehicle, the said hardware- software complex comprises at least one microcomputer for passenger counting in the vehicle, where the hardware-software complex is installed, configured so to provide a possibility for a software- based analysis of an image obtained from video streaming and passenger counting by counting the separated related areas in near real-time mode, and a means for data storage equipped with a random access memory connected to the microcomputer.

According to yet another embodiment of the system, a door opening sensor is used as a means for door position control. According to yet another embodiment of the system, the system further comprises a means for illuminating an area of a vehicle which image contains a video stream from an optical camera.

According to yet another embodiment of the system, the system further comprises a microcontroller coupled to a microcomputer, a sensor of illumination level at a doorway of a vehicle and a means for illumination of the doorway of the vehicle.

According to yet another embodiment of the system, a means for door position control is coupled to a microcontroller and a microcomputer. According to yet another embodiment of the system, a microcomputer is coupled to an optical camera and a means for receiving a signal from at least one navigation system and a means for wireless communication between a hardware- software complex and a remote server.

According to yet another embodiment of the system, a hardware-software complex for obtaining and processing image data comprises a main unit where there are a microcomputer, an optical camera, a means for receiving a signal from at least one navigation system, a means for wireless communication between the hardware-software complex and a remote server, a means for door position control at a doorway of a vehicle and a means for illumination of the doorway of the vehicle, and additional units, coupled to the main one, where there are a microcomputer, an optical camera, a means for illumination of the doorway of the vehicle and a means for door position control at the doorway of the vehicle.

According to yet another embodiment of the system, a hardware-software complex for obtaining and processing image data comprises a main unit where there are a microcomputer, a means for receiving a signal from at least one navigation system, and a means for wireless communication between the hardware-software complex and a remote server, and additional units, coupled to the main one, in the quantity that corresponds to that of doorways of the vehicle in which there are an optical camera, a means for illumination of the doorway of the vehicle and a means for door position control at the doorway of the vehicle. According to yet another embodiment of the system, a hardware-software complex for obtaining and processing image data comprises a main unit where there are a microcomputer, an optical camera, a means for receiving a signal from at least one navigation system, a means for wireless communication between the hardware-software complex and a remote server, a means for door position control at the doorway of the vehicle and a means for illumination of the vehicle doorway, and additional units, coupled to the main one, where there are an optical camera, a means for illumination of the doorway of the vehicle and a means for door position control at the doorway of the vehicle.

The said essential features allow us to achieve the technical result and the following benefits for automatic passenger counting in public transport.

The system disclosed above operates with optical cameras positioned at a minimum height unlike similar devices known in the prior art. The said method for processing of an image thus obtained with an optical camera allows identification of a passenger with the optical camera being positioned at a minimum height as such height depends on that of the vehicle doorway when a frame is overlapped by a moving object at certain time intervals. In this case, passengers passing through the doorway are counted by processing a small area of an image frame, with the coordinates being preset, to reduce the likelihood of error in calculating and, at the same time, to simplify the image processing algorithm. Further, unlike similar devices known in the prior art, the system claimed herein does not require the use of motion sensors to count passengers and can be implemented using only one optical camera that obtains a monochrome image at a minimum resolution - this also simplifies the system to a significant extent and makes the method more protected from interferences and less burdensome for system resources.

Further, the system claimed herein is insensitive to a change in the illumination level, even in a wide range of changes, and absolute brightness of an image does not significantly affect image accuracy. The tests of the system have shown that the system claimed herein can count passengers under the conditions when illumination is insufficient even for manual counting.

Additionally, the system operates in a wide range of doorway widths, even when one optical camera is used, and, with the image processing algorithm, separates passengers on the image thus obtained. The method claimed herein makes the system insensitive to uneven trajectory of passing passengers and passenger size.

The testing completed by the inventors with the system and the method disclosed herein has demonstrated that passenger counting error may be up to 5% in contrast to similar devices and methods known in the prior art, including in particular the prototype, where error exceeded 20% under ideal conditions (uniform bright illumination, no chaotic movement of people, stable movement of a vehicle etc.). Taking into account that the method and the system are preferably intended for public transport, where such "ideal" conditions are practically unlikely, the achieved passenger counts make the system efficient for use under any conditions.

Thus, the results of testing the system and the method under the real-life conditions were obtained by mounting the system on two vehicles (buses) and counting passengers on different days. Error in passenger counting done by the system was determined by comparing the values thus obtained against the results of visual passenger counting done by a controller accompanying the buses with the system installed. The results are shown in the table below.

Table: Results of the system testing in accordance with the claimed invention

Percentage of error could be less since the controller did not count the passengers who went out at bus stops to allow other passengers to board the bus (error in the counting technique).

The results of the tests presented in the table indicate a lower percentage of error (not more than, preferably below, 5%) when passengers were counted automatically compared to the method and system known in the prior art, indicating that the technical result has been accomplished with the method and system claimed herein.

When using the said system and method, error in passenger counting is reduced by selecting optimal points on black and white lines (specific points within a separate area with predefined coordinates are selected, not the lines) to achieve the maximum possible contrast of a resulting single-frame image and provide more optimal conditions for binarization and further segmentation of the image (separation and counting of areas). Further, error is reduced by automatic adjustment of control lines using the same algorithm for choosing optimal points, so that it is enough to specify only a rectangle (an area of each image frame selected according to predefined coordinates of boundaries) in which a search should be done, and when the axis of an optical camera is displaced within certain boundaries, the most contrasting points of choice remain inside this rectangle.

In the method taken as a prototype, similar control lines extend beyond steps when a vehicle was moving, so contrast became minimal and the accuracy of counting reduced and required more rigid fixation of the system and regular adjustment of coordinates of the lines. The invention claimed herein is illustrated by the following example of the method and the system for automatic passenger counting in public transport and by drawings illustrating the following:

FIG. 1 is a block diagram of the embodiment of the system for automatic passenger counting in public transport (the embodiment of the system based on independent computing units).

FIG. 2 is a block diagram of the embodiment of the system for automatic passenger counting in public transport (the embodiment of the system based on one computing unit).

FIG. 3 is a block diagram of the embodiment of the system for automatic passenger counting in public transport (the combined embodiment).

FIG. 4 is a block diagram of the computing unit of the system for automatic passenger counting in public transport.

FIG. 5 is a diagram showing data transfer from the system to a remote server and beyond to users depending on user requests. FIG. 6 is a flowchart showing the embodiment of the method for automatic passenger counting in public transport.

FIG. 7 (a) - (d) is an example of a single-frame image from video streaming from an optical camera mounted above at least one doorway of a vehicle, including designation of the selected area and four control lines within boundaries of the area where: FIG 7 (a) shows a passenger leaving a vehicle and crossing the first control line of the selected area; FIG 7 (b) shows a passenger leaving a vehicle and crossing the second control line of the selected area; FIG 7 (c) shows a passenger leaving a vehicle and crossing the third control line of the selected area; and FIG 7 (d) shows a passenger leaving a vehicle and crossing the fourth (last) control line of the selected area.

FIG. 8 is an example of generation of a binary monochromatic image and processing of such image for subsequent separation of related areas vertically and/or horizontally and the results of counting the number of passengers by counting separated related areas in near real-time mode.

The figurative materials that illustrate the invention claimed herein as well as an example of the particular embodiment of the system for automatic passenger counting in public transport and the related method of the use of the system are in no way intended to limit the claims appended hereto but to explain the essence of the invention.

The system for automatic passenger counting in public transport is preferably built on the unit construction principle. The combination of system units depends on the number of doorways of a vehicle where passenger counting is required. The system always comprises the main unit, which, in particular, provides communication with a server. The system may be powered by the onboard power supply available on the vehicle. An optical camera is mounted above the doorway of the vehicle(zenithal position) to count the number of people passing under the optical camera followed by software-based analysis of images taken from video streaming that come from each optical camera to a microcomputer or several microcomputers (depending on the system configuration).

Depending on the positioning of optical cameras and microcomputers, the following configuration options can be implemented.

In the first option, the system is based on independent computing units (FIG. 1), e.g. a main unit 1 and IM additional units 2 that are part of the hardware- software complex for obtaining and processing image data from at least one optical camera. Each computing unit 1 and 2 comprises a microcomputer 3, an optical camera 4, and door position sensors as a means for door position control at a doorway of a vehicle with LED lighting 5 as a means for illumination of the doorway of the vehicle. Alternatively, infrared or other similar illumination can be used. All units function independently of each other. Therefore, failure of one unit does not lead to a complete shutdown of the system, and the system can continue to work with limited functionality. The microcomputer 3 of each unit 1 or 2 receives video streaming from the respective optical camera 4 and counts passengers according to the method disclosed above and exemplified below. Data exchange between the units 1 and 2 is carried out through data exchange channels that can be implemented on various Ethernet or wireless (Wi-Fi) networks or through a radio channel in various modifications of the system. Each unit (both main and additional one) is mounted above the doorway of the vehicle. The positioning of the main unit 1 depends on the vehicle type and is determined by the efficiency of communication channels between the units. In particular, in two-door vehicles, the main unit 1 is mounted above the front door, while the additional unit 2 is mounted above the rear door. In three-door vehicles, the main unit 1 is mounted above the middle door, while the additional units 2 are mounted above the front and rear doors.

In the second option, the system is configured to comprise one main computing unit 1 and additional units 2 (FIG. 2). In this case, the microcomputer 3 is a part of the main unit 1 only and provides passenger counting by analyzing video streaming from all optical cameras 4, which are part of the additional units 2. Passengers can also be counted on a server once video stream data have been received. The main unit 1 is installed in one of technological compartments of the vehicle. Optical cameras 4 of the additional units are installed above passenger doorways of the vehicle. LED door position sensors 5 may be mounted on the additional units 2, if appropriate. The data exchange is carried out between the units in the way described for the first option. This embodiment of the system provides the opportunity to make maximum use of standard solutions known in the prior art (e.g., IP cameras having Ethernet or Wi-Fi interfaces), and also helps to minimize the dimensions of the additional units 2 mounted directly above the areas of passenger flow passing. In this case, there are no significant limitations for the dimensions of the computing unit of the system positioned in the main unit 1.

The third option is a combined embodiment of the system (FIG. 3). The main unit 1 is configured in the way described in the first option of the system. The additional units 2 are configured in the way described in the second option of the system and are connected to main unit 1 via internal communication channels (Ethernet or Wi-Fi or a radio channel). A video stream is transmitted from the additional units 2 to the microcomputer 3 which provides passenger counting based on data coming from all units and transmits information to the server. Alternatively, data may be transmitted from all units to the server so that passenger counting takes place on the server. The third combined option of the 7 000090 system is most economically feasible given the minimal number of units and a relatively low cost of each additional unit 2. The only limitation is positioning of the microcomputer 3 in the body of the main unit 1 so that the microcomputer 3 should be capable to process video streams coming from all optical cameras 4 of the system.

The structure of the main unit 1 used in the first and the third options of the system and the additional unit 2 used in the first option of the system: except as otherwise specified herein, the description relates to both the main unit 1 and the additional unit 2. The unit comprises the following main elements (FIG. 4): a single-board microcomputer (SBC) 3 having an optical camera 4, specifically a video camera (VC). The main elements provide for the implementation of the main function - passenger counting. Further, the unit contains a microcontroller (MC) 7 and a set of peripheral units that provide the functionality of door position and illumination sensors. The main unit comprises a modem (M) 6 as a means for wireless communication between a hardware-software complex and a remote server, e.g., via a cellular network. In order to provide voltage at levels required for the operation of various parts of the unit, the built-in power supply units (PSU1, PSU2) 8 are used. Power is supplied to the microcomputer 3 through a switch 9 (SI) which is controlled by a microcontroller 7. The microcontroller 7, among other things, performs a watchdog function for the microcomputer 3, specifically the microcomputer 3 sends a signal to the microcontroller 7 on a regular basis (e.g. once per minute) using a separate communication line. As long as the signal is received on a regular basis, no action is taken. If the microcontroller 7 does not receive a signal for a long period of time (for more than 5 minutes) that may indicate a critical malfunction in the microcomputer 3, then the microcontroller 7 switches off power of the microcomputer 3 and switches it on three seconds thereafter for guaranteed capacitors discharge. This provides a hardware reboot for the microcomputer 3 without interrupting power supply to the entire system.

In order to prevent the incorrect operation of the microcontroller 7, power is supplied by a separate single-directional line through the power supply and launching unit (LU) 10. The unit 10 provides separation and filtering of supply voltage of the microcontroller 3 from pulses associated with the switching of relatively powerful consumers such as the microcomputer 3, the modem 6, and a means for illumination of the doorway of the vehicle.

The current location of the vehicle is determined by a means receiving a signal from at least one navigation system - a GPS reception module (G) 11 which is part of the main unit 1 (FIG. 1-4) and a special antenna (A) 12.

Passengers are counted as long as doors of the vehicle are open. In order to control door position, the system comprises a means for door position control at the doorway of the vehicle, e.g. magnetic contact reed sensors (R) 13 can be used. If necessary, other types of sensors may be used, including connection to the standard vehicle sensors. Information from the door position sensor comes to a converter (DC) 14, from which the normalized signal is transmitted to the microcomputer 3 (to activate the counting function) and to the microcontroller 7 (to control the additional lighting). As long as the standard vehicle sensors are used, a galvanic separation unit may be provided between the sensor 13 and circuits of the microcomputer 3 and the microcontroller 7 for electric safety.

An additional LED illumination of the doorway is provided by a means for illumination of the doorway of the vehicle - a LED module 15. The module provides a targeted light flux sufficient to illuminate the passage and provide the high-quality operation of the video camera 4 at nighttime. The LED illumination is turned on by the microcontroller 7 via a switch (S2) 16 provided there is a signal from a sensor of illumination level 17 implemented on the basis of the photo resistor (PR) with the control unit (CU) 18. A LED module driver 15 stabilizes the current that flows through LEDs to prevent them from overheating and malfunction. To prevent "flickering" light, the microcontroller is programmed so that there is signal blocking mode from the illumination level sensor 17 once LEDs have been turned on: if already on, illumination remains switched on until the door has been closed, regardless of a signal coming from the sensor of illumination level 17.

The microcomputer 3 is equipped with a memory card (SD) 19 as a means for data storage.

The method for automatic passenger counting in public transport is implemented using the system disclosed above as follows (FIG. 6, stages: 101- 112).

First, a single-frame image is obtained from video streaming from at least one optical camera 4 mounted above at least one doorway of the vehicle (101). Images are obtained as grayscale preferably at a rate of 60 frames per second.

Then, the brightness value for pixels on an area of the image of each frame, selected in accordance with the preset coordinates of boundaries, using the RAM (RAM) of the microcomputer 3 (102) are stored. The area is separated by establishing the boundaries of the area with the help of two lines located closer to the doorway and farther from the doorway of the vehicle and two lines that restrict the said lines on both sides (a rectangle in FIG 7 a-d). When selecting at least one preferably rectangular area, deviations of coordinates are set for boundaries of the rectangle located closer to the doorway and farther from the doorway of the vehicle to compensate for displacement of the rectangle, e.g. when a vehicle moves and camera moves/rotates because of vibration. 17 000090

Brightness values for pixels of at least one control line inside the selected area and a value and a direction of the optical flow for at least one control line of the selected area in each image frame with the doorway of the vehicle (103) being open are determined based on the stored data. When obtaining the value and the direction of an optical flow, at least four control lines inside the rectangle, parallel to its limits, located closer to the doorway and farther from the doorway of the vehicle (lines inside the rectangle in FIG. 7 a-g) are used.

Then, a temporary image is generated from the stored pixel brightness values using the microcomputer 3 of the hardware and software complex of the system. Further, the maximum time interval, which corresponds to no passenger passing through the doorway of the vehicle (104), is also determined. A time interval at which no passenger passing through a doorway of a vehicle is recorded is used as a maximum time interval corresponding to minimal passenger traffic.

Then, the areas for passenger counting are identified inside the selected area by generating an image from a totality of points inside the selected area with the maximum and minimum brightness values for each point in the selected area at a certain maximum time interval taking into account the mutual arrangement of points relative to the doorway of the vehicle and generating a grayscale image for processing (105), the said image comprises the temporary image based on the saved pixel brightness values and the image of the totality of points inside the selected area with the maximum and minimum brightness values, using pixel brightness values from the RAM. Then, the maximum brightness value is determined for the image thus generated, and the image is checked for compliance with the said value (106). In case of non-compliance, a new grayscale image is generated for another time interval, which corresponds to no passenger passing through the doorway of the vehicle before obtaining an image with brightness values that match the set maximum value. This is followed by binarization of the grayscale image for processing (107) and noise suppression. Further, maximum values (tolerances) are then identified for image defects to separate the related areas vertically and/or horizontally and to process the binary image thus generated by identifying the related areas and image defects for subsequent separation of the related areas vertically and/or horizontally relative to boundaries of the selected area until the defects corresponding to the predefined tolerances disappear (108-110). Image defects for the subsequent separation of the related areas are determined preferable using the convex hull algorithm. In this case, the predefined direction of the optical flow is considered for each separated area (111). Once the direction of movement of each separated area was determined relative to the doorway of the vehicle, the image is then filtered by removing the abnormally large and/or abnormally small related areas and/or by removing related areas associated with abnormally high and/or abnormally low speed to avoid errors in determining the area (i.e. a passenger). Finally, passenger counting is performed by counting the separated areas in near real-time mode (112) using the microcomputer 3. Alternatively, as long as the data is transmitted to the server, the separated areas and passengers are counted directly on the server.

Passenger counts and other related data (GPS data, service data, etc.) are transmitted to the remote server 20 via the modem 6 of the main unit 1 installed on the vehicle (FIG. 5). The remote server 20 may be equipped with a web server 21, configured for data transmission to users' hardware-software complexes of 22 according to user request.

FIG. 8 shows an example of image processing in accordance with the method disclosed above using the system claimed herein, where: 1 is the first control line, 2 is the second control line, 3 is the third control line, 4 is the last control line, 5 is the most contrast image obtained using data from all control lines, 6 is separation of the related areas, and 7 is counting of the separated areas and their result.

Therefore, the invention claimed herein increases the accuracy of passenger counting and, at the same time, simplifies and improves the reliability of the system operation using an image from one optical camera for processing and positioning the optical camera at a height corresponding to that of the doorway of the vehicle, the determination of the number of passengers irrespective of their size and trajectory of motion, and the application of the method under any illumination. Further, the use of the system claimed herein helps to reduce error in counting and, at the same time, to simplify the system, to operate the system under any illumination and to increase reliability and versatility of its use for any vehicle.