Field

The present invention relates to a method, a system, an apparatus and a computer program product related to mixed reality services. More particularly, the invention relates to method, system, apparatus and computer program product related to mixed reality services provided to passengers within an elevator car.

Background

There are many factors, which may cause people feeling uncomfortable while traveling elevators, which causes some people even to avoid accessing elevators.

For example, closed-circuit television (CCTV) typically used for supervision purposes, may increase people's sense of anxiety; people don't feel comfortable knowing that someone may be watching them. Claustrophobia is defined as the persistent fear of enclosed spaces; an elevator car is typically a relatively small and confined box, and it is easy to see how an elevator could cause a claustrophobic reaction. Agoraphobia is the fear of being trapped in a situation in which escape would be difficult or impossible should a panic attack occur. Those with agoraphobia typically avoid clusters of related situations. An elevator would be difficult to escape, and it is not unusual for people with agoraphobia to avoid elevators. Even if not having claustrophobia or agoraphobia, people may feel intimidated by occasional need for communicating with other passengers in the elevator car. For example, a passenger that is not quite next to the elevator doors in a crowded elevator car should tell or touch other passengers to provide space for him/her to step out of the elevator car on the destination landing. Some people tend to find a standing position near to the door of the elevator, which may cause blocking the route for incoming and outgoing passengers. Studies indicate that a significant number of bacteria and viruses are transferred via touching elevator buttons. Bacteria cultured from elevator buttons often exceeds that cultured from toilet surfaces. Recent pandemic situation has surfaced a worry that viruses spreading a contagious disease may be transferred via touching elevator buttons.

Description of the related art

Patent EP3309107 Al discloses detection device and an in-car display for elevator doorway supervision. The system enables indicating presence of an object or a person in a detection zone. Purpose of the system is to enable passengers within the elevator car to identify if there is a person or object blocking the passenger from exiting the elevator car.

Modern computer imaging systems are capable of real-time tracking of objects shown in a captured video image. This technology is often referred to as video tracking, which refers to process of automatically locating a moving object or multiple moving objects over time using an imaging sensor. After initiating, the tracking continues even when tracked objects are not moving.

Mixed reality (MR) refers to merging of real and virtual worlds to produce new environments and visualizations, where physical and digital objects co-exist and interact in real time. Mixed reality usually refers to digitally generated, virtual elements that are spacially aware and responsive, thus capable of interacting with users in the real world.

Various display technologies may be used to facilitate the interaction between users and mixed reality applications. Computer imaging and mixed reality applications known today enable real-time interaction between a user and the artificial products in the mixed reality.

Summary

An object is to provide a method and apparatus so as to solve the problems of improving user experience, safety and people flow in an elevator car that is supervised using an imaging sensor. The objects of the present invention are achieved with a method according to the claim 1 and a computer program product according to the claim 16. The objects of the present invention are further achieved with a system according to the claim 6 and with an elevator car according to the claim 11.

The preferred embodiments of the invention are disclosed in the dependent claims.

The present invention is based on the idea of continuously showing by a display apparatus disposed within the elevator car, an image representing the interior of the elevator car, and to provide mixed reality services for the passengers by the display apparatus and a mixed reality module. The image shown in the display apparatus may be an image captured by the imaging sensor, thus enabling the passengers within the car to view the actual image obtained with the imaging sensor or an image generated based on the image obtained with the imaging sensor, or the image representing the interior of the elevator car may be a mirror reflection of the interior of the elevator car, provided by a mirror element of a so called magic mirror display.

The present invention has the advantage that it enables reducing anxiety of passengers and improves passenger comfort and safety, thus facilitating improved user experience. Embodiments of the invention also facilitate improved flow of people in the restricted space of an elevator car. According to a first method aspect, a method for providing digital services in an elevator car is provided. The elevator car is provided with an imaging sensor, such as a closed-circuit TV (CCTV) camera, configured to obtain a video image from the interior of the elevator car, and a display apparatus. The method comprises superimposing one or more mixed reality elements over a first image of the interior of the elevator car shown by the display apparatus. The mixed reality elements are positioned with respect to the first image based at least on content of the obtained video image.

According to a second method aspect, if the first image is the obtained video image or an image generated based on the obtained video image, the method comprises superimposing said one or more mixed reality elements upon the obtained video image or the image generated based on the obtained video image. If the first image is a mirror image of the interior of the elevator car that is reflected by the display apparatus, the method comprises superimposing said one or more mixed reality elements over the mirror image reflected by the display apparatus.

According to a third method aspect, the method comprises tracking at least one passenger within the elevator car based on the captured video image, associating a destination floor with each tracked passenger, and upon arrival of the elevator car at a destination floor of the at least one tracked passenger, superimposing over the first image at least one mixed reality element indicating the at least one tracked passenger who is about to exit the elevator car at this destination floor.

According to a third method aspect, the at least one mixed reality element further indicates an expected or recommended route of the respective tracked passenger from the current position of the passenger towards the door of the elevator car. According to a fourth method aspect, the method further comprises detecting, based on the obtained video image, a new passenger entering the elevator car and initiating tracking of the new passenger, and providing the new passenger a mixed reality car operating panel (COP) by superimposing the mixed reality COP at or near the portion of the first image representing the new passenger for enabling the new passenger to select a destination floor using touchless interaction with the mixed reality COP.

According to a first system aspect, a system for providing digital services in an elevator car is provided. The system comprises an imaging sensor, such as a closed-circuit TV (CCTV) camera, disposed within the elevator car and configured to obtain a video image from the interior of the elevator car, and a display apparatus disposed within the elevator car. The system comprises a mixed reality module configured to superimpose one or more mixed reality elements upon a first image of the interior of the elevator car shown by the display apparatus. The mixed reality elements are positioned with respect to the first image based at least on content of the obtained video image.

According to a second system aspect, the display apparatus is configured to display the obtained video image or a video image generated based on the obtained video image as the first image, and the mixed reality module is configured to superimpose said one or more mixed reality elements upon the displayed video image. Alternatively, the display apparatus is a magic mirror display apparatus configured to show as the first image a mirror image of the interior of the elevator car that is reflected by the magic mirror display apparatus, and the mixed reality module is configured to superimpose said one or more mixed reality elements upon the mirror image reflected by the magic mirror display apparatus. According to a third system aspect, the system further comprises a tracking module configured to track at least one passenger within the elevator car on basis of the captured video image and an association module configured to associate information on a destination floor with each tracked passenger. The mixed reality module is further configured, upon arrival of the elevator car at a destination floor of the at least one tracked passenger, to superimpose upon the first image at least one mixed reality element indicating at least one tracked passenger who is about to exit the elevator car at this destination floor.

According to a fourth system aspect, said at least one mixed reality element indicating the at least one tracked passenger who is about to exit the elevator car at this destination floor indicates an expected or recommended route of the respective passenger from the current position of the passenger towards the door of the elevator car.

According to a fifth system aspect, the system comprises a detecting module configured to detect, based on the obtained video image, a new passenger entering the elevator car, and a tracking module configured to initiate tracking of the new passenger based on said detection. The mixed reality module is further configured to provide the new passenger a mixed reality car operating panel (COP) by superimposing the mixed reality COP at or near the portion of the first image representing the new passenger, for enabling the new passenger to select a destination floor using touchless interaction with the mixed reality COP.

According to a first apparatus aspect, an elevator car comprising an imaging sensor, such as a closed-circuit TV (CCTV) camera, configured to obtain a video image from the interior of the elevator car, and a display apparatus disposed within the elevator car is provided. The elevator car comprises or is connected to a computer or computer device comprising a mixed reality module configured to superimpose one or more mixed reality elements upon a first image shown by the display apparatus, wherein the mixed reality elements are positioned with respect to the first image based at least on contents of the obtained video image.

According to a second apparatus aspect, the display apparatus is configured to display the obtained video image or a video image generated on basis of the obtained video image as the first image, and the mixed reality module is configured to superimpose said one or more mixed reality elements upon the displayed video image. Alternatively, the display apparatus is a magic mirror display apparatus configured to show as the first image a mirror image of the interior of the elevator car reflected by the magic mirror display apparatus, and the mixed reality module is configured to superimpose said one or more mixed reality elements upon the mirror image reflected by the magic mirror display apparatus.

According to a third apparatus aspect, the mixed reality module comprises or is connected to a tracking module configured to track at least one passenger within the elevator car on basis of the captured video image and an association module configured to associate information on a destination floor with each tracked passenger. The mixed reality module is further configured, upon arrival of the elevator car at a destination floor of the at least one tracked passenger, to superimpose upon the first image at least one mixed reality element indicating at least one tracked passenger who is about to exit the elevator car at this destination floor.

According to a fourth apparatus aspect, said at least one mixed reality element indicating the at least one tracked passenger who is about to exit the elevator car at this destination floor further indicates an expected or recommended route of the respective passenger from a current position of the passenger towards the door of the elevator car.

According to a fifth apparatus aspect, the mixed reality module further comprises or is further connected to a detecting module configured to detect, based on the video image, a new passenger entering the elevator car, and a tracking module configured to initiate tracking of the new passenger based on said detection. The mixed reality module is further configured to provide the new passenger a mixed reality car operating panel (COP) by superimposing the mixed reality COP at or near the portion of the first image representing the new passenger, for enabling the new passenger to select a destination floor using touchless interaction with the mixed reality COP.

According to another aspect, a computer program product is provided that comprises computer executable instructions, which, when executed by a computer or a computer system cause the computer or computer system to perform the method according to any one the method aspects.

According to a yet another aspect, a computer program product embodied in one or more non-transitory computer readable medium(s) having computer readable program code embodied thereon is provided, wherein the computer program code which, when executed by a computer or a computer system cause the computer or computer system to perform the method according to any one the method aspects.

Brief description of the drawings

In the following the invention will be described in greater detail, in connection with preferred embodiments, with reference to the attached drawings, in which Figure 1 is a schematic illustration of an elevator car with an imaging sensor and a display apparatus.

Figure 2 is a schematic illustration of a system for providing mixed reality services within an elevator car.

Figure 3 illustrates modules for providing mixed reality services.

Figure 4 illustrates an exemplary situation in an elevator car

Figure 5 illustrates a change of passenger behaviour in an elevator car.

Figure 6 illustrates another exemplary situation in an elevator car

Figure 7 illustrates a first mixed reality service

Figure 8 illustrates a second mixed reality service

Figure 9 illustrates another type of mixed reality service

Detailed description

Figure 1 shows a schematic visualization of an exemplary interior of an elevator car (10) with a display apparatus (12), seen from the direction of the back wall of the elevator car towards the door (11). The elevator car (10) may be any type of elevator car, and it may be equipped with car operating panel (COP) (16) and one or more further display apparatuses (15). The imaging sensor (13), such as a CCTV camera, for supervising the interior of the elevator car (10) is typically installed at the upper portion of the walls or at or near the ceiling in order to give a good overall view into the elevator car so that the view would not be blocked by tall persons standing on the floor. The imaging sensor may be for example a 2D or 3D camera or an infrared camera or a combination thereof In this visualization, the imaging sensor (13) is hidden from direct sight behind a semi-transparent display apparatus (12). The imaging sensor (13) may also be disposed in a frame of the semi-transparent display apparatus (12). Hiding the imaging sensor (13) or physically coupling it with the display apparatus (12) is not required for technical operation of the system in any configuration, nor with any type of the display apparatus.

According to one embodiment of the invention, the image captured by the imaging sensor (13) is shown in the display apparatus (12) to passengers in the elevator car (10). The display apparatus (12) may be of any known type. In another embodiment, the display apparatus may be of so-called magic mirror type, which appears as a mirror with additional capability to display digitally created items, such as informative and/or mixed reality items superimposed upon the actual mirror image. A magic mirror apparatus basically combines a semi -reflective mirror and a traditional display. The semi -reflective mirror of the magic mirror type display apparatus (12) reflects a mirror image just like a normal mirror, which enables avoiding need of showing the actual image captured by the imaging sensor in the display apparatus. Avoiding need to display the actual image is beneficial in an elevator-installed mixed reality system, since it reduces amount of data required to be communicated towards the display apparatus (12).

The figure 2 is a schematic illustration of main elements of a system for providing mixed reality services within an elevator car.

A computer device or system (20) is coupled to the imaging sensor (13) and to the display apparatus (12). The imaging sensor (13) may further be coupled to a surveillance system (not shown), either with a direct connection or via the computer device or system (20). The computer or computer system (20) comprises modules configured to execute functions and/or method steps. The modules may comprise software, hardware, firmware or any combination thereof. The computer device or system (20) may be further coupled to an elevator control system (25) that controls operation of the elevator. The elevator control system (25), well known in the art, for example receives information of elevator landing calls and destination calls made by passengers and controls operation of the elevator by making decisions on which floor is to be served next. The elevator control system (25) may also be part of the computer or computer system (20).

The figure 3 illustrates modules enabling providing mixed reality services within an elevator car. These modules are preferably implemented in the computer or computer system (20). Although the modules are referred herein to as separate from each other, this functional division is not intended to limit implementation of the system, but functionalities of any modules may be further divided to submodules and/or combined with each other without departing from the scope. Only modules with functionalities relevant to the current invention are described, while further modules may exist.

A central portion of the system is a mixed reality module (30). The mixed reality module (30) comprises or is operatively connected to a computer imaging module (31). The computer imaging module (31) is configured to analyze contents of the video image captured by the imaging sensor, using any known image processing technology. The computer imaging module (31) preferably recognizes objects in the image, which are not part of the background, i.e. normal, standard parts of the interior of the elevator car. The background typically contains elements like walls, ceiling, floor and a car operating panel (COP), which are of no particular interest for the mixed reality services.

The computer imaging module (31) may comprise or be operatively connected to a tracking module (32), which is configured to track objects, such as persons (passengers) throughout the entire elevator travel. The tracking module (32) may use any known object tracking technology. Tracking provides information at least on where in the image each tracked object is positioned currently. Modern object tracking technology is known to be able to track any individual object in a digital image irrespective of changes for example in position or lighting. Recognizing identity of the objects is not necessary, but it may be useful to recognize objects in the image that are humans.

The mixed reality module (30) is preferably operatively connected to the elevator control system (25). This connection may be provided by wired or wireless connection. According to one embodiment, an application programming interface (API) is provided between the mixed reality module (30) and the elevator control system (25). Operative coupling between the mixed reality module (30) and the elevator control system (25) enables for example using mixed reality services for obtaining destination calls.

The mixed reality module further comprises or is operatively coupled to an association module (33). The association module (33) is also operatively coupled to the tracking module (32) for enabling association an object, i.e. a passenger, tracked by the tracking module (32) with information on a destination floor associated with the respective object, wherein the information on the destination floor is obtained based on a destination call made by the passenger.

The mixed reality module (30) produces mixed reality elements, which are displayed in the image shown by the display unit by superimposing the mixed reality elements over an image shown by the display unit. As known in the art, a mixed reality element is a digitally generated, virtual world element that is superimposed over an image representing the real world, preferably in real-time. Position of the mixed reality elements in the image is preferably defined at least on basis of contents of the obtained video image. For example, position of the mixed reality elements superimposed over the image may depend on position of objects in the video image as determined by the computer imaging module (31).

The figure 4 illustrates an exemplary, typical situation in an elevator car. the top view of the elevator car shows a plurality of passengers (40, 40a) occupying the car. Some passengers (40a) have tendency to stand quite next to the door (11), which is schematically illustrated as an opening in the car walls (101), disregarding whether they are about to exit the elevator car at the next stop. A benefit of the invented arrangement of disposing the display apparatus above the door (11) and providing mixed reality services by the display apparatus is that for being able to see the mixed reality services provided by the display unit without uncomfortably looking upwards, passengers (40, 40a) are encouraged to enter further towards the back wall of the elevator car, which leaves more room in the front next to the door (11) for new passengers to enter. This is illustrated in the figure 5, showing how passengers watching content of a mixed reality service provided by the display apparatus above the door inside the elevator car have moved closer to the back wall of the cabin, thus enabling the new passenger (40b) entering the elevator car on a landing.

When the new passenger (40b) enters the elevator car, this is detected from the obtained video image by the computer imaging module (31) and the tracking module (32) may start tracking the new passenger (40b), in addition to any other passengers (40, 40a) who are already being tracked. Upon entering the elevator car, the new passenger (40b) may access a conventional car operating panel (COP) for entering a destination floor. Information on the destination floor selection is stored in the elevator control system (25), which may further provide this information to the association module (33), which associates the tracked new passenger (40a) with the respective destination floor information. The association module (33) preferably stores association information temporarily until the elevator car stops at the respective destination floor and the new passenger (40a) exits the elevator car. This way amount of data handled and stored by the association module is reduced by limiting data only to that concerning passengers currently in the elevator car, and no record of associations between passengers and floors need to be stored that might compromise privacy of the passengers. No identification of the tracked passengers is required either, which also protects their privacy. If several new passengers (40a) would enter the elevator car, the video image may be utilized together with the destination floor information received by the COP for determining which one of the new passengers (40a) makes which destination floor selection.

The figure 6 illustrates another challenging situation in a crowded elevator car. When the elevator arrives at a landing, passenger (40c) wishes to exit, but a plurality of fellow passengers (40a) block his/her route towards the door (11). The passenger (40c) who wishes to exit, needs to request the fellow passengers to give way.

The figure 7 illustrates a first mixed reality service provided by the mixed reality module. The figure illustrates an image (70) shown by the display apparatus, which may be simply a mirror image shown in a mirror of a so called magic mirror, which is known in the art, in which the display apparatus comprises a semi-reflective observation glass or equivalent and a display unit is placed behind the observation glass. The magic mirror thus appears like a normal mirror, wherein digitally produced elements may be shown superimposed over the mirror image, thus appearing like floating in mid-air. Alternatively, the image (70) shown by the display apparatus may be the video image obtained by the imaging sensor, or an image generated on basis of the video image. In such case, the video image is preferably shown by the display apparatus as mirrored from left to right, to give the image an appearance of a mirror image reflected by a conventional mirror. Preferably, the image displayed by the display apparatus is a live, real-time image or a nearly real-time image. In the field of computer imaging, a small delay caused by image processing is allowed for real-time images without departing from the concept of realtime. In some embodiments, the obtained video image is processed for example for privacy protection purposes before it's displayed on the display apparatus, although the obtained video image is still used for generating the displayed image. For example, the displayed image is blurred for not showing faces of persons shown in the image, or the displayed image indicates positions of persons with a wireframe indicating outlines of objects or other equivalent graphical indications of locations of persons shown in the image. The displayed image may also be altered for example by replacing the actual background to a different background. Although the displayed image may be processed to generate the displayed image so that it is not the original obtained video image is not displayed, passengers of the elevator car should preferably be able to recognize the portion or object representing themselves in the displayed image.

The mixed reality module produces at least one mixed reality element (71) to be shown in the image (70) by superimposing the at least one mixed reality element (71) over the image that is shown on the display apparatus. The at least one mixed reality element is positioned in the image (70) based on at least information on content of the obtained video image. For example, the position of the at least one mixed reality element (71) may be defined on basis of position of the tracked objects, such as passengers and/or their body parts, in the video image. In the current embodiment, the at least one mixed reality element (71) visually indicates passengers (40c) who are about to exit the elevator car at the next destination floor, preferably indicating an expected or recommended route from the current position of the passenger towards the door of the elevator car. As disclosed earlier, passengers (40a, 40c) are tracked by the tracking module and associated by the association module so that the mixed reality module can define position of each tracked passenger and thus also to define where in the image on the display the at least one mixed reality element should be shown to reflect the actual scene setting. This visual indication in form of a mixed reality element (71) thus informs fellow passengers (40a) who may at least partially block the expected or recommended route out for the passenger(s) (40c) who are about to exit, and guides the fellow passengers (40a) to make room for allowing exit without need for the exiting passenger (40c) to speak up. The mixed reality service thereby facilitates improved, organic people flow especially when passengers are about to exit the elevator car. Problems caused by possible language barriers between passengers not sharing the same language are also mitigated. Thus, user experience is improved by the mixed reality service.

The figure 8 illustrates a second mixed reality service provided. In the recent pandemic times, passengers entering the elevator car may consider themselves as being put to risk of catching contagious disease by touching the conventional car operating panel (COP) for entering a destination floor. Use of mixed reality enables entering a destination using a virtual, mixed reality COP.

Upon detecting a passenger (40d) entering the elevator car, the passenger appearing as a new foreground object on the background is detected by computer imaging from the obtained video image and based on detecting the entry of a new passenger, the tracking module starts tracking him/her. As an alternative for using the conventional COP, the mixed reality module may then provide each new passenger (40b) with a virtual mixed reality COP (80) that is superimposed upon the image shown by the display apparatus. The mixed reality COP (80) beneficially allows fully touchless selection of the destination floor. The mixed reality COP (80) is preferably shown in the displayed image at a position that is at or near the portion of the displayed image representing the respective passenger, so that the mixed reality COP (80) is easily associable with the respective passenger (40c). For example, the mixed reality COP (80) may at least partially overlap with the image of the respective passenger (40b) shown by the display apparatus. The mixed reality COP (80) may be operated by the passenger (40b) for example by a hand gesture performed in mid-air. For example, by pointing with his/her finger a position mid-air so that the image of the finger in the image shown by the display apparatus points a part of the mixed reality COP (80) that corresponds to the desired destination floor number, the passenger may provide mixed reality module with destination floor information, which is subsequently communicated to the elevator control system and provided for the association module for associating the selected destination floor information with the respective tracked passenger (40b). A further benefit of using the mixed reality COP (80) is that association between the destination floor information and tracked passenger (40b) may be directly established based on analyzing the obtained video image without a need for making assumptions for example based on timing of pressing of a conventional COP and simultaneously detected position of a passenger in the obtained video image. When multiple passengers enter the elevator car at the same floor, each new passenger may be simultaneously provided with their own mixed reality COP (80). Thus, there is no need for the passengers to use the conventional COP one by one, or to approach the conventional COP for checking whether their destination floor has already been selected, but each passenger may select their own destination floor in touchless manner independent of actions of other passengers.

In one alternative implementation, the mixed reality COP (80) is only displayed by the display apparatus if tracking of the passenger (40b) indicates that the passenger does not and/or did not use the physical COP in the elevator car for selecting a destination floor and/or no preregistered destination floor information for the new passenger is provided by some other means. Pre-registration of destination floor information may be provided for example by a passenger using a destination operating panel (DOP) at the landing when placing an elevator call, or the passenger may have used a mobile application in his/her personal mobile device for determining a destination floor. In such case, the association module receives destination floor information from the elevator control system for the new passenger entering the elevator car, and the association module may directly associate the new passenger with the destination floor information, who is tracked by the tracking module.

Other mixed reality services that may be provided by the system according to the invention especially in times of a pandemic or epidemic are facilitating social distancing and use of protective face mask. A computer imaging system may analyze the obtained video image for determining which passengers wear face masks and which passengers do not. Based on this determination, the mixed reality module may provide visual indications in the display apparatus that highlight passengers who do not wear an appropriate face mask, which facilitates maintaining social distancing within the elevator car between the passengers, and/or the display apparatus may display mixed reality elements which remind the passengers of need or requirement to wear a face mask or other equivalent protective gear within the elevator car, when several people are present.

The figure 9 illustrates another type of mixed reality service provided by the mixed reality module. For reducing anxiety of persons travelling in the elevator car, various types of entertainment may be provided by the mixed reality module. For example, the mixed reality module may provide one or more interactive mixed reality elements (90), which may provide entertainment for the passengers (40). In this example, the interactive mixed reality elements (90) are represented by butterflies, which can be virtually "touched" by the passengers by hand gestures mid-air, which hang gestures can be detected by the computer imaging module. The mixed reality elements (90) can thus be made to react to the virtual touching for example by a change in the way and/or direction they move. A playful experience provided by mixed reality may reduce anxiety during the travel. Another example of an entertaining mixed reality service may be displaying a smiley next to or at of the respective images representing passengers shown by the display apparatus, wherein the facial expression of the smiley may be altered based on determining by the computer imaging module a facial expression of the respective passenger and superimposing by the mixed reality module a respective smiley character next to the displayed image showing or representing the respective passenger. Another entertainment service may be kind of augmented reality, in which the image of the passenger shown by the display apparatus is augmented with some additional equipment or clothing, such as a hat or a helmet for example that is generated by the mixed reality module and positioned with respect to the image shown on the display apparatus based on the position information obtained by analyzing the obtained video image. It is apparent to a person skilled in the art that as technology advanced, the basic idea of the invention can be implemented in various ways. The invention and its embodiments are therefore not restricted to the above examples, but they may vary within the scope of the claims.