TITLE

Method and user device for detecting an environment of the user device

TECHNICAL FIELD OF THE INVENTION

The main subject described herein relates to methods for scene analysis, cooperative driving, and advanced adaptive displaying of AR information within a vehicle. More specifically, scene analysis methods are used to identify and categorize (in different semantic classes and levels of importance) the objects of interest that drivers must be aware of their existence in a scene regardless of whether they can see them or not, due to physical limitations. Cooperative driving methods enable for a better and more complete understanding of the driving scene, combining data from different spatial sources, creating also, in this way, more accurate representations. Finally, the provided AR information is related to visual hints and indicators about partially or fully occluded objects of interest (Ool).

BACKGROUND

Information-centric technologies start to play a central role in the recent automotive industry boosting new research trends in semi or fully automated driving systems. Autonomous vehicles, ranging from level 3 to level 5 of autonomy, are expected to safely operate in real-life road conditions. Nevertheless, in the real world, where the number of vehicles and other road users continuously increases, the challenges and dangers in a mixed traffic environment are also increasing. To overcome these challenges, the automotive industry has set as a primary objective to find new ways for reducing accidents and their severity. One factor that may play a crucial role, in that di rection, is to highlight the information related to the occluded objects and the road obstacles that exist in the scene but they are not visible by all drivers and as a result, they may cause accidents. For this reason, the detection, identification, and early visualization of these objects are imperative for the safety of the drivers helping them to know, in an earlier temporal stage, what will be presented next, making them be aware and better prepared for a future event.

The challenge of object detection and recognition is commonly targeted using imaging data (RGB cameras) and computer vision techniques. Although image-based techniques have achieved great success, one common drawback is that they are sensitive to motion blur and changes in lighting and/or even shadows. Additionally, useful information related to the depth of the objects is not available. This can make them unreliable in real use cases, being a major weakness in problems involving human safety. In light of all this, the use of a 3D LiDAR (Light Detection and Ranging) sensor provides more robust sensing capabilities for scene analysis, as well as depth information about objects indicating in this way their proximity to the sensor. Nonetheless, no sensor is able to acquire information that is beyond a particular distance. Additionally, the field-of-view of any sensor is limited and additionally, it can be obstructed by other static or moving objects. Besides these drawbacks, the good, adequate and complete knowledge about the surrounding environment is important not only for the driver but also in a future scenario for the autonomous high-level vehicles that need to perfectly know everything about their environment so that to safely navigate, avoid collisions with objects, plan the best and safest path, etc. One of the advantages of autonomous vehicles is their ability to communicate with each other vehicles forming a cyber-physical system of systems. Many new opportunities arise from the ability of systems to share information, one of which is the transmission of information related to objects of interest that were previously observed by an agent, to other agents of the system who could benefit from such information.

AR systems are used for presenting virtual information to users, superimposing reality with virtual content. For this purpose, a range of different devices can be utilized such as smartphones/tablets, smart glasses, head-mounted displays (HMDs), windshields. In the area of automotive, inherent challenges include the need for non-intrusive information display, avoiding the effects of tunnel vision which could lead to overlooking important information. AR-based technologies (beyond current AR headsets) are expected to be utilized in the near future for providing guidance to the driver, increasing situational awareness, and facilitating cooperation with other vehicles and road users (e.g., pedestrians, cyclists, bicycles).

DISCLOSURE OF THE INVENTION

A method for detecting and presenting an environment of a user device according to the invention, comprises the steps of capturing the environment of the user device by a sensor unit of the user device, receiving remote information from a remote system by a wireless communication system of the user device, wherein the remote information comprises information that is related to adjacent objects, which are present in a surrounding area of the user device, and determining characteristics of objects of interest in the surrounding area of the user device using both of the sensor data that are provided by the sensor unit and the received remote information by the user device.

A user device for detecting and presenting an environment of a user device according to the invention, wherein the user device comprises a sensor unit for capturing the environment of the user device, a wireless communication system, configured for receiving remote information from a remote system of the user device, wherein the remote information comprises information that is related to adjacent objects, which are present in a surrounding area of the user device, a processing unit, configured to determine characteristicsof objects of interest in the surrounding area of the user device using both of the sensor data that are provided by the sensor unit and the received remote information by the user device. The user device is an entity that is capable to perform the method according to the invention. Preferably, the user device is a system of a vehicle or a movable unit. The vehicle is in particular a car or motorbike. The mobile unit can be a smartphone or tablet, which would allow mounting the mobile unit for access on any vehicle.

An environment of the user device is captured by a sensor unit of the user device. Typically traffic actors are present in the environment. A sensor unit can comprise one or more sensors of the same or different types. The sensor unit preferably comprises a LiDAR sensor and/or a camera. The sensor unit is arranged to capture an area that is next to the sensor unit and therefore next to the user device. Due to limitations of the sensor unit, it is possible that there are objects in the surrounding area of the user device, which cannot be detected by the sensor unit, for example because these objects are not visible to the sensor unit due to an obstacle in the field of view of the sensor unit or due to a limitation in the range of the sensor unit. In other words, the sensor data of the sensor unit is merely comprising information in respect to the parts of the environment of the user device that are visible for the sensor unit. In the surrounding area of the user device there can be objects that are not located in the parts of the environment of the user device that are visible for the sensor unit.

Remote information is received from a remote system by a wireless communication system of the user device. The remote system is a system that exists in the surrounding area of the user device and is located at a different position than the user device. Therefore, sensors of the remote system can likely detect information that might be hidden for the sensor unit of the user device, although the object is in the surrounding area of the user device. For example, objects that are hidden for the sensor unit of the user device might be captured by a senor unit of the remote system. Also, the remote system itself can be an object in the environment of the user device and information about the remote system itself is known to the remote system but can possibly not be captured by the sensor unit of the user device. Therefore, additional information is provided from the remote system to the user device. The remote information comprises information that is related to adjacent objects, which are present in a surrounding area of the user device. The adjacent objects can be inside or outside the area that is visible for the sensor unit of the user device.

The positions of objects of interest in the surrounding area of the user device is determined using both of the sensor data that are provided by the sensor unit and the received remote information by the user device. That is, the knowledge/information about objects in the surrounding area of the user device that is acquired by the remote system and the sensor unit of the user device is combined. Therefore, additional information is available that increases the options for detecting objects of interest and characteristics of the objects of interest. The size of the surrounding area does not have any specific limitation but might be limited by technical considerations. An object of interest can be any object and different conditions can be set for determining whether an object is an object of interest or not. In particular, any object that might be relevant for a save movement of the user device can be an object of interest.

For determining the characteristics of objects of interest, it is advantageous that the remote information comprises information that describes the characteristic that is determined for detected objects.

For example, when determining a position of objects of interest as a characteristic, it is advantageous that the remote information comprises information that describes a position of detected objects. This way, a relative position of such an object in relation to the user device can be calculated, which allows to present this object in a proper position by the augmented reality system. However, it is not mandatory that the remote information comprises information that describes a position of detected objects. For example, the position of an object might already be known to the user device but the remote information describes whether the object is present or not as the characteristic of this object. For example, the object might be a construction site or pot hole that has been previously detected by the user device. In this case, it is sufficient that the remote information is indicating that the construction site or pot hole is still there.

The dependent claims define preferable embodiments of the invention.

Preferably, at least some of the objects of interest are presented to a user by using an augmented reality system, wherein each one of the objects of interest is presented on a position of a display of the augmented reality system that corresponds to the respective determined position of the object in the real world. At least some of the objects of interest are presented to a user by using the augmented reality system. The presented objects can be objects that are detected by the sensor unit of the user device or that the user device has knowledge of because of the remote information. Each one of the objects of interest is presented on a position of a display of the augmented reality system that corresponds to the respective determined position of the object in the real world. That is, a user is given the impression that an object of interest is visible when watching the traffic scenario. In particular, an object of interest can be presented to be visible, even though the object is covered by an obstacle in the real world from a point of view of the user. Also, an object of interest that is visible in the real world from the point of view of the user can be marked by an indicator to present the object to the user. Therefore, a user of the augmented reality system is made aware of relevant objects in his vicinity. An augmented reality system, also referred to AR system, is a system for providing AR visualizations. Accordingly, the method is preferably a method for detecting and presenting an environment of the user device.

Preferably, the characteristic is a position of the object in the real world and the position on the display for presenting the object of interest is derived from the characteristic.

For determining the position of objects of interest, it is advantageous that the sensor data of the sensor unit comprises information that describes a position of detected objects. This way, the position of objects of interest can be determined for some object only based on the sensor data of the senor unit and can be determined for other objects of interest only based on the remote information. Also, the position of one single object of interest can be determined by combining information about the position of this object that is determined by the sensor unit and information about the position of this object that is provided with the remote information.

Preferably, the user device is further determining additional characteristics of the objects of interest using both of the sensor data that are provided by the sensor unit and the received remote information, wherein in particular a shape, type, color and/or size of an object of interest is determined. This allows to combine further information that is provided by different resources. In particular, it is advantageous when different characteristics of one object of interest is determined using the sensor data of the sensor unit and the remote information, respectively. Such additional characteristics of the objects of interest can be used for various purposes and are in particular used for presenting the objects of interest to a user by using the augmented reality system.

Preferably, the method further comprises transmitting remote information to a remote system by the wireless communication system of the user device, wherein the remote information comprises information that is related to adjacent objects, which are present in a surrounding area of the user device. In particular, the remote information comprises information that is based on the sensor data of the sensor unit. Optionally, the user device is a system of a vehicle, and the remote information is defining a status of the vehicle, in particular a position, speed, or steering angle of the vehicle.

Preferably, the augmented reality system comprises a set of AR-Glasses or a Heads- up-display for presenting the objects of interest to a user.

Preferably, information that is referring to the same object of interest in the sensor data that are provided by the sensor unit and the remote information is detected based on a saliency value that is generated for each object of interest based on the sensor data and a saliency value that is generated for each object of interest based on the remote information or that is provided with the remote information. Saliency estimation is used for the extraction of values per each point of a point cloud. These values represent the geometrical importance of these points (bigger values correspond to more important points). Geometrically important is assumed to be a point that lies in a sharp area (e.g., a corner), less important is a point that lies on an edge and nonimportant is a point that lies in a flat area. A point cloud is preferably provided with the sensor data of the senor unit, in particular by a LiDAR Sensor of the sensor unit. Thus, one or more saliency values can be calculated that can be used as an identifier of an object that is captured by the sensor unit. Accordingly, one or more saliency values can be generated by the remote system and can be communicated with the remote information or can be generated from sensor information that has been received with the remote information. An object of interest can be identified by the saliency value, which allows to identify that sensor data and remote information is referring to a same object based on these saliency values. The saliency mapping can be used as a descriptor for the registration of two or more point clouds, for example in a data base.

Preferably, the remote system comprises a further vehicle or a traffic infrastructure, wherein the further vehicle or a traffic infrastructure is generating the remote information. That is, remote information can be received from other vehicles or can be received from adjacent traffic infrastructure. Therefore, a large source of remote devices can be provided.

Preferably, the remote information comprises sensor data and is received directly or via a server from the further vehicle or the traffic infrastructure. Sensor data is information that is provided from sensors of the remote system. This allows that a very short propagation delay is achieved when communicating the remote information. Therefore, a position or other characteristic of the adjacent objects can be quickly considered when determining the objects of interest and can be precisely displayed via the augmented reality system, in particular in a case in which the position for displaying an object of interest is calculated based on the remote information. Providing the sensor data via a server is advantageous, as a time for establishing a direct communication can be reduced.

Preferably, the sensor data is image data that is provided by a camera system or a set of point clouds that is provided by a LiDAR system. Such sensor data allow that a further analysis is performed on the side of the user device. In addition or in the alternative, the sensor data comprises data like the GPS location, the steering angle of the vehicle, its velocity etc. However, Image and point cloud are preferably determined and used for capturing the visual part of the scene

Preferably, the remote information comprises information that describes characteristics of objects that are present in a surrounding area of the user device. Preferably, the characteristics are communicated in combination with an object identifier, which allows to assign further information to objects of interest that have already been detected based on the sensor information that is provide by the sensor unit of the user device. An analysis for determining the characteristics of objects can be performed on the remote system, which decreases calculation requirements on the user device. Also, such information can determined once and can be stored for later use. Accordingly, it is advantageous that the characteristics are provided in combination with a time stamp in the remote information. The iinformation that describes characteristics of objects that are present in a surrounding area of the user device are preferably directly defining this characteristic, for example as a value, wherein in particular no further analysis is necessary to extract the characteristic from sensor data.

Preferably, the characteristics of the objects describe one or more of the following a saliency value of the object, a classification of the object, geometric characteristics of the object, a velocity of the object, a moving direction of the object, a position of the object, and/or a timestamp, which defines a time of detection of the object by the remote system. A saliency value is an indicator that represents the importance of an object in a view of a scene. The classification of the object defines a type of the object, for example if an object is a vehicle, a pedestrian, a traffic light or similar. The geometric characteristics in particular define a size, dimensions silhouette and/or shape of the object. A characteristics of an object that is stored on the server can be set to expire based on the timestamp.

Preferably, the remote system comprises a database, wherein characteristics of multiple objects are stored in the database and wherein the characteristics of selected objects are transmitted from the database to the user device as remote information in response to a request from the user device, and/or characteristics of objects are added or updated in the database by the user device, wherein the characteristics of the added or updated objects are derived from the captured traffic situation by the user device. The database can be installed on a server or in the further vehicle or traffic infrastructure. Thus, either a centralized or decentralized approach or a combination of both can be applied. With the server, it is possible to provide an entity that is buffering information that is related to different objects and that can be requested and received by the user device when needed. Therefore, a time delay can be present in between storing information on the server by the remote system and transmitting this information as remote information to the user device. This leads to the effect that it is not necessary that an object is captured by the sensors of a vehicle or traffic infrastructure of the remote system at the same time at which the information is required by the user device, which is in particular suitable for characteristics that are not quickly changing over the time. Same effect can be achieved when providing the database or multiple databases on the further vehicle or traffic infrastructure.

Specifically a decentralized database that comprises characteristics of the objects can be provided by the combination of multiple vehicles and/or infrastructure that is capable to provide remote information. Preferably, the objects of interest are presented by displaying a corresponding graphical element on the display of the augmented reality system. The graphical elements are VR information for displaying objects. This is advantageous, as the exact appearance of an object is not necessarily known to the augmented reality system, as characteristics of the object, including the position of the object are might be known, but the exact appearance of the object of interest might not be known. Therefore, it is advantageous to present the object by displaying a graphical element, which is preferably selected based on the characteristics of the object that is represented by the graphical element and which are preferably received with the remote information. Exemplary graphical elements are dots, arrows and icons of any type.

Preferably, a shape, size and/or colour of the graphical element is selected based on remote information. The shape, size and/or colour of the graphical element is preferably selected based on the characteristics of the object that is represented by the graphical element. Also, an information that is displayed together with the graphical element is preferably selected based on the characteristics of the object that is represented by the graphical element.

Preferably, a presentation mode of the augmented reality system is configurable by a user according to at least one of the following options: setting a presentation mode for different presentations of the objects of interest, setting light conditions that can affect the transparency and the intensity of the information that is presented by the augmented reality system, setting a save mode for limiting the number of displayed objects of interest, setting a quality level, for limiting a quality of the presentation of the objects of interest by the augmented reality system, setting a prediction mode, which is limiting the displayed objects of interest to such objects that will be within a certain distance towards the user device in the future. Thus, the augmented reality system can be set to present the objects of interest according to a user taste. Preferably, a priority level is assigned to the objects of interest based one or more priority criterions and displaying only the determined objects of interest to the user having a priority level that is above a priority threshold level. That is, if all objects that are detected by the user device would be displayed simultaneously to the user via the augmented reality system, the user would not be able to process this load of information. Therefore, it is advantageous that only the most relevant objects of interest are presented to the user, which can be done by assigning a priority level and presenting only the objects of interest with a specific priority level or higher.

Preferably, the one or more priority criterions are set up to increase the priority level of an object of interest, if the object of interest is a specific type of object, a moving object, in a position that is within a predefined range of a trajectory of the user device, or within a specific range from the user device.

Preferably, the priority threshold level is adjustable by a user. That is, depending on an actual situation, the user might select to see more or less information presented via the augmented reality system.

Preferably, an object identifier is created for each determined object of interest and any remote information and sensor data that is related to an object of interest is identified by assigning the object identifier to the respective object of interest. In other words, a system is created that comprises an object based data analysis. Due to the object identifier, object information can be shared by different systems, for example by the remote system and the user device. At any point of time, object information can be added, deleted or updated using the object identifier. For example, a change of an indicated sign of a traffic light can be communicated in the remote information to the user device using the object identifier. In this case, it is not necessary to communicate any additional information that might not have changed, for example the position of the traffic light in the real world. Systems, methods, and other embodiments described herein relate to acquiring, processing, transmitting, and finally displaying data (using Augmented Reality (AR) technologies) to the drivers during their driving process in a mixed traffic environment (in real-time). These data represent information related to static or moving "objects of interest" (Ool) (such as traffic signs and lights, pedestrians, vehicles, potholes, etc) occluded by other static or moving objects (buildings, trees, vehicles, etc) in a dynamically changing scene. The method includes acquiring data, by different types of sensors (RBG camera, LiDAR, etc), indicating information (such as color, position, dimensions, shape, etc) of Ool located in a nearby and visible area of the vehicle equipped with these sensors. The method includes the processing (fusion, denoising, segmentation, compression, saliency-based registration, completion, etc) of these multi-sensory data per agent (i.e., data come from the internal ego-vehicle sensors) and per Ool (i.e., data corresponding to the same Ool that come from different external sources via cooperative driving), meaning that the same Ool can be partially or complete captured by the sensors of different vehicles. The method includes the transmission of the proceeded data (in an appropriate form) to the road infrastructures via a communication protocol (Vehicle-to-lnfrastructure (V2I) communication) for further processing. The method includes the identification of occluded objects, in a surrounding area of the vehicle's environment, and appropriately displaying them in an AR environment, allowing in this way the drivers to see behind objects.

A preferable system, comprises multiple user devices according to the invention, wherein each of the user devices is configured to provide remote information for the other user devices.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 illustrates an example of real-time AR visualization of synchronous content.

Figure 2 illustrates an example of real-time AR visualization of asynchronous content. Figure 3 illustrates an example of the pipeline for the identification and classification of objects of interest.

Figure 4 illustrates examples of different approaches of AR visualization.

Figure 5 illustrates examples of different reasons that affect the adaptive visualization of the VR information.

Figure 6 illustrates examples of different approaches for the visualization of the identified objects of interest based on different parameters.

Figure 7 illustrates an exemplary flowchart for the method according to the invention.

Figure 8 illustrates a user device according to the invention.

METHODS DESCRIPTION

This summary is provided to briefly describe the characteristics of the invention and to introduce the main concepts that will be further described, in more detail, in following section "DETAILED DESCRIPTION".

An exemplary flowchart for the method 10 according to the invention is illustrated in Figure 7. The method 10 comprises the steps of Capturing 11 a traffic situation by a sensor unit of a user device, wherein the traffic situation is representing an environment of the user device, receiving 12 remote information from a remote system by a wireless communication system of the user device, wherein the remote information comprises information that is related to adjacent objects, which are present in a surrounding area of the user device, determining 13 characteristics of objects of interest in the surrounding area of the user device using both of the sensor data that are provided by the sensor unit and the received remote information by the user device, wherein the characteristics are positions objects of interest, and presenting 14 at least some of the objects of interest to a user by using an augmented reality system, wherein each one of the objects of interest is presented on a position of a display of the augmented reality system that corresponds to the respective determined position of the object in the real world. The method is executed by a user device 20, as depicted in Figure 8. The user device 20 comprises a senor unit 21 for capturing a traffic situation, a wireless communication system 22, configured for receiving remote information from a remote system of the user device, wherein the remote information comprises information that is related to adjacent objects, which are present in a surrounding area of the user device, a processing unit 23, configured to determine characteristics of objects of interest in the surrounding area of the user device using both of the sensor data that are provided by the sensor unit and the received remote information by the user device, wherein the characteristics are positions objects of interest, and an augmented reality system 24, configured to present at least some of the objects of interest to a user, wherein each one of the objects of interest is presented on a position of a display of the augmented reality system that corresponds to the respective determined position of the object in the real world.

The method 10 concerns cars of level 2-5 automation and can play an important role in increasing the trust and acceptability of the drivers for the new technologies that are a necessary condition for the final development of fully automated solutions. The method 10 solves the problematic state where drivers have no (or only partial) knowledge about the accurate position of moving and/or static objects which are relatively close to them but are partially or totally hidden because of other objects. This partial knowledge comes as a natural result due to the limited driver's and sensors' field of view. Objects of interest, also referred to as Ool, are assumed objects (includes other vehicles, pedestrians, obstacles, etc.) that are in the vicinity of a vehicle but not immediately visible to the driver, due to obstruction by other static and/or moving objects (buildings, trees, other larger vehicles. The interest in these objects lies in the fact that they could potentially be involved in an accident with the driver. The Ool has to be observed by a vehicle's sensors (type A sender) in order to be displayed in another vehicle (type B receiver). Otherwise, the infrastructures can send information related to static objects in case where no vehicle exists so that to send contemporary information. Type A indicates vehicles equipped with sensors to acquire information from the physical environment and to send it to the road infrastructure. Type B indicates vehicles equipped with AR displaying devices capable to render the AR information of the occluded objects. A vehicle can be both of Type A and B regarding if it sends or receive information.

The method 10 offers a future insight into a driving scene that the driver will encounter in the next few seconds, providing to driver with more time to act or make better decisions about a forthcoming event. Drivers can be warned about potential hazards that may not yet be in the field of view or they are obstructed and thus increase their performance and decision-making abilities. The method 10 increases the driver's situational awareness through automated Ool detection, visualization, and information sharing to other connected vehicles. The method can be used, in real-time, for cases where the user drives in mixed traffic and dynamically changing environments.

A step of the method comprises determining 13 the positions of objects of interest in the surrounding area of the user device using both of the sensor data that are provided by the sensor unit and the received remote information by the user device.

This step includes the following basic steps:

(i) Saliency map extraction of the capture 3D scene where the saliency value is estimated at any point of the scene based on its local geometry, as well as the geometry of neighbouring points.

(ii) Objects of interest are identified and the relative information or characteristic (e.g., position, coordinates of the vertices constituting the point cloud, saliency values as geometric unique characteristics of the object, timestamp of the observation, etc) are stored and after further processing (fusion of data with the complementary information of other vehicles pursuing better results) can be transmitted to other nearby vehicles for the AR-based visualization.

(iii) The proceeded information of the detected objects is projected in the AR coordinate system as 2D transparency images.

Saliency estimation is used for the extraction of values per each point of a point cloud. These values represent the geometrical importance of these points (bigger values correspond to more important points). Geometrically important is assumed to be a point that lies in a sharp area (e.g., a corner), less important is a point that lies on an edge and non-important is a point that lies in a flat area. The saliency mapping can be used as a descriptor for the registration of two or more point clouds. These point clouds represent the visual information of the same scene that however has been captured by different LiDAR devices that are located in different positions and from different views. The saliency mapping as a descriptor is utilized to distinguish and highlight important areas of a geometry surface and this can be used then for identifying coherent features and areas between the different point clouds of the scene. Additionally, saliency mapping can facilitate the process of registration, enhancing the performance and decreasing the computational time, since only the most important, coherent and recognizable information is used for the registration.

The vehicle includes sensors by multiple sources (e.g., LiDAR, cameras, etc.) that may acquire a large number of data for the local environment of the vehicle. These sensors are part of the sensor unit 21. Multi-sensor-based detection methods are used to collect and proceed with the information of the environment. The method provides extensive scene analysis and understanding multi-processing approaches, including, among others, object recognition (vehicles, pedestrians, cyclists, traffic signs, potholes, obstacles) for a safe, realtime, and robust identification under a wide range of different and challenging conditions (weather, light, traffic, etc.), where the information could receive through multi types of sensors, such as camera (image/video analysis), LiDAR (point cloud analysis). To further increase the accuracy of the results, the method 10 takes also into account the cooperative information of other vehicles. The cooperative information is given by the remote information. To achieve this, the advanced saliency-aware analysis of the scene (per each agent) is combined with those of other neighbouring vehicles and then a real-time registration of the point clouds is performed so that all the relative cooperative information, received by different coordinate worlds, are finally integrated into the same coordinate system so that to be efficiently represented into a common environment. To notice here, that the information after the aforementioned processing (via the cooperative Simultaneous Localization and Mapping (SLAM)) is everything that needs a completely autonomous vehicle (level 5) to have perfect knowledge about its surrounding environment (what it can "see" by its own sensors and what it can "sense" by the sensors of other vehicles) in order to perform a safe and secure autonomous driving without the need of human intervention. The method uses a multi-agent system for processing and sharing information between vehicles (collaborative driving between connected vehicles (V2V) or between vehicles and infrastructure (V2I )) so that vehicles can be notified of incoming Ool at any time even when there is no direct line of sight. In this way, the information of an invisible object from vehicle will be received and displayed (via a VR, virtual reality, representation) in an AR interface, due to the fact that other vehicle(s) can see it via its/their sensors. This is illustrated in Figure 1.

The method 10 focuses on sharing information about Ool to other vehicles through centralized infrastructures (spatiotemporal information). The vehicle transmits the proceeded information, related to the detected Ool, to the road infrastructure. The information is coupled with a timestamp and the GPS location of the vehicle at that instance. The information is combined, proceeding, and continuously updated with any other relevant information that has been or will be received by other vehicles providing more accurate results. This is depicted with Figures 1 and 3.

When a vehicle identifies an Ool, the vehicle sends a request to the infrastructures and, after further inspection, the new Ool is either discarded or added. Vehicles may also send information regarding already known Ool when they come across them, in case a static object needs updating in the database, e.g. it has increased in size or has been fixed. The method provides real-time AR visualization of content both in a synchronous and asynchronous mode. The synchronous mode is related to spatiotemporal information which consists of static and dynamic (i.e., moving) Ool. This mode requires the presence of other vehicles, equipped with sensors, in the same area and at the same moment. The asynchronous mode is related only to spatial information (static objects), providing to the drivers the stored information in cases where other vehicles are not available, to this area at the specific moment, to capture occluded information. This is depicted with Figure 2.

The road infrastructure transmits to any vehicle, in the vicinity of the Ool, alerting via AR visualization (autonomous vehicles or human operators) about potential hazards from a large distance and thus helping alleviate the inability of the sensors to identify objects from such range. There is a need for periodical evaluation of the static objects and updating of the relative information in the case of changes (potholes being repaired or worsened). For the evaluation, a geometry-based descriptor, representing each static object, is used. Such descriptor describes characteristics if the static object and is communicated as remote information. Thus, every vehicle encountering the static object at a satisfying range calculates the descriptor of the object's surrounding area. The new descriptor is then transmitted to the infrastructure and is used to confirm whether the information is up-to-date or should be updated.

Depending on the setup of the wireless communication system 22, state-of-the- art tools and equipment like mobile edge computing, MEC, V2V, and V2X communication can be utilized for the transmission of information between spatial neighbouring vehicles. Also other novel technologies like orchestrators controllers can facilitate the communication extending the range of services and functionalities, enabling increased connectivity and interoperability between infrastructure elements and vehicle devices, streamlining the processing and analysis of large volumes of data in a structured and integrated way, and acting faster and safer in the face of road events. The connection of infrastructures and vehicles is on the basis of the development of standards in the field of vehicular communications (ETSI TC ITS, ISO TC204 WG16 /CALM, IEEE 802.11p, 5G).

The information of the occluded Ool is presented to the driver via AR interfaces so that the driver is able to see useful information of objects that are behind other objects. The visualization can be used to facilitate different use cases, (e.g., self-driving including utilities and quality of life applications, handover scenario where the driver resumes manual control of the vehicle, etc). For moving Ool, a virtual representation of the Ool will be displayed as the object is moving. When the available visual information (VR transparency) is not accurate or a lot of visual parts of the virtual object's shape are missing (due to the sensor's faults, partial visibility of the objects), it is preferable to provide this information using other alternatives visualization approaches like arrows in different colors for different objects, radar-like information, default shapes per object, bounding boxes, etc. This is illustrated in Figure 4.

The visualization of Ool is performed by projection when presenting 14 at least some of the objects of interest to a user by using an augmented reality system. Assuming that the position is known, for the LiDAR relative to the world, a transformation matrix can be constructed transforming the points of the point cloud from the LiDAR relative coordinate system to the AR interface's coordinate system. The transformation between two different coordinates systems is typically done by applying serially in series a scale, a rotation and then a translation transformation. Since both coordinate systems are orthonormal, the scaling can be omitted. Afterward, for projecting the points of the point cloud to the AR interface of the augmented reality system 24, we assume a simple pinhole camera model. If the AR interface is, for example, an AR windshield, then the windshield represents the image plane and the head of the driver the principal point with coordinates (xO,yO). That way, the focal distance f represents the distance from the driver to the image plane. With the dimensions of the image plane (windshield), and specifically the aspect ratio, known, the frustum is fully defined and the projection can be made from a point in 3D windshield coordinates (x,y,z) to pixels (u,v) on the image plane using the following equation:

An undesirable property is the sparsity of the projected pixels attributed to the sparsity of the point cloud. To overcome this limitation, iterative nearest neighbour methods are used on the image space to fill the gaps between projected points.

The method 10 utilize AR interfaces for the presentation (via virtual transparent layouts) of information related to partially or completely occluded objects (moving and static). The AR interfaces of the vehicle display, in a non-distracting manner, the location and nature of the potentially upcoming Ool. The VR information can be displayed to any device that uses AR technology (Head-up display (HUDs), smart glasses, etc.) since the method has not any special requirements. The VR information is presented in such a way that it is understandable and explanatory to any driver without disturbing him/her and negatively affecting their safety during the driving process.

The way that the method follows to visualize the VR content, representing the information of the occluded Ool, can be affected by different reasons (adaptation of the visualized information). This is depicted in Figure 5. More specifically, the method 10 can follow different visualization approaches (e.g., the type of information, the level of importance, the contrast of transparency, the level of details, etc) depending on various circumstances that may occur in real cases, trying however to stand in the most efficient and safe way for the driver. The level of transparency will be automatically adjusted based on light conditions and the driver's head direction. Different types of visualization approaches can be available ensuring the driver's personal preferences (bounding boxes, arrows of the location of the occluded object, etc) and following principles like understandability by all users and free of biases (genders, ages, education, culture, etc). The visualized information has to follow some rules like that it is presented only what is to increase the safety and awareness of the driver, avoiding the overload of visual content that may be distractive for the driver. There is a prioritization of the information, meaning that some Ool are more important to be presented by others. Different levels of details can be set by the user's preferences and it is related to the type of objects that they want to be displayed. Static objects: range of the road, traffic signs, etc, Semi-static objects: potholes, etc, Dynamic objects: pedestrians and other road users. The method can display only the most relevant occluded content based on the driver's near future driving intention (predicted or via pre-defined path planning) to avoid overload and potential obstructive information.

DETAILED DESCRIPTION

Various embodiments and advantageous details of the above described method 10 and user device 20 are described in further detail below.

With reference to Figure 1, the component for the realtime AR visualization of synchronous content 100 via the augmented reality system 24 is presented. A vehicle 110, which is assumed as the sender of synchronous information, is equipped with external LiDAR and camera sensors 120. The sensors 120 continuously acquire data from the surrounding environment, such as point clouds and images 130 (in separated files per frame). These data 130 represent the captured information of the real scene which is in the range of the sensors' field of view.

Additional data, like the steering wheels data 140 captured by the internal sensors of the vehicle, are also utilized, in combination with the point cloud and the images 130, for the scene analysis (i.e., saliency-aware mapping and scene registration) 160, via fusion of visual and cooperative localization methods, in order to recognize and identify the static or dynamic objects of interest 170 of the scene. The GPS location of the vehicle and the accurate time (timestamp) of the data acquisition 150 are also taken into account for the creation of the spatial and temporal "signature" of the data. The vehicle 110 constantly communicates with the road infrastructure 180 by sending and receiving information in multiple ways. For example, by sending information (such as the point clouds and images 130, the GPS location and time stamp 150) in a one direction communication, and by sending and receiving information for the cooperative scene registration 160 and the objects' recognition and identification 170 in a bidirectional way. All the relative information is stored/retrieved by the database of the cloud 185. Once the information is stored in the database of the cloud is available to be received as remote information and displayed in any neighbouring vehicle 195 which is in the same area at the same moment according to its GPS location and the timestamp 151. Before the final visualization, the information is appropriately adjusted, via an adaptive visualization component 191, in order to satisfy both personalized and security/safety requirements and finally is displayed in a device that allows the visualization of the VR information 190. It can be seen that the remote information comprises sensor data and is received directly or via a server 185 from a further vehicle, which is a neighbouring vehicle 195. The sensor data can be image data that is provided by a camera system or a set of point clouds that is provided by a LiDAR system.

With reference to Figure 2, the component for the real-time AR visualization of asynchronous and static content 200 is presented. This process takes place in two different temporal moments. First, a first vehicle (sender of information) 201 acquires and sends spatial information about static objects of interest to the cloud databases 260. This information comprises information that describes characteristics of objects can be provided as remote information to further vehicles, for example a second vehicle 240 that comprises the user device 20 in the future.

In a future temporal moment, the second vehicle (receiver of asynchronous information) 240 will receive and display the relevant information. The former describes an event in a foregone moment 250, while the latter describes an event in a forthcoming moment 251. More specifically, the process starts with the scene analysis 202 of the observed environment by the sensors of the vehicle 201. The process includes the saliency-aware mapping extraction, the cooperative scene registration 160, and the identification and recognition of Ool 170. If a static Ool has been identified 205, then this Ool is checked for if it is a new one or not 206. If the Ool is new, then the relative information is stored 207 in the database 212, otherwise, it is further checked for if the already stored information has the exact same characteristics as those of the observed 208. If not then the corresponding information is updated 209 in the database 212 (this means that the Ool exists in the database but it has changed e.g., repaired, destroyed, etc.), otherwise, the system stop looking for information in the specific frame and continues the scene analysis process trying to find an Ool to the next frame.

The cloud, in which each road infrastructure is connected with, includes four databases:

(i) The database with the cooperative driving information 210.

(ii) The knowledge database 211 is used for recognition, classification, etc tasks. This database includes pre-trained models and priors knowledge.

(iii) A database includes the spatial information related to static, semi-static identified objects (coordinates) 212, and

(iv) a database includes the spatiotemporal information for dynamically changed or moving identified objects (coordinates and timestamp) 213.

Therefore, the remote system comprises a database 260, wherein characteristics of multiple objects are stored in the database 260, wherein in particular the database that includes the spatial information related to static, semi-static identify objects (coordinates) 212 is such a database. Also, the database that includes the spatiotemporal information for dynamically changed or moving identified objects (coordinates and timestamp) 213 is a database that stores characteristics of multiple objects.

In the given example, the database 260 comprises characteristics of semi-static objects or moving objects that describe a spatial information, which describes a position of the respective objects. In addition, the database can comprise information that describes a saliency value of these objects, a classification of these objects, a geometric characteristics of these objects, a velocity of these object, a moving direction of these objects, a position of these object, and/or a timestamp, which defines a time of detection of these objects by the remote system.

The data, which are related to temporal information, is expired after some seconds 214 since they are useless after the pass of some time, in a dynamically- changed environment. On the other hand, a second vehicle 240 continuously makes queries 220 to the cloud database (through the connection with the road infrastructures), based on the spatial information of its location (regarding the GPS sensor and the results of cooperative localization methods), in order to retrieve any relevant information. The characteristics of selected objects are transmitted from the database 260 to the user device as remote information in response to a request from the user device, which is given by the queries 220.

If there is any static Ool in the specific geographic area 221 then it is appropriately displayed 230 in an augmented reality system of the vehicle 240.

With reference to Figure 3, a pipeline for the identification and classification of Ool 300 is presented, as preferably performed when determining the positions of objects of interest in the surrounding area of the user device using both of the sensor data that are provided by the sensor unit and the received remote information by the user device.

Firstly, all the required information is acquired by the vehicle's internal and external sensors 301. The information is used to estimate the saliency aware mapping of the scene 302. The saliency map is sent to the cloud, through a connection with the nearest road Infrastructure 320, for further processing (i.e., visual localization, cooperative localization, saliency aware registration, etc) 330 taking into account relative spatiotemporal and cooperative information that has been received by other neighbouring vehicles 350. The processing in 330 returns the registered scene and extra cooperative localization information 303, providing, in this way, better results (i.e., completion of partially occluded Ool) and more complete knowledge of the surrounding environment. Then, a scene segmentation 310 process takes place in order to facilitate the object identification 311 algorithms to extract possible objects of interest. To certainly identify if a possible Ool is really an Ool or not, the information, related to the possible Ool, is sent to the cloud 320, and then object recognition (retrieval, matching, etc) algorithms are utilized 340. The knowledge about the recognized object of interest 312 returns to the vehicle, labelled also with further information related to its classification in different types of categories like

(i) semantic classification (what type of object is (vehicle, pedestrian, potholes, cat, etc.)),

(ii) level of importance (e.g., a fast running vehicle very close to the ego-vehicle, the range of the road far away from the vehicle),

(iii) static or dynamic Ool, (iv) quality of the observation (as it has been reconstructed by the cooperative information), etc. 360.

With reference to Figure 4, different approaches of AR visualization 400 via the augmented reality system 24 are presented. In this example, we assume a mixed traffic situation including a number of objects of interest, comprising multiple vehicles 401, 403, 405, a pedestrian 404, and a cyclist 406 moving in different directions. All vehicles, existing in the same spatiotemporal area, can communicate with each other and exchange information using the road infrastructure 402. The ego vehicle 401 comprises the user device 20 and is not able to directly see the rest vehicles and the other moving objects of the scene, due to obstructive objects (e.g. buildings 407, 408). In this case, the vehicle 401 relies on the information that the right 403 and the left vehicle 405 can send as remote information so that to have a better understanding of its surrounding environment. The ego vehicle 401 can receive and "sense" this information. Additionally, the driver can see and be aware of this information while it is appropriately displayed on the augmented reality system 24. There are a lot of different display approaches that can be used apart from the transparency of the real reconstructed shape of the occluded object, which in many cases could be more distractive than helpful, due to the limited observation of the object by the other vehicles too. Some display approaches are by

(i) using default represented shapes per different types of objects 410 (e.g., one representative car icon for all cars),

(ii) using representative arrows with different colors (each color represents different types of objects) 420,

(iii) using bounty boxes visualizing also the dimension range of the object 430,

(iv) using a radar-like presentation to display information related to the position of the object 440. This, using one of the aforementioned options (i) to (iv), the objects of interest are presented by displaying a corresponding graphical element on the display of the augmented reality system. A shape, size and/or color of the graphical element can be selected based on remote information. For example, when using representative arrows with different colors and each color represents different types of objects, the type of object might be previously communicated to the user device 20 as remote information.

With reference to Figure 5, the adaptive visualization component is illustrated, showing the external parameters that may affect the way and the type of the presented VR information 500 on the augmented reality system 24. The 3D visualized VR information 501, which is rendered to the drivers' display device to increase their situational awareness, is adaptive and can be adjusted based on different parameters like

(i) Personalized preferences 510, related, for example, to the type of presentation (e.g., real reconstructed shapes, bounding boxes, default representative shapes, arrows in different colors, etc.) that is used for the display of the Ool. That is, a presentation mode can be set for different presentations of the objects of interest.

(ii) Light conditions that can affect the transparency and the intensity of the VR displayed content (520). That is, light conditions that affect the transparency and the intensity of the information that is presented by the augmented reality system can be set.

(iii) Safer mode - Less distraction, where only the less required information is displayed 530. That is, a save mode for limiting the number of displayed objects of interest can be set.

(iv) Quality of the acquired information representing an Ool 540. That is, a quality level for limiting a quality of the presentation of the objects of interest by the augmented reality system can be set.

(v) Prediction - Display only the relative information that will be encountered later 550. That is, a prediction mode, which is limiting the displayed objects of interest to such objects that will be within a certain distance towards the user device in the future can be set.

(vi) Scalability and prioritization 560.

The primary objective is the security and protection of the driver and, on the other hand, in cases that it does not affect the safety of the driver, the displayed information can follow the driver's personal preferences or priorities. By the aforementioned options (i) to (v), a presentation mode of the augmented reality system 24 is configurable by a user and, the augmented reality system can be set to present the objects of interest according to a user taste.

With reference to Figure 6, different approaches for the visualization of the identified Ool 600 are presented. In this example, we assume a mixed traffic situation which is similar to this that has been presented in Figure 4. Additionally, in this example, there is a static pothole 601, a traffic light 602, and a visible, by the vehicle 401, vehicle 603 which however will not be displayed in the AR environment of the augmented reality system, since the driver of the vehicle 401 has a direct view of this. The intention of the driver to turn right 604 is also known in this case. The displayed VR visual content may vary due to many parameters relying primarily on the driver's safety and secondly on the driver's preferences. Firstly, a case where all the possible information 610 of the identified Ool is presented to the AR environment of the driver. Alternative, other possible displaying approaches are: (i) to display only important information 620 (traffic light is not crucial information),

(ii) to display only dynamic Ool 630,

(iii) to display only Ool that the driver will encounter in a short forthcoming future moment (e.g., based on the path planning, or the Al prediction of the route, or by internal sensors showing that the driver turns on a flash to turn left or right, etc.) 640,

(iv) to display Ool based on prioritization (only the geometrically nearest Ool) 650,

(v) to display VR content in a safe mode which means the content to be the least possible distractive (the most important and most relevant information) 660.

Therefore, it can be understood that a priority level can be assigned to the objects of interest based one or more priority criterions and that only the determined objects of interest to the user having a priority level that is above a priority threshold level are displayed. The priority level can be set is a way that either one of the aforementioned criteria (i) to (v) is fulfilled.

To achieve this one or more priority criterions can be set up to increase a priority level of an object of interest, if the object of interest is a specific type of object, a moving object, in a position that is within a predefined range of a trajectory of the user device, or within a specific range from the user device. The priority threshold level is adjustable by a user.

List of reference signs

10 Method for detecting an environment

11 Capture the environment

12 Receive remote information from a remote system

13 Determine the characteristics of the objects of interest in the surrounding area

14 Present the objects of interest

20 The user device

21 Sensor unit

22 Wireless communication system

23 Processing unit

24 Augmented reality system that corresponds to the respective determined position of the object in the real world

100 Real time AR visualization of synchronous content

110 Vehicle (sender of synchronous information)

120 LiDAR and Camera Sensors

130 Point Cloud and images of the scene

140 Steering wheel data

150 GPS Location and time information from the sender vehicle

151 GPS Location and time information from the receiver vehicle

160 Saliency analysis and Scene registration processes

170 Objects of interest identification process

180 Communication Road Infrastructure

185 Cloud and cloud's databases

190 Visualization of the objects' information

191 Component for the adaptation of the visual VR content

195 Vehicle (receiver of synchronous information)

200 Real time AR visualization of asynchronous content

201 Vehicle 1 (sender of information)

202 Scene analysis process

205 Check if the received information is a static Ool or not. If a static Ool is identified, then the process continues to 206

206 Check if the received information of the static Ool is new. If the information is new then the process continues to 207, otherwise continues to 208

207 Store the information in the database

208 Check if the received information is the same with the stored. If it is not the same, then the process continues to 209

209 Update the information of the database

210 Database with the cooperative driving information

211 The knowledge database

212 Database with spatial information for static, semi static objects (coordinates)

213 Database with spatiotemporal information for dynamically changed or moving objects (coordinates and timestamp) The temporal-related information will be expired after some seconds

Make query based on the location of the vehicle

Check if there is any static object of interest in the specific area. If there is, then the process continues to 230

Visualize the information

Vehicle 2 (receiver of synchronous information)

Event in a foregone moment

Event in a forthcoming moment

The cloud

Component for the identification and classification of Ool

Information acquired by the vehicle's sensors

Saliency aware mapping estimation

Registration and cooperative localization

Scene segmentation

Object Identification

Recognized and labeled objects of interest are returned

Access to the cloud and communication with road Infrastructure

Processing and Cooperative information

Processing and object recognition and classification

Relevant information by other vehicles

(Semantic classification, Level of importance, Static or dynamic, Quality of the observation)

Different approaches of AR visualization

Vehicle receiver of occluded information

Road infrastructure

Right vehicle sender

Pedestrian

Left vehicle sender

Cyclist

Left building

Right building

Default represented shapes per different types of objects

Arrows with different colors representing different types of objects

Bounty boxes representing the dimension range of the object

Radar-like presentation to display information related to the position of the object

Adaptive visualization of the VR information affected by different reasons

Adaptive 3D visualized VR information

Personalized preferences

Light conditions

Safer mode - Less distraction

Quality of the acquired object of information

Prediction - Only Information that will be encountered later

Scalability and prioritization

Pothole Traffic light Visible car Intension of the driver to turn right All possible information Only important information (traffic light is not a crucial information) Only dynamic Ool Only Ool that the driver will encounter in the future (based on the path planning, or the Al prediction of the route) Prioritization (only the geometrically nearest Ool) Safe mode (the most important and most relevant information)