The invention relates to a method for providing visual information about at least part of an environment, the view onto which is at least in part concealed by a vehicle, on a display device, wherein the vehicle is detected by means of at least one sensor device and the visual information about the at least part of the environment, the view onto which is at least in part concealed by the vehicle, is requested by means of a communication unit. In case of an availability of the requested visual information, the visual information is received by the communication unit and is displayed on the display device. The invention also relates to a computer program product, a mobile communication device and a communication system. Systems for providing so-called see through functions in vehicles are known from the prior art. For example, US 2014/0341434 A1 describes a dynamic fusion method of images, where broadcast information from surrounding vehicles of a host vehicle is received, it is determined whether at least one of the surrounding vehicles travels in the same lane as the host vehicle, which is then defined as a neighboring vehicle of the host vehicle according to the broadcast information. Moreover, it is determined whether the neighboring vehicle is too close to the host vehicle and blocks the view of the host vehicle. Then a transparantization or translucentazation process is performed on the neighboring vehicle in an image captured by the host vehicle when the neighboring vehicle blocks the view of the host vehicle. Thereby the broadcast information may include both image and position information of the surrounding vehicles. Based on the position information it can be determined, which of the vehicles is a neighboring vehicle and the image content provided by this neighboring vehicle can be embedded in the images captured by the host vehicle. Especially in dense traffic situations the determination of the position of the surrounding vehicles with regard to the host vehicle has to be very accurate to be able to reliably determine the neighboring vehicle. However, e.g. GPS sensors might be too inaccurate for that purpose and other more accurate positioning systems are very expensive.

Moreover, those systems for providing see through functions require the vehicles to be equipped with several environmental sensors, such as cameras. Consequently, vehicles for providing such see through functions are very expensive. Therefore, it is an object of the present invention to provide a method for providing visual information about at least part of an environment, the view into which is at least in part concealed by a first vehicle, a computer program product, a mobile communication device and a communication system, which allow for a more cost-effective and still an as accurate as possible way to provide such a see through function.

This object is solved by a method, by computer program product, by a mobile

communication device and by a communication system with the features according to the respective independent claims. Advantageous embodiments of the invention are presented in the dependent claims, the description and the drawings.

According to the invention a method for providing visual information about at least part of an environment, the view onto which is at least in part concealed by a first vehicle, on a display device is provided, wherein the first vehicle is detected by means of at least one sensor device and the visual information about the at least part of the environment, the view onto which is at least in part concealed by the first vehicle, is requested by means of a communication unit. In case of an availability of the requested visual information, the visual information is received by the communication unit and displayed on the display device. Thereby, after detecting the first vehicle, a license plate of the first vehicle is detected and an identifier of the first vehicle is derived from the detected license plate, wherein the visual information is requested using the derived identifier of the first vehicle to identify the first vehicle, which conceals the view onto the at least part of the

environment, the visual information about which is requested.

So advantageously, the license plate of the first vehicle can be used as identifier to identify the first vehicle. The identifier can then be for example the license plate number or at least part thereof. For detecting the license plate also the at least one sensor device, which can be for example a camera, can be used. To derive the identifier from the license plate optical character recognition or similar image content analysis methods can be used. So, when a vehicle, in this case the first vehicle, is detected as well as its license plate, the visual information relating to this first vehicle can be requested in an easy way by using the identifier of the first vehicle derived from its license plate. Consequently, no position information has to be determined and exchanged to request the correct visual information relating to the vehicle, which is currently blocking the view. As detecting and recognizing a license plate as well as the license plate number thereon can be performed in a very cost-effective and still accurate way, the invention allows for providing see through functionalities in a very cost-effective and accurate way as well. Preferably, the first vehicle constitutes a vehicle travelling ahead of a host vehicle, which in the following is also called second vehicle. Moreover, advantageously, for performing the method a mobile communication device, like a smartphone, can be used, especially also several mobile communication devices, e.g. one associated with the at least one first vehicle and one associated with the second vehicle as explained in the following can be used.

So, according to an advantageous embodiment of the invention, the visual information is provided in form of at least part of a first video stream captured by a first device, which is associated with the first vehicle, especially a first mobile communication device within the first vehicle. For capturing the visual information advantageously a camera of the first device, especially the first mobile communication device, like a smartphone, can be used. For that purpose, the first device can be placed somewhere behind the windshield of the first vehicle. Preferably, the first device is positioned such within the first vehicle behind the windshield that the field of view of the camera of the first device covers at least part of the environment in front of the first vehicle. Moreover, at least one or several or each of the images of the first video stream can be displayed on the display device each as a whole or only a certain image region of at least one or several or each of the captured images of the first video can be streamed and/or displayed. For example only a central region of the images can be streamed and displayed on the display device. Moreover, it is also possible that the content of the received images or image regions of the first video stream is analyzed with regard to the image content of temporally corresponding images of a second video stream as described later, and only that part of the images of the first video stream is displayed, which is concealed by the first vehicle from the perspective of the at least one sensor device.

Moreover, according to another advantageous embodiment of the invention, the at least one sensor device and/or the communication unit and/or the display device are provided as part of a second mobile communication device, especially a smartphone associated with a second vehicle trailing the first vehicle. Moreover, this second mobile

communication device preferably comprises a processing unit, which is capable of controlling the at least one sensor device, which here again is preferably configured as a camera, the communication unit as well as the display device in such a manner, that the second mobile communication device is configured to perform the method as described before. Moreover, also the second mobile communication device may be configured such, that also the capabilities as described with regard to the first device can be provided by the second mobile communication device as well. In other words, also the second mobile communication device can be positioned behind the windscreen of the second vehicle such that the field of view of the camera of this second mobile communication device covers at least part of the environment in front of the second vehicle. The images captured by the camera of the second mobile communication device can on the one hand be used to detect the first vehicle travelling ahead as well as its license plate and identify the license plate number. And on the other hand, the images captured by the camera of the second mobile communication device can, if requested, be provided to other vehicles, e.g. a trailing vehicle, as visual information to provide see through functionalities for this trailing vehicle.

Advantageously, all necessary components, especially except for a central data processing unit as explained later, used to provide these see through functionalities can be provided in a cost-effective way by a mobile communication device and an application program running on that mobile communication device for executing the method according to the invention or its embodiments. Consequently, a vehicle does not have to be equipped with any kind of sensor, positioning system or other component to provide see through functionalities, but those see through functionalities can be provided simply by a mobile communication device. This way any arbitrary vehicle can be provided with see through functionalities without needing any special components.

According to another advantageous embodiment of the invention images of the environment including at least part of the first vehicle are provided in form of a second video stream by means of the at least one sensor device and the second video stream is displayed on the display device, wherein the received visual information is displayed embedded in the second video stream. Advantageously, a user, like a driver of the second vehicle, can at the same time be provided with information about the environment, which can directly be seen by the at least one sensor device, like the camera of the second mobile communication device, as well as with the additional visual information about the part of the environment, the view onto which is blocked by the first vehicle travelling ahead. Thereby, the received visual information is displayed at least embedded in some of the images of the second video stream. For example, in case there is no vehicle travelling ahead of the second vehicle, the second video stream can still be displayed on the display device, however, no additional visual information is displayed embedded in this second video stream. Moreover, the received visual information preferably is displayed embedded in the second video stream as long as the first vehicle is being detected by the at least one sensor device. In case the first vehicle is not detected anymore, then also the streaming of the first video is terminated.

Furthermore, according to another advantageous embodiment of the invention, a certain displaying position within at least one of the images of the second video stream is determined and the visual information is displayed at the determined displaying position in the at least one of the images of the second video stream. For example, the visual information can be displayed in form of an overlay over the displayed images of the second video stream. Generally, the received visual information can be displayed anywhere on the display device, especially also anywhere in the at least one of the images of the second video stream. However, to occlude as less as possible information provided by the displayed second video stream, it is advantageous to display the received visual information at a certain determined position within the at least one of the images of the second video stream fulfilling that requirement.

Therefore, according to another very advantageous embodiment of the invention for at least one of the images of the second video stream an area, which represents the first vehicle within the at least one of the images of the second video stream, is determined and the visual information is displayed at the determined area in the at least one of the images of the second video stream. So, the position of the first vehicle within the at least one image of the second video stream can be determined, especially the area of the at least one image containing the image data relating to the first vehicle, and then the visual information can be displayed as an overlay over this certain determined area, especially within this area. Thereby, the received visual information can fill the whole determined area completely or only part thereof. So advantageously, by displaying the received additional visual information on the back of the first vehicle travelling ahead in the second video stream it can be avoided that important environmental information provided by the second video stream is occluded by the displaying of this additional visual information.

As already mentioned above, the visual information is preferably provided in form of a continuous stream of images, especially a first video stream, which is then embedded image by image in the second video stream. Thereby, according to another advantageous embodiment of the invention, it is very advantageous, when a gradient mask is applied to at least one image of the received first video stream, and the at least one image of the received first video stream is displayed embedded in the second video stream according to the applied gradient mask. By means of the gradient mask very smooth transitions from the at least one image of the received first video stream and the surrounding parts of the images of the second video stream can be provided. Such gradient mask can for example define a degree of transparency of each of the images of the received first video stream, which varies spatially over each image of the first video stream. Such gradient mask can for example be designed such that the transparency in the center portion of the images of the first video stream is zero and is increasing towards the edges of the respective images of the first video stream.

According to another advantageous embodiment of the invention the first vehicle is detected based on at least one image, which may be one of the images of the second video stream or a different one, captured by the at least one sensor device, wherein in case several vehicles are detected based on the at least one image, that one of the detected vehicles, which is closest to a defined central area of the captured image, is selected as the first vehicle. In other words, the vehicle travelling straight ahead can be easily determined by simply selecting the vehicle in front, which is the most central one in the image.

According to another advantageous embodiment of the invention the visual information is transmitted from the first device to a central data processing unit, especially a cloud server, and is transmitted from the central data processing unit to the second mobile communication device. Therefore, vehicles, or especially the mobile communication devices associated with the respective vehicles, can simply register with a web service using the respective license plate numbers as identifiers and by registering with such web service signalize that the respective mobile communication devices are capable of and willing to share visual information capturable by the respective mobile communication devices. So advantageously, the availability and distribution of the visual information can be performed centralized by the central data processing unit. A further great advantage thereof is, that no direct communication between vehicles or the mobile communication devices is required. So if a driver wants to use the see through functionalities, such a driver does not need to purchase a car with a compatible system. Most users have smartphones anyway and the see through functions can be implemented completely car independent, which makes it possible to retrofit all the cars with see through

functionalities and no costs.

According to another advantageous embodiment of the invention the visual information is transmitted from the first device to the central data processing unit only after the central data processing unit has received the request for the visual information. In other words, the first device only starts streaming its captured first video stream upon request.

Thereby, unnecessary data transmission can be avoided.

As also mentioned above, all functions and method steps executed by the first device associated with the first vehicle can similarly also be performed by the second mobile communication device. So, preferably, the second mobile communication device communicates an availability of a second visual information capturable by the at least one sensor device together with an identifier related to a license plate of a second vehicle, namely the second vehicle mentioned before, the second mobile communication device is associated with, to the central data processing unit. So, also environmental information captured by the second communication device can be shared with other vehicles, e.g. a vehicle trailing the second vehicle in the same way as explained with regard to the sharing of the visual information provided by the first device to the second mobile communication device.

Also here it is preferred, that the second visual information is transmitted from the second mobile communication device to the central data processing unit, especially only upon request from the central data processing unit. Here again, unnecessary data transmission can be avoided.

The invention also relates to a computer program product, especially an application program or a so-called app for a mobile communication device like a smartphone. The computer program product comprises program code, which is stored in a computer readable medium, and which when executed by a processor of an electronic control device causes the processor to detect a first vehicle concealing a view onto at least part of an environment, to request a visual information about the at least part of the environment, the view onto which is at least in part concealed by the first vehicle, and in case of an availability of the requested visual information to receive the visual information and to display the visual information on a display device. Moreover, the program code when executed by the processor causes the processor to derive an identifier of the first vehicle from a detected license plate of the first vehicle and to request the visual information using the derived identifier of the first vehicle to identify the first vehicle, which conceals the view onto the at least part of the environment, the visual information about which is requested.

The invention also relates to a mobile communication device for providing visual information about at least part of an environment, the view onto which is at least in part concealed by a first vehicle, wherein the mobile communication device comprises at least one sensor device, especially a camera, which is configured to detect the first vehicle, wherein the mobile communication device further comprises a communication unit, which is configured to request the visual information about the at least part of the environment, the view onto which is at least in part concealed by the first vehicle. Moreover, the communication unit is configured to receive the visual information in case of an availability of the requested visual information. Moreover, the mobile communication device is configured to detect a license plate of the first vehicle and to derive an identifier of the first vehicle from the detected license plate, and to request the visual information using the derived identifier of the first vehicle to identify the first vehicle, which conceals the view onto the at least part of the environment, the visual information about which is requested.

Especially, the mobile communication device according to the invention or its

embodiments can be configured as described before with regard to the first device and/or the second mobile communication device.

Furthermore, the invention also relates to a communication system comprising at least one mobile communication device according to the invention or one of its embodiments and a central data processing unit, wherein the central data processing unit is configured to receive a request for the visual information and the identifier from the at least one mobile communication device, and to check on the basis of the identifier, whether the visual information is available and in case the visual information is available to transmit the visual information to the at least one mobile communication device.

The advantages described with regard to the method according to the invention and its embodiments similarly apply for the computer program product, the mobile

communication device and the communication system according to the invention.

Moreover, the embodiments described with regard to the method according to the invention constitute further advantageous corresponding embodiments of the computer program product, the mobile communication device and the communication system according to the invention.

Further features of the invention are apparent from the claims, the figures and the description of figures. The features and feature combinations mentioned above in the description as well as the features and feature combinations mentioned below in the description of figures and/or shown in the figures alone are usable not only in the respectively specified combination, but also in other combinations without departing from the scope of the invention. Thus, implementations are also to be considered as encompassed and disclosed by the invention, which are not explicitly shown in the figures and explained, but arise from and can be generated by separated feature combinations from the explained implementations. Implementations and feature combinations are also to be considered as disclosed, which thus do not have all of the features of an originally formulated independent claim. Moreover, implementations and feature combinations are to be considered as disclosed, in particular by the implementations set out above, which extend beyond or deviate from the feature combinations set out in the relations of the claims.

Therein show:

Fig. 1 a schematic illustration of a communication system with two mobile

communication devices, each associated with a corresponding vehicle, for providing visual information about at least part of an environment, the view onto which is at least in part concealed by one of the vehicles, according to an embodiment of the invention;

Fig. 2 a schematic illustration of a functional diagram for illustrating a method for providing visual information according to an embodiment of the invention;

Fig. 3 a schematic illustration of the function performed by a cloud server of a communication system according to an embodiment of the invention; Fig. 4 a schematic illustration of the detection of the vehicle travelling ahead on the basis of a captured image as performed by a mobile communication device according to an embodiment of the invention;

Fig. 5 a schematic illustration of the extraction of the license plate number as identifier on the basis of the image captured by a mobile communication device according to an embodiment of the invention; and

Fig. 6 a schematic illustration of the embedding of the visual information provided by the vehicle travelling ahead into an image of the video stream provided by the mobile communication device of the trailing vehicle according to an embodiment of the invention.

Fig. 1 shows a schematic illustration of a communication system 1 comprising in this example two mobile communication devices, namely a first mobile communication device 2 and a second mobile communication device 3, each associated with a respective vehicle, namely a first vehicle 4 and a second vehicle 5, wherein the first vehicle 4 is travelling ahead of the second vehicle 5. Each of the mobile communication devices 2, 3 comprises a processing unit 6, 7, at least one sensor device, which is configured as a camera 8, 9 and a communication unit 10, 1 1 . The communication units 10, 1 1 are capable of communicating with a central data processing unit, which in this example is a cloud server 12. This communication is illustrated by the arrows 13.

Also each of the mobile communication devices 2, 3 comprises a display device 14, 15. Preferably, the mobile communication devices 2, 3, which are preferably configured as smartphones but can also be configured as a tablet PC, a laptop, a notebook, and so on, are positioned behind the windshield of the associated vehicles 4, 5, such that the respective fields of view FOV1 , FOV2 of the cameras 8, 9 of the respective mobile communication devices 2, 3 cover at least part 16a, 16b of the environment 16 in the front of the respective vehicles 4, 5.

In this example, the view of the camera 8 of the second mobile communication device 3 onto at least part of the environment 16 is concealed by the first vehicle 4 travelling ahead. The invention and its embodiments now allow for a very advantageous see through functionality, which advantageously can be independent of any component of the corresponding vehicles 4, 5. To perform this see through function an application running on the processing unit 6, 7 of the first and second mobile communication devices 2, 3 can cause the corresponding processing units 6, 7 to execute a method as described in the following. Thereby functions performed by the first mobile communication device 2 can similarly be performed by the second mobile communication device 3 as well and the other way round.

First of all, the camera 8 of the second mobile communication device 3 continuously captures images of the part 16b of the environment 16 within its field of view FOV2. The captured images can be analyzed and based on such an analysis vehicles like the second vehicle 4 can be detected within those captured images. To detect vehicles known object detection and classification methods can be used. Especially known learning algorithms and methods like AdaBoost, support vector machine or deep learning can be used. In case several vehicles are detected, then the most central on the captured images is selected, in this case the first vehicle 4, for which the visual see through information shall be requested. So as soon as the first vehicle 4 is selected, the license plate is detected on the basis of the captured images as well as and the license plate number is extracted from the detected license plate 17. Also here above-named algorithms and methods can be used to recognize and read the license plate number. After that, the second mobile communication device 3 sends a request by means of its communication unit 10 to the cloud server 12 for requesting the visual information, which is concealed by the first vehicle 4. Together with the request also the derived license plate number is transmitted to the cloud server 12 as identifier of the first vehicle 4. After that, the cloud server 12 checks, whether the requested visual information related to the first vehicle 4 is available. This is the case if the first mobile communication device 2 has registered with the cloud server 12 providing its license plate number as identifier and signalizing that an environmental video stream is available. So if this is the case, the cloud server 12 informs the second mobile communication device 3, that the requested visual information is available and communicates to the first mobile communication device 2 that the visual information has been requested. After that, images captured by the camera 9 of the first mobile communication device 2 are streamed in form an environmental live video via the cloud server 12 to the second mobile communication device 3. At the same time, the second mobile communication device 3 keeps on capturing images by means of its camera 8 of the part 16b of the environment 16 and provides these images in form of a second video stream. The received visual information in form of the first video stream provided by the first mobile communication device 2 is now embedded into this second video stream image by image by means of the second mobile communication device 3 and displayed on the display device 14. This procedure is now illustrated in more detail in Fig. 2 to Fig. 6 and explained in the following.

Fig. 2 shows a schematic illustration of a functional diagram for providing visual information according to an embodiment of the invention. The functional components of the mobile communication devices 2, 3 are the built in camera video stream module 18, a first communication module 19, which performs the function of a cloud video manager, a vehicle detection and selection module 20, a license plate reader 21 , a second communication module 22, which performs the function of the video stream receiver, and a video composer and rendering module 23. On the other hand the cloud server 12 comprises a cloud video dispatcher module 24. The performed functions of the respective modules are now explained in more detail with regard to Fig. 3 to Fig. 6.

Fig. 3 shows a schematic illustration of the cloud video dispatcher module 24. The cloud server 12 as already explained, can communicate with the first and second mobile communication devices 2, 3 as well as with any arbitrary number of further mobile communication devices N. All such mobile communication devices 2, 3, N are

communicating their availability to the cloud video dispatcher module 24 with the license plate as identifier as described before. Moreover, also all mobile communication devices 2, 3, N can retrieve a video stream for specific vehicles, which are identified based on their license plate number. So on the one hand all mobile communication devices 2, 3, N, which want to share their captured video stream with other vehicles to provide see through functions can signalize their availability to the cloud video dispatcher module 24 together with their respective license plate numbers, and on the other hand all mobile communication devices 2, 3, N, which want to provide the see through function, can request visual information from the cloud video dispatcher 24 for specific vehicles, which are identified based on their license plate numbers.

Now the detection and selection of the vehicle travelling ahead is described in the following with regard to Fig. 4. Fig. 4 shows a schematic illustration of an image, which is captured by the camera 8 of the second mobile communication device 3, especially as part of a second video stream. At the same time this captured second video stream can be displayed on the display device 14 of the second mobile communication device 3. So this second video stream is provided by the built in camera video stream module 18. Based on the captured images of this second video stream the vehicle detection and selection module 20 vehicles are detected and one of them is selected. In this example, in the image 25 three vehicles 4, 26, 27 have been detected, wherein only the most central one, in this case the first vehicle 4, is selected by the vehicle detection and selection module 20.

After that, as illustrated in Fig. 5, the license plate reader 21 , which can be also some kind of suitable algorithm or image analysis method, identifies the license plate 17 and reads the license plate number 17a. The extraction of the license plate number 17a out of the image 25 is illustrated by the arrow 28. Based on the identified license plate number 17a the first communication module 19, namely the cloud video manager requests from the cloud server 12 the visual information, which is blocked at least in part by this first vehicle 4 travelling ahead, whereby the communication module 19 also transmits the identified license plate number 17a to identify the first vehicle 4, with regard to which the visual information is requested. In case the requested visual information is available, the cloud video dispatcher module 24 of the cloud server 12 transmits the first video stream provided by the first mobile communication device 2 to the second communication module 22, which is a video stream receiver. The received first video stream is then passed image by image to the video composition and rendering module 23, which composes the first video stream provided by the built in camera video stream module 18 and the received first video stream. This is illustrated in Fig. 6.

Fig. 6 shows on the right hand side an image 25a provided by the built in camera video stream module 18 captured by the camera 8 of the second mobile communication device 3 and the received visual information in form of an image 29 of the streamed first video provided by the camera 9 of the first mobile communication device 2 and on the left hand side the composed image 31 . To provide this composed image 31 the video composition and rendering module 23 first determines a certain area in the image 25a captured by the second mobile communication device 3 as part of the second video stream, which in the following is therefore called second image 25a. This area 30 constitutes the part of the second image 25a, which represents the first vehicle 4. The first image 29, namely the image of the received first video stream is then overlaid over the second image 25a within the defined area 30. Moreover, also a gradient mask can be applied to the remote video stream, namely to the first image 29, especially each first image 29 of the first video stream, to provide smooth transitions from the first image 29 to the surrounding parts of the second image 25a. The gradient mask can for example define a degree of

transparency of the respective images of the first video stream, whereby the transparency is increasing from the centre to the edges of the respective images. Moreover, it is also possible to display or stream only part of the field of view captured by the first device, like to stream only video data relating to a certain central area of the field of view of the camera of the first device. This has the advantage, that the most relevant see-through information can be provided to the user of the second vehicle and at the same time the data traffic can be reduced. Also more sophisticated modifications are possible, like displaying only parts of the visual information, which relate to that part of the environment, which is concealed by the first vehicle traveling ahead. The finally composed image 31 is then displayed on the display device 14 of the second mobile communication device 3. This is especially performed in each time step namely from each image of the received first video stream and the corresponding images of the captured second video stream. Another possibility is to use a different algorithm or a look up table. As a result, the user will be able to see through the vehicle travelling ahead and for example overtake safely. By the use of smartphones to perform the described method using an app running on the smartphone, such a see through function can be provided in a very cost-effective and comforting way. All the apps can be connected to the same cloud server. Thanks to the license plate that will be the identifier of the vehicle travelling ahead, and by using image processing the smartphone application can detect, identify a vehicle, retrieve an available video stream via the web server and map it into its own video stream. In this way, any driver won’t need to purchase a car with a compatible system. Most of the drivers have a smartphone and being car independent provides the great advantage of retrofitting all the cars with no costs.