The present application relates to vehicle camera systems. It is an object of the present specification to provide an im ^¬ proved method and device for the detection of droplets on a camera lens. In a broader sense, the method is also used for detecting droplets on a transparent surface in front of the camera lens, such as a windscreen or on a transparent lens cover.

The droplet detection procedure according to the present specification makes use of the lens property of droplets. When there are droplets of rain on the lens of a camera each drop- let itself acts as a further lens and contains a miniature view of the entire scene that is being seen by the main camera image .

Due to the smallness of the droplets the focal plane of the droplets is very close to the lens surface of the camera lens. Behind the focal plane of the drops the light rays diverge again and the droplets act as wide angle lenses. As a conse ^¬ quence, the droplets appear as tiny wide angle lenses at the focal plane of the camera lens, which is far behind the focal plane of the drops and produce a minimized image of the scene.

Furthermore, the droplets have a large curvature and act as fish eye lenses which can cover a large part of the scene or even provide a 180° view that covers the complete scene which is observed through the camera lens.

The image that is produced by the droplet is in general ro ^¬ tated. Usually the image rotation is about 180° degrees, but for some shapes of droplets the rotation angle can be differ ^¬ ent from 180°. The rotation can be found out by identifying a demarcation line between a bright and a dark area, such as the horizon, and observing its rotation.

In general, the droplet image is distorted. If the drops are not spherical but thicker at one end, the wide angle effect is stronger at the thicker end of the drop and the distortion is stronger. The imaging plane where the drop image appears sharp is located behind the imaging plane in which the scene appears sharp. If the objects are sufficiently far away, the imaging planes are close to the respective focal planes.

If the depth of focus of the camera lens does not cover the image plane of the droplet and the image plane of the camera lens, the effect can be compensated according to the present specification by focusing part of the camera light to a second sensor, which is placed at a distance at which the droplet im ^¬ ages appear sharp for a typical droplet size and using the memory representation of this image for the detection of water drops .

The droplets, which form "little windows" often stick to the surface and do not move from frame to frame. According to the present specification, this property can be used to improve the identification of the droplets by using multiple image frames. If it is detected that an object resembles the scene and the boundaries are not changing or are changing only slightly, the detection of a droplet can be confirmed.

Likewise, the droplet detection can be confirmed or improved by detecting whether the content of an object, such as the relative illumination of image portions changes in a similar way as the complete observed scene. Consecutive image frames can be considered as forming a three dimensional cube. Accord ^¬ ing to an extended embodiment, miniature versions of the N frame cube are detected in the N frame cube.

Furthermore, a droplet detection according to the present specification can also be initialized by detecting boundaries of objects and confirming the detection by matching the content of the detected object with the scene.

According to one embodiment, the method is carried out on an image which is not already corrected for lens distortions. The image detection can be carried out on simple mono cameras that provide grey scale images. If colour images are available, the colour information can be used to improve the detection quality. If image frames from a stereo camera are available, the difference between the stereo images may be used in addition to speed up the detection of droplets. According to the present specification, rain drops are detected by identified image areas which appear to be a small scaled view of the entire image.

According to one embodiment, the entire image view is scaled to various small sizes within the known range of likely rain ^¬ drop sizes. Then, these scaled down areas are matched against areas of the main image by pattern matching. If small, entire views of the complete image are found on top of image patches it can be concluded that the image patches correspond to rain- drops in that part of the image.

According to one embodiment, the detected raindrop areas are reported or forwarded to other algorithms as areas that should be ignored or treated with caution. If the proportion of the rain drops is large, it may even be decided that the entire image cannot be trusted for other object detection algorithms because it is known to be severely obscured.

The current specification discloses a computer implemented method for detecting droplets on a lens of a vehicle camera. Image data with image frames is received from a vehicle cam ^¬ era. A synthetic droplet image is generated by reducing a first image frame of the image data to a predetermined size of a droplet, which is within a pre-determined range of droplet sizes .

The reduction in size can in general involve some sort of av- eraging or interpolation procedure to map the pixels of the image frame to the size of the synthetic droplet image.

Herein, "the image frame" does not have to include every pixel of the image frame. For example it also refers to a major por ^¬ tion of the image frame.

Thereby a reduced size image is obtained, and the reduced size image is rotated by a pre-determined rotation angle, which is typically 180 degrees. In particular, a 180 degree rotation may be obtained without copying computer memory portions just by changing an indexing of the lines of the memory representa ^¬ tion of the image frame or of the synthetic droplet image. For example, a computer program may be operative to access the computer memory in a reverse order. The rotation may also be obtaining by copying memory content, however.

A similarity is determined or computed between the synthetic droplet image and a detection portion of a second image frame at a pre-determined position of the detection portion. Herein, the detection portion has essentially the same shape and size as the synthetic droplet image. In particular, the first and second image frame may refer to the same image frame or to copies of the same image frame. In an alternative embodiment, the first and second image frames are obtained from separate image sensors but cover essentially the same region of the scene .

The comparison is repeated for different positions of the de- tection portion. For example, the comparison may be repeated for different positions of the detection portion until at least 80% of the frame area is covered with a pre-determined resolution of positions. The pre-determined resolution may be chosen as 1 pixel, which means that close-by detection regions are offset by 1 pixel with respect to each other, or it may be lower. A droplet, or a droplet candidate, is detected if the similarity exceeds a predetermined threshold.

According to a further application, a ratio of the area occupied by the droplets and the area of the image frame, and an alert message is sent if the ratio exceeds a predetermined value. The locations of the detected droplets are forwarded to other image processing applications.

The comparison between the synthetic droplet and the image portion is performed in parallel for multiple image portions. For example, the comparison can be carried out by multiple threads, which a scheduler program or a scheduler hardware component can assign to different processors or processor ker ^¬ nels depending on the availability of hardware resources. According to a further embodiment, which provides further options to carry out the comparisons in parallel, the second im ^¬ age frame is stored in a first memory location and in a second memory location.

A second synthetic droplet image is generated by reducing the first image frame of the image data to a second predetermined size of a droplet, thereby obtaining a reduced size image and by rotating the reduced size image by a pre-determined rota- tion angle, which is typically 180 degrees.

A similarity is determined between the second synthetic drop ^¬ let image and a detection portion of the second image frame at the second memory location at pre-determined positions of the detection portion. Similarly, the detection portion has the same shape and size as the second synthetic droplet image. A droplet, or a droplet candidate, is detected if the similarity exceeds a predetermined threshold. According to a further embodiment the first image frame is separated into regions with differing brightness for example regions below and above a horizon. The comparison between a synthetic droplet image and a detection portion comprises separating the detection portion into corresponding regions and comparing the regions of the first image frame with the regions of the detection portion. Thereby, the droplet detec ^¬ tion may be improved or it may be accelerated.

According to a specific embodiment, the position and shape of the synthetic droplet is adjusted according to a pre ^¬ determined lens distortion function and according to a position of the detection portion in the second image frame. This embodiment takes into account that droplet may appear differ- ently in different locations, in particular next to the border of the image frame.

According to a further embodiment, the first image frame is obtained from a first sensor and the second image frame is ob ^¬ tained from a second sensor, wherein an optical distance from a camera lens to the first sensor is different from an optical distance from the camera lens to the second sensor. This em ^¬ bodiment can take into account that the droplets change the effective focal distance of the lens, and the image of the droplet appears focused at a different optical distance than the image plane of the main image.

In particular, the sizes of the droplets can be chosen accord- ing to a pre-determined geometric progression or according to a pre-determined statistical distribution of droplets.

In order to speed up the detection procedure, previously de ^¬ termined positions of droplets can be used as starting posi- tions for comparing the synthetic droplets with underlying portions of the image frame. For example, the determined drop ^¬ let positions of the last 10 frames may be used to predict the positions of the droplet in a present frame. The specification also discloses a computer program for executing the steps of the aforementioned method and a computer readable storage medium with the computer program.

In a further aspect, the current specification discloses an image processing device for a vehicle camera which is opera ^¬ tive to execute the aforementioned detection method. In particular, the present specification discloses an image processing device which comprises an input connection for receiving image data from the vehicle camera, the image data comprising image frames and a computation unit that is con- nected to the input connection.

The computation unit is operative to generate a synthetic droplet image by reducing a first image frame of the image data to a predetermined size of a droplet, thereby obtaining a reduced size image and by rotating the reduced size image by a pre-determined rotation angle. Furthermore, the computation unit is operative to determine a similarity between the syn ^¬ thetic droplet image and a detection portion of a second image frame at a pre-determined position of the detection portion, wherein the detection portion has the same shape and size as the synthetic droplet image, and to detect a droplet if the similarity exceeds a predetermined threshold.

In a further embodiment, the computation unit comprises a multi-kernel graphics processor. Furthermore the computation unit provides scheduling means to assign comparisons of a syn ^¬ thetic droplet image with multiple image portions of an image frame to different processor kernels of the multi-kernel graphics processor.

Furthermore, the present specification discloses a kit with the image processing device wherein the vehicle camera is con- nectable to the image processing device. For example, the im ^¬ age processing device can be supplied as part of the vehicle camera or separately for attachment to the car electronics.

In a further embodiment, the vehicle camera is adapted to fo ^¬ cus the camera image to a first sensor and to a second sensor. The first image frame is obtained from the first sensor and the second image frame is obtained from the second sensor.

Furthermore, the present specification discloses a vehicle, such as a passenger car, a utility car, a bus etc., with the kit installed in the vehicle.

The vehicle camera is mounted to the vehicle such that the ve ^¬ hicle camera points to a scene outside the vehicle and is con ^¬ nected to the image processing device. The image processing device is provided as an electronic component of the vehicle or of the vehicle camera.

The subject matter of the present specification is now ex ^¬ plained in further detail with respect to the following Figures in which

Fig. 1 shows a vehicle with a camera system,

Fig. 2 shows a camera image with droplets recorded by the camera system of Fig. 1,

Fig. 3 shows a droplet recognition procedure for detecting droplets, such as the droplets of Fig. 2,

Fig. 4 shows a further droplet recognition procedure that involves multiple comparisons in parallel,

Fig. 5 shows a further droplet recognition procedure that involves multiple comparisons in parallel, and

Fig. 6 shows a droplet recognition procedure that involves copying of memory areas and performing the comparison in parallel for the multiple memory areas.

In the following description, details are provided to describe embodiments of the application. It shall be apparent to one skilled in the art, however, that the embodiments may be prac ^¬ ticed without such details.

Some parts of the embodiments have similar parts. The similar parts may have the same names or similar part numbers. The de ^¬ scription of one similar part also applies by reference to an ^¬ other similar parts, where appropriate, thereby reducing repe ^¬ tition of text without limiting the disclosure. Fig. 1 shows a car with a camera system. Figure 1 shows a car 10 with a surround view system 11. The surround view system 11 comprises a front view camera 12, a right side view camera 13, a left side view camera 14 and a rear view camera 15. The cam ^¬ eras 11 - 14 are connected to a CPU of a controller, which is not shown in Fig. 1. The controller is connected to further sensors and units, such as a velocity sensor, a steering angle sensor, a GPS unit, and acceleration and orientation sensors.

Fig. 2 shows a camera image 16 with droplets 17. For simplic- ity, the droplets 17 are shown with uniform shapes and sizes. In a real picture the shape, the orientation and also the sizes of the droplets vary.

Fig. 3 shows a droplet recognition procedure, in which a syn- thetic droplet image 18 is compared to corresponding underly ^¬ ing image portions at multiple locations. By way of example, Fig. 3 shows a direction 20 in which the synthetic droplet im ^¬ age is moved across the image frame. The image frame may be scanned in various ways, for example line by line, column by column, according to a star pattern, on circular trajectories and so forth. The synthetic droplet image 18 is obtained by reducing the complete image frame 16' to a pre-determined droplet size and by rotating the reduced size image by 180 degrees. The syn ^¬ thetic droplet image 18 will in general also contain reduced size versions of the droplets while the images produced by the droplets do not contain these artefacts.

As long as the droplets only cover a minor portion of the screen, this difference does not affect the droplet recogni- tion. The synthetic droplet images are only compared with re ^¬ spect to an approximate similarity. In general, the droplet images also show artefacts which are due to the irregular shape of the droplets. In a simple realization, a similarity of the synthetic droplet image 18 with an underlying image portion is measured with a least squares estimate that compares the pixel brightness. Ac ^¬ cording to other realizations, the comparison also makes use of detected image features of the synthetic droplet 18, such as divisions into dark and bright image areas or regions of high contrast, lines, feature points, vanishing points etc.

Often, the camera image contains features that are aligned along street boundaries, which essentially have the same van- ishing point. This feature can be used for comparison provided that the distortion of the droplets is not such that it pre ^¬ vents the determination of a vanishing point.

On the other hand it can be advantageous to compare only sim- pie features of the synthetic droplet image 18 that do not re ^¬ quire extensive image processing. For example, according to one realization, the pixels of the synthetic droplet image are averaged to effective pixels with a bigger size than the original pixels and the effective pixels are compared with ef ^¬ fective pixels of the underlying image portion. According to another realization, only a subset of the pixels is compared, such as every second pixel.

Fig. 4 shows a further droplet recognition procedure in which a synthetic droplet image 18 is compared to multiple different image portions of an image frame 16' in parallel. According to one exemplary embodiment, the comparisons are assigned to dif- ferent computational threads. A scheduler assigns the threads to different processor kernels of a graphics card according to the availability of computational resources.

For easier visualization, the synthetic droplet images 18, 19 in the Figs. 4 - 6 are shown bigger than their typical size would be. For example, a droplet size could be 1/8 or 1/10 of the horizontal or the vertical extension in the respective di ^¬ rection or smaller. Among other factors, this property depends on the size of the lens.

The droplet images 18 are assigned to rectangular portions of the image 16' and are compared with corresponding underlying image portions of the rectangular portion of the image 16'. Fig. 5 shows a further droplet recognition procedure in which the image frame 16' is compared with differently sized syn ^¬ thetic droplet images, a synthetic droplet image 18 of a first size and a synthetic droplet image 19 of a second size. The area sizes of the synthetic droplet images 18, 19 are chosen within a pre-determined range of droplet sizes.

In the position of the synthetic droplets 18 of Fig. 5, the synthetic droplet images 18 are moved to a position in which one of the synthetic droplet images 18 has a high similarity to an underlying portion 17' of the image frame 16. In particular, the darker foreground almost coincides with the image of the foreground provided by the droplet on the camera lens and the brighter sky region almost coincides with the image of the sky provided by the droplet on the camera lens.

In one realization according to Fig. 5, columns 21, 22, 23 of synthetic droplet images 18 are moved across the image frame 16' in a horizontal direction. In other words, the columns of synthetic droplet images are compared to underlying image por ^¬ tions of the image frame 16' at consecutive locations, and the consecutive locations are moved from left to right. The col ^¬ umns contain droplet images 18' of different sizes and/or of different vertical alignments.

The direction of movement 20 from left to right is only pro ^¬ vided by way of example. For example, the columns 21, 22, 23 could be moved from right to left or the synthetic droplet im- ages could be grouped into rows which are moved from bottom to top or from top to bottom.

Fig. 6 shows a further droplet recognition procedure. Accord ^¬ ing to the procedure of Fig. 6, the image frame 16', which is stored in a memory area 24 of a computer memory, is copied to a first memory area 25 and to a second memory area 26. The dots in Fig. 6 indicate that the memory area 24 can be copied to more than two memory areas . Synthetic droplet images 18 of a first size are compared with corresponding underlying image portions of the image frame 16' of the first memory area 24. Similarly, synthetic droplet im ^¬ ages 19 of a second size are compared with corresponding un- derlying image portions of the image frame 16' of the second memory area 26.

In an alternative embodiment, the first memory area 25 is pro- vided by the original memory area 24. A copying of computer memory areas according to the embodiment of Fig. 6 can facili ^¬ tate a parallel processing. For example, the comparisons that involve the copied memory areas 25, 26 can be carried out in parallel by different processor kernels or they can be as- signed to different threads.

Although the above description contains much specificity, this should not be construed as limiting the scope of the embodi ^¬ ments but merely providing illustration of the foreseeable em- bodiments. The above stated advantages of the embodiments should not be construed especially as limiting the scope of the embodiments but merely to explain possible achievements if the described embodiments are put into practice. Thus, the scope of the embodiments should be determined by the claims and their equivalents, rather than by the examples given.