RELATED APPLICATION The present application claims the benefit of US provisional application

61/167,226 filed on April 7 ^th , 2009, the disclosure of which is incorporated herein by reference.

FIELD OF THE INVENTION The present invention relates to imaging systems, and more particularly, the present invention relates to an imaging system with light distortions compensation system and methods.

BACKGROUND OF THE INVENTION AND PRIOR ART An image frame acquired by an image sensor is generally subjected to light distortions caused by elements that block at least a portion of the light from reaching at least a portion of the image sensor array. The blocking elements can be dirt, glass defects and/or liquid such as rain drops, sun light rays dispersed towards the image sensor array after hitting dirt elements and other. Typically, dirt causes at least partial blocking of the light and thereby blocked sensor array pixels sense less energy. Typically, glass defects and/or liquid such as rain drops, cause blurring effect on the acquire image frame. Typically, sun light rays dispersed towards the image sensor array causes high energy spot sensed by sensor array pixels. There is therefore a need for and it would be advantageous to have a camera system with a noise cancelation device to undo at least a portion of the light distortions caused by elements that block at least a portion of the light from reaching at least a portion of the image sensor array.

SUMMARY OF THE INVENTION

According to teachings of the present invention, there is provided a computerized image acquisition system for reducing light distortions related noise in an acquired image frame. The system includes multiple optical image acquisition devices, each having substantially the same field of view (FOV), each configured to acquire one or more image frames and aligned to view substantially the same scene. The system further includes an image fusion module, wherein at least two of the image acquisition devices synchronously acquire image frames, wherein alignment offset parameters are determined between each pair of image acquisition devices, wherein correspondence is determined between the acquired image frames, wherein distorted pixels are determined in either of the corresponding image frames, and wherein the image fusion module fuses the corresponding image frames to form a fused image frame, wherein the determined distorted pixels are compensated for in the fused image frame. The correspondence is determined between image frames, acquired by a pair of the image acquisition devices substantially simultaneously, including by using the alignment offset parameters.

It should be noted that in general, the present invention is described, with no limitations, in terms of an image acquisition system having two image acquisition devices. But the present invention is not limited to two image acquisition devices, and in variations of the present invention, the image acquisition system can be similarly embodied with three image acquisition devices and more.

According to further teachings of the present invention, there is provided a method for reducing light distortions related noise in an acquired image frame by a computerized image acquisition system, the system including multiple optical image acquisition devices, optionally video cameras, each having substantially the same FOV and aligned to view substantially the same scene and each configured to acquire one or more image frames and aligned to view substantially the same scene. Preferably, each pair of image sensor arrays of adjacently disposed optical image acquisition devices are calibrated whereby obtaining alignment parameter between the adjacently disposed image sensor arrays of the pair.

The method includes the steps of: a) acquiring image frames by at least two of the image acquisition devices; b) determining corresponding image frames, acquired synchronously, including using the alignment offset parameters; c) selecting a first image acquisition device from the image acquisition devices, and a second image acquisition device from the image acquisition devices; d) selecting a pair of respective image frames from the corresponding image frames, the pair having a first image frame, acquired by the first image acquisition device, and a second image frame, acquired by the second image acquisition device, wherein the first image frame and the second image frame have been acquired substantially simultaneously; e) determining distorted pixels in either of the corresponding image frames by comparing the corresponding image frames; and f) fusing the corresponding image frames into a fused image frame, wherein the determined distorted pixels are compensated for, in the fused image frame.

In a first embodiment of the present invention, the determining of the distorted pixels includes the step of summing up the first image frame and the second image frame, thereby creating a noise integrated image frame. The first image frame is also subtracted from the second image frame to create a difference image frame. The next step is to compute the absolute value of each pixel in the difference image frame thereby creating a noise image frame. Finally, adding the noise image frame from the noise integrated image frame thereby creating the fused image frame.

Preferably, in other embodiments of the present invention, the methods of the present invention further includes the step of subdividing the respective image frames of the pair of respective image frames into respective, spatially symmetric, first and second groups of pixels, wherein the subdividing is performed before fusing the respective image frames. A group of pixels includes NXM pixels, wherein N and M are integers. In some variations of the present invention, both N and M are equal to 1. In other variations of the present invention, both N and M are the dimensions of the corresponding image frames.

In a second embodiment of the present invention, the determining of the distorted pixels includes the step of setting the first image frame to be a primary image frame and the second image frame to be a secondary image frame. Then, for each pair of respective primary and secondary groups of pixels from said respective subdivided primary and secondary image frame, calculating a differential signal to noise factor between the respective primary and secondary groups of pixels, thereby creating a differential noise group.

The fusing of the primary image frame from the secondary image frame is performed on each pair of respective group of pixels in the primary image frame and the secondary image frame, as follows: if the differential noise group is less than a pre selected threshold values, the value of the respective group of output pixels in the fused image frame is set to be the value of each respective pixel in said primary group of pixels. Else, the value of the respective group of output pixels in the fused image frame is set to be the values of each respective pixel in the secondary image frame.

In a third embodiment of the present invention, the determining of the distorted pixels includes, for each pair of respective first and second groups of pixels from said respective subdivided first and second image frame, calculating a differential signal to noise factor between the respective first and second groups of pixels, thereby creating a differential noise group.

The fusing of the first image frame from the second image frame is performed on each pair of respective group of pixels in the first image frame and the second image frame, as follows: determining a first weight factor based as a function of the differential noise group, calculating a first weighted group of pixels, wherein each first weighted pixel in the first weighted group of pixels is assigned the product of the value of the respective pixel value in the first group of pixels and the first weight factor, determining a second weight factor based as a function of the differential noise group, calculating a second weighted group of pixels, wherein each second weighted pixel in the second weighted group of pixels is assigned the product of the value of the respective pixel value in the second group of pixels and the second weight factor, and setting the value of the individual pixels in the respective group of output pixels in the fused image frame, to be the normalized sum of the respective individual values of the first weighted pixel and the second weighted.

In a forth embodiment of the present invention, the determining of the distorted pixels includes the step of computing the average values of each pair of respective pixels of the respective image frames. The fusing of the corresponding image frames includes the step of setting the value of the respective pixels in the fused image frame to be the computed average value. In variations of the present invention, the computed average pixel values are weighed average values, whereas each pixel of the pair is multiplied by a weight factor, whereas the sum of the weight factors of each pair is 100%. In variations of the present invention, the method further includes selecting a first image frame to be the primary image frame and a second image frame to be the secondary image frame, wherein the fusing creates a first fused image frame, the method also includes selecting the first image frame to be the secondary image frame and the second image frame to be the primary image frame, wherein the fusing creates a second fused image frame. Then, a final fused image frame is computed by averaging the first fused image frame with the second fused image frame.

Tn variations of the present invention, the computation of the differential noise group includes the steps of summing up the value of the individual pixels in the first group of pixels, thereby obtaining a first energy sum, summing up the value of the individual pixels in the second group of pixels, thereby obtaining a second energy sum subtracting the second energy sum from the first energy sum, thereby obtaining an energy differential noise group, and assigning the value of the energy differential noise group to the differential noise group.

In other variations of the present invention, to determine blurry groups of pixels, the computation of the differential noise group includes the measurement of image sharpness for primary and secondary images and subtracting the second sharpness from the first sharpness values, thereby obtain an sharpness differential noise group, then assigning the value of the sharpness differential noise group to the differential noise group. The sharpness value calculated individually to first and second groups of pixels by applying a filter to the said group of pixels, thereby obtaining a filtered group of pixels, summing up the absolute values of the individual pixels of the said filtered group thereby deteπnining the sharpness of the said group of pixels. In variations of the present invention the filter is a high pass filter. In other variations of the present invention the filter is a band pass filter. In still other variations of the present invention, to determine at least partially saturated groups of pixels, the computation of the differential noise group includes the steps of counting the number of pixels greater than a pre determined threshold value in the first group of pixels, thereby obtaining a first hot spot value, counting the number of pixels greater than the pre determined threshold value in the second group of pixels, thereby obtaining a second hot spot value, subtracting the second hot spot value from the first hot spot value, thereby obtaining a hot spot differential noise group, and assigning the value of the hot spot differential noise group to the differential noise group.

In still other variations of the present invention, the computation of the differential noise group combines several algorithms with weight factors, wherein the combination includes the steps of computing an energy product of the energy differential noise group and a pre determined weight factor, computing a sharpness product of the sharpness differential noise group and a pre determined weight factor, computing a hot spot product of the hot spot differential noise group and a pre determined weight factor, and summing up the energy product, the sharpness product and the hot spot product, thereby obtaining the differential noise group. It should be noted that the pre determined weight factor can be threshold value, a multiplication factor value or any other factor. The pre determined weight factor can also be computed dynamically, preferably using a Kalman filter.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will become fully understood from the detailed description given herein below and the accompanying drawings, which are given by way of illustration and example only and thus not limitative of the present invention, and wherein:

FIG. 1 is a block diagram illustration of an imaging system having a noise reduction device for a camera system, according to embodiments of the present invention; FIG. 2 is a block diagram illustration of an imaging system having a noise reduction device, according to variations of the present invention;

FIG. 3 schematically illustrates an embodiment of a computerized noise reduction device for a camera system, according to some variations of the present invention, configured to perform a methodology built into an image fusion module; FIG. 4 schematically illustrates an embodiment of a computerized noise reduction device for a camera system, according to other variations of the present invention, configured to perform a methodology built into an image fusion module; FIG. 5 schematically illustrates schematically illustrates an embodiment for detecting energy loss type of light obstruction, in a computerized noise reduction device for a camera system, according to variations of the present invention;

FIG. 6 schematically illustrates schematically illustrates an embodiment for detecting sharpness loss type of light distortion, in a computerized noise reduction device for a camera system, according to variations of the present invention; and

FIG. 7 schematically illustrates schematically illustrates an embodiment for detecting hot spots type of light distortion, in a computerized noise reduction device for a camera system, according to variations of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention now will be described more fully hereinafter with reference to the accompanying drawings, in which preferred embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided, so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The methods and examples provided herein are only illustrative and not intended to be limiting.

By way of introduction, a principal intention of the present invention includes providing a noise reduction device for a camera system having two or more image sensors. In particular, the noise reduction device compensates for light distortions caused by elements that block at least a portion of the light from reaching at least a portion of the image sensor array.

The methodology behind the noise reduction device of the present invention includes two basic phases: noise detection and noise distortion compensation. The noise detection phase is based on one or more of the following method: a) Light blocking type of noise, caused by opaque or semi-opaque elements, causes reduction in the light energy reaching the image sensor array. b) Blurring type of noise, caused by clear elements such as water drops, glass defects and the like, causes scattering of the incoming light energy and thereby blurring at least a portion of the image. c) Hot spots type of noise, caused by a light source such as sun light rays dispersed towards the image sensor array and thereby, causing high energy spot sensed by sensor array pixels.

Therefore, the present invention provides means for detecting the different type of light related noise.

In the present invention, the distortion compensation phase is performed by fusing two (or more) image frames synchronously acquired by two (or more) image sensors aligned to view substantially the same scene. The fusion of the two image frames is based on one or more of the following: a) Selecting the image frame determined as having less noise. b) Selecting regions from respective regions in the image frames, determined as having less noise. c) Averaging the two image frames. d) Weigh averaging, wherein the weigh is computed of from respective regions in the image frames.

The distortion compensation phase may be performed on the whole image, on groups of pixels or on individual pixels. For energy related noise, image energy is computed and compared in corresponding images. For blurring related noise, high pass or band pass filters may be used to detect the noise.

Reference is made to Figure 1, which is a block diagram illustration of an exemplary imaging system having a light distortions related noise reduction device 100 for a camera system, according to embodiments of the present invention. Noise reduction device 100 includes two image acquisition devices HOa and HOb, each of which yields respective image frames 130a and 130b, and image fusion module 140 which yields a single output image frame 150. Image acquisition devices 110 are configured to acquire one or more image frames and have substantially the same FOV, pointing substantially to the same distal object 20. Each camera has different noise sources, such as glass 120 having light distorting element such as dirt, and other noise sources, yielding different distortions in the respective image frames 130. Image fusion module 140 processes image frames 130 to yield a single output image frame 150. in which the noise of both cameras is substantially removed. It should be noted that image acquisition devices HOa and HOb are preferably aligned, preferably in sub-pixel accuracy, to ensure that image acquisition devices HOa and HOb are pointing substantially to the same distal object 20.

In variations of the present invention, more than two image acquisition devices 110 are employed in the noise reduction camera system. Image fusion module 140 may be embodied in various methods, to operatively yield an output image frame 150.

Reference is now made to Figure 3, which schematically illustrates a first embodiment of a computerized noise reduction device 300 for a camera system. configured to perform a methodology built into an image fusion module 340. In the example shown in Figure 3, protecting glass 120 is covered with dirt spots 122, which dirt spots 122 at least partially block light from reaching image acquisition devices HOa and HOb. Image frames 130a and 130b include respective noisy regions 132a and 132b, which respective noisy regions 132a and 132b are substantially different in the two image frames (130a and 130b). The methodology of the first embodiment includes the following steps: a) adding image frames 130a and 130b by sum unit 342, to yield a noise integrated frame 346a; b) computing the difference between frame 130a and 130b (absolute value) to yield a noise frame 346a; and c) adding noise frame 346a to noise integrated frame 346a whereby creating an output image frame 350, which output image frame 350 has noisy regions 132a and 132b substantially removed.

Preferably, in other embodiments of the present invention, the methods of the present invention further includes the step of subdividing the image frames 130a and 130b into respective groups of pixels, the respective groups of pixels being spatially symmetric, wherein a first group of pixels represents a selected group of pixels from image frame 130a and wherein a second group of pixels represents a selected group ol ^* pixels from image frame 130b, wherein the subdividing is performed before fusing the image frames 130a and 130b. A group of pixels includes NXM pixels, wherein N and M are integers. In some variations of the present invention, both N and M are equal to 1. In other variations of the present invention, both N and M are the dimensions of the corresponding image frames.

Reference is now made to Figure 4, which schematically illustrates a second embodiment of a computerized noise reduction device 400 for a camera system, configured to perform a methodology built into an image fusion module 440. In the example shown in Figure 4, protecting glass 120 is covered with dirt spots 122, which dirt spots 122 at least partially block light from reaching image acquisition devices 110a and 110b. Image frames 130a and 130b include respective noisy regions 132a and 132b, which respective noisy regions 132a and 132b are substantially different in the two image frames (130a and 130b). In the example shown in Figure 4, image frames 130a and 130b are tessellated into 16 groups of pixels to form respective tessellated image frames 446a and 446b.

The methodology of the second embodiment includes the following steps: a) selecting image frame 130a as the primary image frame and image frame 130b as the secondary image frame; b) for each pair of respective primary and secondary groups of pixels from the respective subdivided primary image frames 130a and secondary image frame

130b, calculating a differential signal to noise factor between the respective primary and secondary groups of pixels, thereby creating a differential noise group; and c) to fuse image frames 130a and 130b, for each pair of respective group of pixels, performs the following steps: i. if the differential noise group is less than a pre selected threshold values, the value of the respective group of output pixels in fused image frame 450 is set to be the value of each respective pixel in the primary group of pixels; else ii. the value of the respective group of output pixels in image frame 450 the fused image frame is set to be the values of each respective pixel in the primary group of pixels. In a third embodiment includes the following steps: a) for each pair of respective first and second groups of pixels from the respective subdivided first image frames 130a and second image frame 130b, calculating a differential signal to noise factor between the respective primary and secondary groups of pixels, thereby creating a differential noise group; and b) to fuse image frames 130a and 130b, for each pair of respective group of pixels, performs the following steps: i. determining a first weight factor based as a function of the differential noise group ; ii. calculating a first weighted group of pixels, wherein each first weighted pixel in the first weighted group of pixels is assigned the product of the value of the respective pixel value in the first group of pixels and the first weight factor; iii. determining a second weight factor based as a function of the differential noise group; iv. calculating a second weighted group of pixels, wherein each second weighted pixel in the second weighted group of pixels is assigned the product of the value of the respective pixel value in the second group of pixels and the second weight factor; and v. individual pixels in the respective group of output pixels in image frame 150, to be the normalized sum of the respective individual values of the first weighted pixel and the second weighted.

A variety of computation method can be performed to compute the differential noise group. Different methods are used to determine the noise that can be caused by the different light distorting elements such as dirt, water drops and other noise sources.

Reference is now made to Figure 5, which schematically illustrates an embodiment for detecting energy loss type of light obstruction, in a computerized noise reduction device 500 for a camera system, configured to perform a methodology built into an image fusion module 540. In the example shown in Figure 5. protecting glass

120 is covered with dirt spots 122, which dirt spots 122 at least partially block light from reaching image acquisition devices 110a and 110b. Image frames 130a and 130b include respective noisy regions 132a and 132b, which respective noisy regions 132a and 132b are substantially different in the two image frames (130a and 130b). In the example shown in Figure 5, image frames 130a and 130b are tessellated into 16 groups of pixels to form respective tessellated image frames 546a and 546b.

To determine pixels with energy loss, the computation of the differential noise group includes the steps of summing up the values of the individual pixels in a first group of pixels 547a by a computational unit 544a, thereby obtaining a first energy sum; summing up the values of the individual pixels in a second group of pixels 547b by a computational unit 544b, thereby obtaining a second energy sum; subtracting the second energy sum from the first energy sum, thereby obtaining an energy differential noise group, and assigning the value of the energy differential noise group to the differential noise group.

Reference is now made to Figure 6, which schematically illustrates an embodiment for detecting sharpness loss type of light distortion, in a computerized noise reduction device 600 for a camera system, configured to perform a methodology built into an image fusion module 640. In the example shown in Figure 6, protecting glass 120 is covered with blurring elements 122, which rain drops spots 122 blur at least a portion of image frames 130a and 130b. Image frames 130a and 130b include respective noisy regions 132a and 132b, which respective noisy regions 132a and 132b are substantially different in the two image frames (130a and 130b). In the example shown in Figure 6, image frames 130a and 130b are tessellated into 16 groups of pixels to form respective tessellated image frames 646a and 646b.

To determine blurry groups of pixels, the computation of the differential noise group includes the steps of applying a filter 642a to a first group of pixels 647a, thereby obtaining a first filtered group of pixels, summing up (by a computational unit

648a) the absolute values (computed by a computational unit 644a) of the individual pixels of the first filtered value thereby determining the sharpness of the first group of pixels. Then, applying a filter 642b to a first group of pixels 647b, thereby obtaining a second filtered group of pixels, summing up (by a computational unit 648b) the absolute values (computed by a computational unit 644b) of the individual pixels of the second filtered value thereby determining the sharpness of the second group of pixels.

The following steps include subtracting the sharpness of the second group of pixels from the sharpness of the first group of pixels, thereby obtaining a sharpness differential noise group, and assigning the value of the sharpness differential noise group to the differential noise group. In variations of the present invention the filter is a high pass filter. In other variations of the present invention the filter is a band pass filter.

Reference is now made to Figure 7, which schematically illustrates an embodiment for detecting hot spots type of light distortion, in a computerized noise reduction device 700 for a camera system, configured to perform a methodology built into an image fusion module 740. Image frames 130a and 130b include respective noisy regions 132a and 132b, which respective noisy regions 132a and 132b are substantially different in the two image frames (130a and 130b). In the example shown in Figure 7, image frames 130a and 130b are tessellated into 16 groups of pixels to form respective tessellated image frames 746a and 746b.

To determine pixels with at least partially saturated groups of pixels, computation of the differential noise group includes the steps of counting the number of pixels greater than a pre determined threshold value (computed by a Comparator 742a) in a first group of pixels 747a by a counter 744a, thereby obtaining a first hot spot value, counting the number of pixels greater than the pre determined threshold value

(computed by a Comparator 742b) in the second group of pixels 747b by a counter 744b, thereby obtaining a second hot spot value, subtracting the second hot spot value from the first hot spot value, thereby obtaining a hot spot differential noise group, and assigning the value of the hot spot differential noise group to the differential noise group.

In still other variations of the present invention, the computation of the differential noise group includes the steps of computing an energy product of the energy differential noise group and a pre determined weight factor, computing a sharpness product of the sharpness differential noise group and a pre determined weight factor, computing a hot spot product of the hot spot differential noise group and a pre determined weight factor, and summing up the energy product, the sharpness product and the hot spot product, thereby obtaining the differential noise group.

In a fourth embodiment, the methodology of image fusion module 140 includes the following steps: a) setting the value of the individual pixels of output image frame 150 to be the average of the respective individual pixel values of image frames 130a and 130b.

In a fifth embodiment, the methodology of image fusion module 140 includes the following steps: a) computing weighed value of each pair of corresponding pixels (or groups of pixels), whereas the sum of the weight factors of each pair is 100%. b) setting the value of the individual pixels of output image frame 150 to be the weigh average of the respective individual pixel weighed values of image frames 130a and 130b.

In a sixth embodiment, the methodology of image fusion module 140 includes the following steps: a) computing the energy of image frame 130a; b) computing the energy of image frame 130b; and c) determining: i. if the energy of image frame 130a is less than the energy of image frame 130b, setting the corresponding output image frame 150 to be image frame 130b; else ii. setting the corresponding output image frame 150 to be image frame 130a.

Reference is made to Figure 2, which is a block diagram illustration of an exemplary imaging system having noise reduction device 200, according to variations of the present invention. As in system 100, imaging system 200 includes two image acquisition devices 110a and 110b for acquiring respective image frames 130a and 130b. Both image acquisition devices 110 have substantially the same FOV and pointing substantially to the same distal object 20. Imaging system 200 further includes an image sensor fusion module 140a which fusion module selects valid pixels/groups- of-pixels from either image frames 130a and 130b to yield a single output image frame 150a, an image fusion module 140b which fusion module selects valid pixels/groups- of-pixels from either image frames 130a and 130b to yield a single output image frame 150b, an averaging module 240, which averaging module averages image frames 150a and 150b to yield a single output image frame 250.

In a seventh embodiment, the methodology of image fusion modules 140a, 140b and averaging module 240 includes the following steps: a) selecting image frame 130a as the primary image frame and image frame 130b as the secondary image frame, wherein fusing image frame 130a and image frame 130b creates a first fused image frame 150a; b) selecting image frame 130b as the primary image frame and image frame 130a as the secondary image frame, wherein fusing image frame 130a and image frame 130b creates a second fused image framel50b; and c) averaging each pair of pixels from image frames 150a and 150b and setting the value of a corresponding final output pixel in image frame 250 to the computed average.

The invention being thus described in terms of embodiments and examples, it will be obvious that the same may be varied in many ways. Such variations are not to be regarded as a departure from the spirit and scope of the invention, and all such modifications as would be obvious to one skilled in the art are intended to be included within the scope of the claims.