PETRESCU, Stefan Mirel (Str. Clucerului, Nr. 55 Ap. 3Sector 1, Bucharest, RO)
NANU, Florin (Str. Clucerului Nr 55 Ap. 3, Sector 1, Bucuresti, RO)
CAPATA, Adrian (35 Ardilaun Road, Newcastle, Galway, IE)
CORCORAN, Peter (Cregg, Claregalway, Galway, IE)
SUSANU, George (268 Tirellan Heights, Headford Rd, Galway, IE)
PETRESCU, Stefan Mirel (Str. Clucerului, Nr. 55 Ap. 3Sector 1, Bucharest, RO)
NANU, Florin (Str. Clucerului Nr 55 Ap. 3, Sector 1, Bucuresti, RO)
CAPATA, Adrian (35 Ardilaun Road, Newcastle, Galway, IE)
CORCORAN, Peter (Cregg, Claregalway, Galway, IE)
| I l Claims: 1. A color filter enhancement method for a portable digital image acquisition device, comprising: using optics, a color sensor array and a processor to acquire and process digital images; obtaining a first relatively underexposed and sharp image of a scene ("sharp image") by exposing a first set of pixels of the sensor array; obtaining a second relatively well exposed and blurred image of the same scene ("blurred image") by exposing a second set of pixels of the sensor array for a longer duration than the sharp image, and wherein the second set of pixels interleaves the first set of pixels; and applying color or luminance information, or both, from the blurred image to the sharp image, thereby constructing a color-enhanced and/or luminance-enhanced version of the sharp image. 2. The method of claim 1, wherein obtaining the blurred image comprises digitally exposing color pixels of the sensor array for a first digital exposure duration; and wherein obtaining the sharp image comprises digitally exposing white pixels of the sensor array for a second digital exposure time shorter than the first digital exposure duration. 3. The method of claim 2, wherein the constructing of the color-enhanced and/or luminance-enhanced version of the sharp image comprises using data from both the color pixels exposed for the first digital exposure duration and the white pixels exposed for the second digital exposure duration. 4. The method of any of claims 2-3, wherein the digitally exposing of the color pixels and the white pixels for different exposure times comprises clocking the color pixels and the white pixels independently. 5. The method of any of claims 2-4, wherein the digitally-exposing of the color pixels and the white pixels for different exposure times comprises including sensor data over different temporal ranges. 6. The method of claim 5, wherein the different temporal ranges are overlapping. 7. The method of claim 6, wherein a first temporal range corresponding to the digitally- exposing of the color pixels comprises an entire second temporal range corresponding to the digitally-exposing of the white pixels. 8. The method of any of claims 2-7, wherein the color pixels comprise greater motion blurring effect than the white pixels due to the color pixels being digitally-exposed for a longer duration than the white pixels, and the method further comprises compensating blurring in the color pixels using less-blurred data from the white pixels. 9. The method of any of the above claims, wherein the second digital exposure time comprises not more than half of the first digital exposure time. 10. The method of any of the above claims, wherein the second digital exposure time comprises approximately a third of the first digital exposure time. 1 1. The method of any of the above claims, wherein the color sensor array comprises a CMOS-based sensor. 12. The method of any of the above claims, further comprising storing, transmitting, or displaying the color-enhanced and/or luminance-enhanced version of the sharp image, or a further processed version, or combinations thereof. 13. The method of any of the above claims, wherein at least a segment of the exposure periods of the sharp and blurred images temporally overlap. 14. A portable digital image acquisition device, comprising: optics and a color sensor array for acquiring a color digital image; a processor; one or more processor-readable media having code embedded therein for programming the processor to perform a color filter enhancement method according to any of the above claims. 15. One or more processor-readable media having code embedded therein for programming the processor to perform a color filter enhancement method according to any of the above claims. |
Field of the Invention
The present invention relates to an image processing method and apparatus.
Description of the Related Art
Sensor arrays for digital cameras exist for capturing color photos. Sensors known as
RGBW sensors are used for capturing red, green, blue colors, and for capturing luminance information for multiple pixels of an array of pixels. The red, green and blue pixels include filters such that only certain narrow ranges of wavelengths of incident light are counted. The white pixels capture light of red, green and blue wavelengths, i.e., of a broader range of wavelengths than any of the blue, green and red pixels. Thus, the white pixels are typically brighter than any of the blue, red and green pixels if they are exposed for the same duration.
Noise removal algorithms tend to blur face regions in an undesirable manner. Noise removal algorithms are described at United States patent applications nos. 1 1/856,721
(FN-204), 1 1/861,257 (FN-223), 10/985,650, 1 1/573,713, 1 1/421 ,027, 1 1/673,560, 11/319,766, 11/744,020, 11/753,098, 1 1/752,925, and 12/137,113.
Kodak has developed a RGBW color filter pattern differing from the previously known Bayer Color Filter. The RGBW pattern of Kodak is referred to as a Color Filter Array
(CFA) 2.0. One half of cells in a RGBW pattern are panchromatic, i.e. sensing all color spectrum (Y component) - usually called white cells. This way more light energy is accumulated in the same amount of time than for color pixels. A Bayer filter uses only
1/3 (~ 0.33) of color spectrum energy. An RGBW filter uses 4/6 (~ 0.67) of energy, where 1/2 comes from white cells and 1/6 from RGB cells. A CFA Array looks something like the following:
WBWG ...
BWGW ...
WGWR ... RWRW ...
Summary
A color filter enhancement method is provided for a portable digital image acquisition device. The method includes digitally exposing color pixels of a color sensor array for a first digital exposure duration. White pixels of a color sensor array are digitally exposed for a second digital exposure time shorter than the first digital exposure duration. A color filter enhanced digital image is generated using data from both the color pixels exposed for the first digital exposure duration and the white pixels exposed for the second digital exposure duration shorter than the first digital exposure duration. The color filter enhanced digital image and/or a further processed version is stored, transmitted, communicated, displayed, and/or projected.
The second digital exposure time may be less than half of the first digital exposure time, for example, it may be approximately a third of the first digital exposure time.
The digitally exposing of the color pixels and the white pixels for different exposure times may include clocking the color pixels and the white pixels independently.
The digitally-exposing of the color pixels and the white pixels for different exposure times may involve including sensor data over different temporal ranges. The different temporal ranges may be overlapping. A first temporal range corresponding to the digitally-exposing of the color pixels may include an entire second temporal range corresponding to the digitally-exposing of the white pixels.
The color pixels may include greater motion blurring effect than the white pixels due to the color pixels being digitally-exposed for a longer duration than the white pixels. The method may further include compensating blurring in the color pixels using less-blurred data from the white pixels.
The color sensor array may include a CMOS-based sensor.
One or more processor-readable media are also provided that have code embedded therein for programming the processor to perform a color filter enhancement method in accordance with any of the methods described herein.
A portable digital image acquisition device is also provided including optics and a color sensor array for acquiring a color digital image, a processor, and one or more processor- readable media having code embedded therein for programming the processor to perform a color filter enhancement method that comprises any of the methods described herein.
Brief Descriptions Of The Drawings
Several embodiments will now be described, by way of example, with reference to the accompanying drawings, in which:
Figure 1 is a block diagram illustrating a method of color filter enhancement for a portable digital image acquisition device in accordance with certain embodiments; and
Figure 2 is a block diagram illustrating a portable digital image acquisition device with color filter enhancement in accordance with certain embodiments.
Description of the Preferred Embodiment
An advantageous implementation of the present invention may use a modified CMOS sensor array. In particular, a form of imaging sensor referred to as the Color Filter Array (CFA) incorporates an alternative arrangement of light sensitive elements to the more conventional Bayer pattern array employed by most standard sensors. Specifically, a CFA sensor incorporates an array of pixels arranged in the pattern: WBWG ... BWGW ... WGWR ...
RWRW ... and so on ...
This contrasts with the more conventional Bayer pattern, viz:
BGBG ... GRGR ... BGBG ... GRGR ...
Note that the CFA arrangement includes special pixels which are sensitive to white light, rather than being specifically tuned to a Red, Green or Blue spectral response. As these white pixels are sensitive to white light, which combines all three color sensitivities, they do tend to charge more rapidly than the conventional pixels which are sensitized to Red, Green or Blue spectral frequencies.
In this embodiment, the sensor array is designed to enable the white pixels which are commingled with the RGB pixels to be clocked independently of those color specific pixels.
Such an arrangement implies that these W pixels are arranged on a separate data bus and are clocked independently of the RGB pixels. Thus, it is possible to extract a "luminance- only" image from the sensor independently of the "chrominance image" provided by the
RGB pixels. This involves additional real estate on the sensor IC, but fortunately, as this embodiment may use a smaller image, the density of W pixels employed by the standard CFA sensor may be advantageously reduced. In practice a 4: 1 or 8:1 scaling-down of the "luminance image" is adequate and implies that only 1 in 16, or even 1 in 64 pixels should be W pixels.
Now as the luminance-only image obtained from the white pixels will charge more quickly, it can be offloaded from the sensor as a fully exposed image about three times faster than the RGB pixels. Where an underexposed image is sufficient, the speed of acquisition of this luminance image can be significantly more than an order of magnitude faster than the corresponding RGB image. Further, this luminance-only image derived from the W pixels can be offloaded while the RGB pixels are still charging. This ensures that sharp, underexposed images can be obtained at a time which overlaps the acquisition of the normally exposed, blurred main "chrominance image".
The offloading of the sharp, underexposed image may be timed to co-incide with the midpoint of the acquisition of the normally exposed, blurred color image.
Other advantages of this modified sensor arrangement should also be noted. For example, it is possible to implement an embodiment described in US published patent application US20050041 121 , where a first luminance image is obtained immediately prior to the main RGB acquisition and the actuation of the flash; a second luminance image is obtained during the main RGB acquisition and these two luminance images are used to determine the location of potential red-eye defects. The final determination and correction of red-eye defects can be performed on the main RGB image. This approach has the advantage of eliminating the need to perform alignment between the pair of luminance images and the final RGB image can be enhanced (sharpened) by combining it with one of the two temporally proximate luminance images.
In another embodiment, a face tracking algorithm can be run on the luminance images acquired form such a modified sensor. Such face tracking algorithm may be based on classifier cascades trained on luminance images this approach enables the face tracker to operate directly on the luminance images acquired from the W pixels. Advantageously, where such face tracker algorithm operates using a fixed size classifier chain as described, for example, at US patents 7,315,631, 7,403,643, 7,317,815, 7,269,292, and 7,316,630, and US published application no. 2008/0037840, 2008/0037839 and 2008/0037827, and US application serial no. 12/063,089, the main tracking algorithm for detecting new faces can operate on the luminance images obtained from the sensor, independent of a normal preview stream obtained from the main array of RGB pixels. A refined analysis can be subsequently performed on the corresponding RGB image. A particular advantage of this configuration arises where low-lighting conditions occur. As noted above the W pixels charge at a rate at least three times faster than the conventional RGB pixels and thus they can obtain better images in low light conditions. This in turn leads to an extension of the lower lighting levels at which a face tracker algorithm can successfully operate and enables higher frame rates to be achieved in extreme conditions.
Now referring to Figures 1 and 2, embodiments are provided of enhanced methods, hardware and software involving color sensor arrays such as RGBW sensor arrays that include separate color RGB pixels and white W pixels or luminance pixels. In particular, the W pixels tend to saturate or at least are more greatly exposed than the RGB pixels. The RGB pixels receive on average 1/3 of the light intensity of the W pixels, and thus tend to be underexposed or at least less exposed than the W pixels on average by a third and at most more than 1/3. In accordance with certain embodiments, the RGB pixels may be clocked independently of the W pixels and thus can have a longer exposure time.
Figure 1 illustrates a method in accordance with certain embodiments. Color pixels of a color sensor array are digitally exposed for a first digital exposure duration at (2), while white pixels of the array are exposed at (4) for a second digital exposure time shorter than the first digital exposure duration. The second digital exposure time may be half or less than the first digital exposure time, e.g., a third of the first digital exposure time. The exposure times for the red R, blue B, and green G pixels may be the same as or different from each other, for example, in accordance with certain determined color enhancements of the image. The exposure times for any of the red R, blue B, green G and/or white W pixels may vary over the multiple pixels of the sensor array according to spatially- dependent color enhancements, contrast enhancements, and/or intensity enhancements. The pixels may be further adjusted based on other digital image enhancement methods as may be described in other references.
The different digital exposure durations may be provided either by gathering light for different exposure durations, and/or by using data gathered for different periods of time, i.e., while not using other data gathered at times outside a determined period particularly for the white W pixels, and/or by using computed data based on gathered data such as by multiplying or otherwise increasing color pixel data such as by adding data from surrounding pixels in a stand alone approach or in tandem with a dedicated or unrelated smoothing process. Alternatively, a color sensor may be provided that includes a ratio of more or less color pixels to white pixels such as to contribute to intensity balancing and/or certain pixels may be turned off or data gathered from certain pixels may simply not be used in pre-balancing the intensities, or may be discarded in a post-balancing filter. Certain pixels may be blocked, reduced or filtered mechanically, optically, electronically, or digitally such as in accordance with program code, and certain pixels may be likewise enhanced as to the extent of their respective contributions.
The color pixels and the white pixels may be clocked independently or differently such as to provide the different exposure durations. The color pixels and white pixels may be clocked over different temporal ranges which are proximate in time and may or may not overlap. In one example, a temporal range of exposure of the white pixels may lie entirely or partially within a longer temporal range of exposure of the color pixels or a subset thereof, or a temporal range of exposure of the white pixels may just precede or just follow a longer temporal range of exposure of the color pixels or subset thereof.
The color pixels will generally exhibit a greater motion blurring effect than the white pixels due to the color pixels being digitally-exposed for a longer duration than the white pixels. Thus the method of Figure 1 may include at (5) compensating blurring in the color pixels, for example, by using less-blurred data from the white pixels.
A color filter enhanced digital image is generated at (6) using data from both the color pixels digitally or otherwise exposed for the first digital exposure duration and the white pixels digitally or otherwise exposed for the second digital exposure duration. The color filter enhanced digital image or a further processed version is stored, transmitted, communicated, displayed, and/or projected.
Figure 2 illustrates in block form a portable digital imaging apparatus such as a portable camera or camera phone. The apparatus may include optional display 12 and flash 14, as well as optics 16 and digital camera sensor 18. Alternatively, methods according to certain embodiments may be applied to film images that are digitized and processed accordingly. The color sensor includes several white pixels W, as well as red pixels R, blur pixels B and red pixels R, although not necessarily in the ratios or positions indicated in the example sensor (18) of Figure 2. The color sensor array (18) may be or may include a CMOS-based sensor or sensor elements.
The portable digital imaging apparatus include a processor 20 and memory 22 including program code for image processing and handling of image acquisition. The digital image acquisition duration for color pixels RGB is increased compared with that for white pixels W in the sensor array 18, or that for white pixels W is decreased compared to color pixels
RGB. There are multiple ways that the apparatus of Figure 2 can achieve this. In one aspect, an image is exposed to the sensor for the same time period for both color pixels RGB and white pixels W, and then data gathered by the white pixels for only a fraction of that duration is used along with the color pixel data for the entire duration or at least a longer duration than that of the white pixels W. The apparatus may be programmed such that the ratio of the durations used may be proportional to the sensor configuration, such as to depend on the ratio white pixels W to color pixels RGB. For example, the array 18 shows a 1 : 1 ration of white pixels W to color pixels RGB, but that ratio could be 1 :2 or 1 :3 or even 2:1 or otherwise. The white pixels W may be the same or different size as the color pixels RGB.
In another aspect, clocking component 24, which can include a single clock or multiple clocks, turns on the white pixels for gathering image data for a different, shorter period than it does for the color pixels. For example, the clocking component 24 may trigger the beginning of gathering data by both the color pixels RGB and the white pixels W at time to. Then, the clocking component may trigger the stopping of gathering data by the white pixels at time tj . The color pixels RGB would continue to gather data after time tj. Then, the clocking component 24 would trigger the stopping of gathering data by the color pixels RGB at time t 2 . Alternatively, the clocking component can trigger the color pixels RGB and the white pixels W to begin gathering data at different times t 0 ! and t 0 2 , and may stop the gathering of data by the white pixels W and color pixels RGB at the same time or such that the color pixels RGB are stopped before the white pixels W, and the durations may overlap entirely, partially overlap or not overlap at all. In the latter case, it is desired that the later period of the two data gathering periods closely follows the earlier period, so that the color pixels gather data of substantially the same scene as the white pixels.
The clocking component 24 could be used in the earlier aspect too. The data would be gathered by the white pixels W and color pixels RGB for the same duration between time to and ti according to the clocking component. However, the clocking component would tag other times t 2 , t 3 , and perhaps times, between time to and ti. Then, only data gathered by the white pixels between times t 2 and t 3 would be used, while the data gathered by the color pixels RGB between times to and ti, or some other duration longer than t 3 minus t 2 , would be used.
In a further aspect, the sensor design may be modified so that the acquisition time can be independently increased for color pixels RGB compared to white pixels W. White pixels W in this aspect would be exposed during a normal period of time (or slightly under exposed) depending on the given scene condition. As color pixels RGB use only 1/3 of light energy comparing to white pixels W, the exposure time for the color pixels RGB would be increased, e.g., by 3 times, without suffering significant over exposure assuming that white pixels W were not overexposed and that the color distribution is approximate equal for the red R, blue B and green G components of the scene.
A factor is that there may be movement of scene objects or sensor shake, jitter or panning. Then, the RGB pixels RGB would be more motion-blurred, e.g. by 3 times, compared with the white pixels W, because the exposure duration of the RGB pixels is so much longer. For this, the RGB blurring can be compensated by a software component that will use the correction from white pixels W. A process described at US application
11/856,721 and PCT Application No. PCT/EP2007/009939, may be used to enable the white pixels W in such a sensor to act as a relatively sharp and possibly underexposed reference image, or one or more dedicated reference images may be used, where the color pixels RGB provide a relatively well exposed but blurred color image. A single sharp image can be provided by such an array. In the context of adjusting images with motion blur effects, United States patent application serial nos. 1 1/566,180, 10/985,657, 10/985,650, 10/986,562, 1 1/753,098, 12/137,1 13, 1 1/690,836, 1 1/856,721 , 1 1/764,578, PCT/US2008/061328, and 61/023,774 are referred to. This process may be performed and/or provided according to a dynamic range compression algorithm or similar. This may allow enhancement of dynamic range of an image as two sub-images are acquired with different exposures. Different acquisition durations of white pixels W and color pixels RGB is possible with CMOS based sensors.
Embodiments of the invention enable a full exposure range of a RGBW sensor array to be used.
In addition, in methods that may be performed according to preferred embodiments herein and that may have been described above, the operations have been described in selected typographical sequences. However, the sequences have been selected and so ordered for typographical convenience and are not intended to imply any particular order for performing the operations, except for those where a particular order may be expressly set forth or where those of ordinary skill in the art may deem a particular order to be necessary.