METHOD AND SYSTEM TO REDUCE FLfCKER ARTfFACTS IM CAPTURED

IMAGES RELD

Embodiments of the invention relate to a method and system to reduce flicker artifacts in a captured image.

BACKGROUND

The instantaneous alternating current (AC) power is a continuously varying sinusoidal wave. Therefore, the instantaneous light intensity function of a lamp aiso varies accordingly and can be modeled as:

where /' is the observed instantaneous light intensity. / is a constant term respect to time. .-4 [ cos[β;(M-(9)] | is a varying term and is due to tempora!

fiuctuatioπs in ambient light , A is a constant, & ~2πf \s the power frequency at

either 5QHz or 60Hz, t is the time, and θ is the initial phase. The varying term causes artifacts in a captured image. This phenomenon is known as light flicker. / ^! {f) is a periodic function. Because of the absoiute term, its period is 1/{2f), haif of the cosine period.

Light flicker could cause noticeable artifacts during image and video capture. This is because the image pixel vaiue is proportional to the integral of the instantaneous light function:

where or -θ is the starting time or the initial phase, T is the integration time or the camera exposure time, V is the captured image pixel value. Y {a J) has a period

of T - 1/2/ with respect to T, where r is the half of the power frequency period.

From this equation, it can be seen that the Y value changes as a function oia ,

except when T is an integer multiple of r _s 1/(2/). In that case, the Y value stays the same regardless of the initial phase a .

Depending on a camera's shutter mechanism, the light flicker causes various artifacts in the image, if T is not set to an integer number of the period r . There are two popular types of shutters, namely snapshot and rolling shutter, In the snapshot case _: al! rows in the image are exposed at the same starting and ending time. If a sequence of video frames or still images is captured in the snapshot mode, each frame is exposed at a different time. Pixel values then vary from frame to frame even if the scene is static, which is easily noticeable during video playback.

In the rolling shutter case, only a number of rows are exposed at any- given time, imagine an exposure window that covers the exposed rows. The width of the window is the number of rows. The window moves one row at a time from the top to the bottom , and then rotates back to the top. At each wsndow location, pixe! values in the longest exposed row are read out. The exposure time is nβ , where n is the number of rows in the window and β is the time difference

between the read-outs of two consecutive rows, β is usually constant within

each image. Because each row is exposed at a different starting time, the pixe! values shift from row to row. The net effect is a vertical periodic signal being superimposed on the original image. Figure 1 illustrates this effect. The image is generated based on Equation (2). A constant S is used for al! the pixei location, in other words, without the flicker, the image should be flat. Due to the flicker, sinusoidal variations along the vertical direction can be observed

When the magnitude of the fluctuation A is small, the bright and dark stripes in Figure 1 are less noticeable in a single image. However, in a sequence of video frames, because of the phase difference between two consecutive frames, the bright and dark stripes move from frame to frame and are thus more noticeable.

One method to correct for flicker artifacts in a captured image is to set the exposure time T to an integer number of the period, 1/(2f). However, with this method the allowable exposure time is constrained to a fixed set of values, in the

example of 60Hz power Sine, they are; 1/120 s, 2/120 S ₁ 3/120 s. Any values between those numbers would not be allowed. Moreover the exposure time cannot be set to be smaller than 1/12O s without avoiding the flicker.

SUMMARY

In one embodiment, a method for controlling artifacts in an image due to a presence of flicker in ambient light is provided. The method comprises determining a power frequency for the ambient Sigh!; determining a magnitude and phase of flicker fluctuation in the image όue to the flicker; and adjusting the image for the flicker fluctuation based on the power frequency, magnitude, and phase.

Other aspects of the invention wiil be apparent from the detailed description below.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention wii! now be described, by way of example, with reference to the accompanying diagrammatic drawings, m which:

Figure 1 show a high-Sevei block diagram of a camera system, in accordance with one embodiment of the invention;

Figure 2 shows a high-level block diagram of an image processor, in accordance with one embodiment of the invention; and

Figure 3 shows a flowchart for a fucker reduction technique, in accordance with one embodiment of the invention

DETAILED DESCRtPTfON

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the invention, ft will be apparent, however, to one skilled in the art, that the invention may be practiced without these specific details, in other instances, structures and devices are shown at biock diagram form only in order to avoid obscuring the invention.

Reference in this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. The appearances of the phrase in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Moreover, various features are described which may be exhibited by some embodiments and not by others. Similarly, various requirements are described which may be requirements for some embodiments but not other embodiments.

Embodiments of the present invention disclose techniques reducing or removing light flicker artifacts in captured images , regardless of whether the images are still images or video images. The techniques are designed for rolling shutter cameras, in one embodiment, there are three steps. The first step is to

determine the AC power frequency, e.g. 50Hz or 60Hz. The second step is to determine the magnitude and the phase of the flicker fluctuation in captured images, given the power frequency. The third step reduces or removes the fucker artifacts from the images based on the determined power frequency, magnitude, and phase. Advantageously, the exposure time T need not be limited to an integer number of the period, 1/(20-

Embodiments of the present invention aiso cover an image processor that impiements the techniques of the present invention. A camera system that includes an image processor that implements the techniques of the present invention is aiso covered.

Turning now to Figure 1 of the drawings, there is shown a camera system 10 in accordance with one embodiment of the invention. The camera system 10 includes camera optics 12 coupled to an image sensor 14. The image sensor 14 is in turn coupled to an image processor 16. As will be seen from Figure 2 of the drawings, the image processor 16 includes flicker reduction logic 18, and an image buffer 20 _; among other components that have not been shown so as not to obscure the invention, but which components one of ordinary ski!! in the art would understand to form part of the image processor 16, The flicker reduction iogic 18 may be implemented in software, hardware _: or a combination of both software and hardware. One function of the flicker reduction logic 18 is to

process an image captured by the image sensor 14, thereby to at ieast reduce flicker artifacts in the image due to ambient light.

The techniques to reduce flicker reduction in captured images are based

on the Equation (2). Since Y(a J') as defined in Equation (2) has period of π/ω

with respect to T, in one embodiment it is assumed that 0 < / ^' < π/ω . Y(α ) may

be evaluated to the following four cases, in one embodiment of the invention:

in on embodiment, Y(α, T) may be evaluated to the following:

fπ one embodiment 1A and 1 B may be combined, and 2A and 2B may be combined as:

!t is to be observed that the first term IT is a constant term with respect to time

while the second term is the siπusoidai term. Moreover, when T - kπjω where k

is an integer, both 1 and 2 may be evaiυated to:

Equation (3) confirms that when 7 ^' ~ kπjω , Y (a, T) is a constant function with

respect to a , the initial time.

Turning now to Figure 3 of the drawings , there is shown a flowchart of the flicker reduction technique performed by the image processor 16, in accordance with one embodiment of the invention. As will be seen, at biock 22 the power frequency for the ambient Sight is determined. Thereafter, at block 24 a magnitude and phase for the fucker fluctuation is determined. Finally at block 26, the image is adjusted for the fucker fluctuation based on the power frequency, magnitude, and phase of the ambient light.

Determ i nation of power frequency at block 22

There are two matches for the power frequency, viz. 50Hz or 60Hz. Thus in order to determine the power frequency, in one embodiment differences between images taken at exposure times matched to possible choices (50Hz or 60Hz) for the power frequency are compared as follows. First two sets of images under the ambient or AC powered light are taken. Each set consists of two images shot of the same scene. For simplicity of discussion, assume the first

images are taken at time 0. while the second images are taken at time a _\ and

a ₂ , respectively. Let

The exposure times are given by J _s ^~ kπ/ω, and /;. ^~ kπjω _: , where I is

the exposure time for the first image and T ₂ is the exposure time for the second

image, and where k is an integer number set to be suitable for the light condition. Further a i and a ₂ are selected so that

Based on the foregoing, either the first and second image in the first set wil! be identical, or the first and second image in the second set will be identical, depending on whether the power frequency is 50Hz or 60Hz, In practice, noise and possibie motion would introduce smail differences between the two images. Thus, in one embodiment the images are adjusted for motion using a motion compensation technique before comparing the images. However, if the exposure time is matched to the wrong frequency, Equation (3) no ionger hoids and there should be a significant difference between the two images, in one embodiment, differences calculated from the two image sets are compared. The power frequency that gives the smaller error or difference is selected as the power frequency for the ambient light, it is to be appreciated that the use of 50Hz and

60Hz as possible choices of the power frequency of the ambient Sight is for illustrative purposes only and that techniques disclosed herein are applicable for other frequencies of ambient light.

Pelerminatioπ of Flicker Magnitude and . Phase.at.biock.24

The flicker magnitude and phase _: A and a in Equation {2} _: are difficult to estimate because the flicker term is not directly compared. There is also the signal term !T that differs at each pixel location, in order to remove the signal term, in one embodiment two images are taken. A first image is taken with the

exposure set to be a full period of V, ./; - πjω . For the first image one of the

images taken at block 22 may be used. Thereafter, a second image is taken with

an exposure of a half period of Y ₁ 7: ~ π/(2ω) . it is assumed that the first row of

the first image is taken at time O ₁ and the first row of the second image is taken at timeα., . in other words, the second image has an initial phase of <χ _s .

Going back to the previous derivations for Case i and Case 2, when

After further derivations, the two cases may be combined Io be;

Compared to Equation (3), we obtain: 10

Z( a ) also has a period o W is to be noted that the new term Z is S 5 independent of the term IT. In other words, Z on!y depends on the flicker fluctuation and is independent of the actual sϊgna! at a pixe! location. Therefore, the Z value shouid be identical for pixels in the same row. If the Z value of any pixel from each row, or the average Z value for each row is taken , a sequence

values ay be obtained, where n is the row number, β 0 is the time difference between the read-outs of two consecutive rows, and α ₀ is

the initial phase for the first row.

In one embodiment a may be estimated by finding the zero crossings of Z, Since

Z-O and going from negative to positive, when

Z=O and going from positive to negative, when

Therefore, by determining the zero crossing and the direction where Z is going, the value of α at that iocation may be estimated as follows. if 2 is zero and going from negative to positive, then;

where ^c is an integer, n is an integer number of pixel width, υ is a fraction of one

pixel width. Since Z is periodic, the effective value α ₀ is limited to

That constraint eliminates k from the equation so that σ,, can be estimated.

Similarly α _c may be estimated if Z is zero and going from positive to negative.

Since equation (5) has a maximum and minimum value of

In one embodiment the maximum values and minimum values in the Z sequence are found, and A is calculated from Equation (6). Of course, this is only an approximation because the Z sequence is a set of discreet samples of the

actuai Z curve. Another technique for determining A is to use estimated α _u value

and recalculate A from Equation (5), In one embodiment the values at different pixel locations are averaged to get a more accurate estimate. In one embodiment, the results from both of the methods for determining A are combined.

Row Averages

One way to calculate Z values is to calculate Equation (5) at every pixel.

To obtain a sequence of Z with respect to α , in one embodiment a Z value from each row is selected. In another embodiment an average of the Z values for each

pixel is determined and used as the Z value for that row. Since the subtraction of V values in Equation (5) is a linear operation, in one embodiment the pixel values in each row are averaged first before calculating the Z values. Using the row averages can save a large amount of memory, instead of storing Z values for the entire image, only a column of Z values is needed, it can also speed up the processing because Z values at each pixel location need not be calculated. In general using the row average reduces the noise and gives more accurate estimates.

Average of Z values over multiple periods

Since Z is a periodic function, the entire sequence of Z values need not be analyzed. Thus, in one embodiment, just one period of the Z sequence is calculated. In another embodiment better an average of all the periods is calculated to smooth out the noise. To average over all periods, in one embodiment a conversion is performed to convert the unit of the period in time expressed in seconds to number of rows using the equation:

where β is the shutter time delay between two consecutive rows, and m is the

length of the period for the Z sequence in terms of number of rows. For a roliing shutter, the exposure time is usually set in terms of number row delays. For an

exposure time set to be half period of the AC power, the following relationship holds:

where n is the number of rows being exposed. It will be observed that m is same as π, the number used to set exposure time.

Smoothness and Noise Removal

In practice, image data is usually noisy. Thus in one embodiment a smoothing or noise removal step may be performed. Typical smoothing methods include Gaussian filtering, rectangular window filtering, median filtering, etc. First, the row averages are filtered along the vertical direction. Then after Step 2b where an average over all the periods is determined, the Z averages are smoothed. Usually the longer the filter length, the smoother is the data, allowing

the initial phase a _Q to be estimated more accurately, In one embodiment, filters

of various lengths are used. Shorter ones are used to estimate A, while longer ones are used to estimate α ₀ .

Estimation of the zero crossing locations

Often, a zero crossing for the Z function may fall in between two pixels in which case, in one embodiment an interpolation operation is performed to obtain the pixel value of the Z function at the zero crossing. The interpolation operation may be a linear interpolation between the two closest pixels that span the zero crossing. For example if the zero crossing p lies in between point pi and P ₂ with the corresponding Z values Zi and Z2, p may be estimated by:

For each period, there are two zero crossing values, one going from negative to positive and the other from positive to negative. These points are half a period apart. Suppose the two points are labeled as Q ₁ and q ₂ , and ^ ₁ < Q ₂ - Thus, the zero crossing results may be verified by the by the following condition:

where thresh could be set to, for example 5%, for a high confidence in the result. The finai value of q-i may be further adjusted by the average of the two:

In one embodiment an adjustment for Q2 is performed which is similar to the adjustment for Q ₁

Adjusting the image for flicker fluctuation at block 24

In one embodiment, having determined the flicker magnitude and the initial phase in accordance with the above techniques, the fucker reduction logic

18 at block 24 adjusts for flicker artifacts on any images taken with an exposure time that is not an integer multiple of the half period of the AC power.

Theoretically, the estimated values can be applied to all subsequent frames, if the light condition is not changing and the interna! clock is accurate. However, over time error accumulates, and in one embodiment the magnitude and phase are re-estimated periodically or when lighting conditions change.

In one embodiment adjusting an image for fucker artifacts includes, calculating the fiicker varying term in Case 1 and Case 2 for each row and subtracting that term from each pixel in that row. The resulting image may be dimmer than the original image. Thus, in one embodiment a constant may be added to the pixels in each row so that the two images have the same average intensity.