The invention relates to a method for processing image data provided by the image sensor, the image data comprising an array of pixels.

The invention further relates to an image processor unit for processing image data provided by an image sensor, wherein the image data comprising an array of pixels.

The invention further relates to a computer program for carrying out the above- mentioned method.

Digital imagers are widely used in everyday products, such as smart phones, tablets, notebooks, cameras, cars and wearables. Many of the imaging systems in those products have the feature of automatic focusing (AF) in order to produce sharp images/videos. The traditional approach to automatic focusing is based on contrast detection. The lens is moved to the position where the scene contrast is the highest. Contrast automatic focusing is generally slow, and the technology of phase detection autofocus (PDAF) can then be coupled with it to improve the speed, and at times accuracy, of focusing.

PDAF sensors have the so-called phase detection (PD) pixels arranged across the whole sensor area. The phase information derived from the phase detection pixels can be used to determine the focal length, which in turn drives the lens to move to the position with the optimal focus, and that process is typically very fast if the phase information is accurate enough. The phase detection pixels are typically arranged periodically in the horizontal and vertical directions, across the entire sensor. After extracting the phase information from those pixels, they must be concealed or corrected. Otherwise, they could appear as a mesh of clustered defects across the image. It is, therefore, essential to have a phase detection pixel concealment (PPC) module in the image signal processor to maintain high image quality.

N. El-Yamany: “Robust Defect Pixel Detection and Correction for Bayer Imaging Systems,” in: IS&T International Symposium on Electronic Imaging 2017, p. 46-51 , discloses a method for identifying singlets and couplets of hot pixels, cold pixels or mixture of both types. Those pixels are not corrected early in the image processing pipeline, demosaicing and filtering operations will cause them to spread and appear as coloured clusters that are detrimental to image quality. The method operates on the raw data coming from the Bayer sensor. A defect pixel is identified if two conditions are met. In the first condition, it is checked whether the pixel is significantly different from its same-colour neighbours in the S x S Bayer window centred at that pixel. In the second condition, it is tested whether the local brightness difference in the pixel is significantly higher for hot pixels or lower for cold pixels than the smallest local brightness difference for each colour channel when the 3 x 3 Bayer window is centred at the pixel.

A detected defect pixel is replaced with the robust, detail-preserving estimate, which is determined by use of directional filters.

E. Chang: “Kernel-size selection for defect pixel identification and correction”, in: Proc. SPIE 6502, Digital Photography III, 65020J, 20-02-2007, describes a bounding m in-max filter with varying kernel sizes for defect pixel correction.

S. Wang, S. Yao, O. Faurie, and Z. Shi: “Adaptive defect correction and noise suppression module in the CIS image processing system”, in: Proc. SPIE Int. Symposium on Photoelectronic Detection and Imaging, vol. 7384, p. 73842V-1 -6, describes a spatial adaptive noise suppression algorithm which combines a defect pixel correction function for implementation in a CMOS image sensor chip. A centre weighted median filter is provided to correct defect pixels. Random noises are treated separately according to the detail level of their background.

A. Tanbakuchi, A. van der Sijde, B. Dillen, A. Theuwissen and W. de Haan: ..Adaptive pixel defect correction", in: Proc. SPIE Sensors and Camera Systems for Scientific, Industrial and Digital Photography Applications IV, vol. 5017, pp. 360-370, 2003, presents a defect correction algorithm utilizing raw Bayer image data. If a pixel in an image is found to be defective, the neighbouring pixels provide the best information for interpolating the defective pixel. Directional derivates are used to correlate the nearest points to the defect colour plane.

M. Schoberi, J. Seiler, B. Kasper, S. Foessel and A. Kaup: „Sparsity-based detect pixel compensation for arbitrary camera raw images”, in: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1257-1260, 2011 , discloses an algorithm for detecting defect pixels and interpolating the missing values for the defect pixels. Previously interpolated pixels are re-used.

An object of the present invention is to provide an improved method and image processor unit which provides a robust scheme for detecting defect pixels.

The object is achieved by the method comprising the features of claim 1 , the image processor unit comprising the features of claim 12 and the computer program according to claim 13. Preferred embodiments are disclosed in the dependent claims.

It is proposed to carry out the steps of: a) Determining the respective local brightness difference between a selected pixel an a set of pixels located adjacent to the respective pixel and assigned to the same colour of the respective pixel; and b) Identifying the selected pixel as a defect pixel when the local brightness difference for the selected pixel exceeds an upper threshold value and/or is less than the lower threshold value, and when the local brightness difference for the selected pixel exceeds a weighted maximum local brightness difference determined for the set of pixels located adjacent to the selected pixels and/or is less than a weighted minimum local brightness difference determined for the set of pixels located adjacent to the respective pixel.

For identifying a selected pixel as a defect pixel, two detection conditions are combined. The conditions are, in principle, independent form the colour filter array used by the image sensor for providing the pixel array. Different types of defects can be identified by a combination of the conditions, which can be easily tuned by a given threshold value and a weighting factor.

By comparing the local brightness difference for the selected pixel with the given threshold, it is checked whether the selected pixel is significantly different from its same-coloured neighbours in a small window, which includes the selected pixel and the set of surrounding pixels. The selected pixel can be centred in the small window, when the method is carried out in the pixel array having the pattern of a respective colour filter array, i.e. before the step of demosaicing.

With the second condition, the local brightness for the selected pixel is compared against the local brightness differences determined for the neighbouring pixels located adjacent to the selected pixel. Preferably, it is checked whether the local brightness difference in the selected pixel is significantly different from the local brightness differences of the same-colour neighbours in the small window centred at the selected pixel. The maximum local brightness difference and minimum local brightness difference is not necessarily the absolute maximum or minimum value of a set of local brightness differences. This is understood as one selected maximum I minimum value, e.g. the first (absolute) or preferably the second or even third, fourth, ... maximum I minimum value of a set, which corresponds to excluding the absolute maximum I minimum values as outliners from the set of values expressing the local brightness difference. Thus, also a local maximum I minimum value or an adjusted maximum I minimum value is included in the meaning of maximum I minimum local brightness difference according to the present invention.

The second condition allows to differentiate between a real defective pixel and a small image detail.

The combination of the two conditions provides a robust scheme for identifying the selected pixels as defect, which can be easily carried out on an image data stream comprising pixel data of arrays of pixels with low computational effort and reliable results. The method can be used for various colour filter arrays, such as standard Bayer, Quad Bayer, RGBW, Nonacell and Hexadeca Bayer colour filter array patterns. It is possible to easily identify singlets or couplets of hot defects, cold defects or a mixture of both types of defects. The method is also based to identify higher-order defects.

For determining the respective local brightness difference for a selected pixel in step a), an average value for the set of pixels located adjacent to the respective pixel can be calculated. The respective local brightness difference for the selected pixel can be then calculated as a difference between the value of the selected pixel and the calculated average value of the set of adjacent pixels.

The adjacent pixels can be located directly adjacent to the selected pixel. It is also possible to consider adjacent pixels which are located close to the selected pixels with pixels in between the selected pixel and the adjacent pixel in a pixel array.

The set of adjacent pixels considered for determining the respective local brightness difference for a selected pixel can be selected from only those adjacent pixels which are assigned to the same colour than the selected pixel. The average value of the adjacent pixels of the set of pixels is calculated to determine the local brightness difference as the difference between the value of the selected pixel and the calculated average value for the set of adjacent pixels.

Thus, each selected pixel can be assigned to a respective local brightness difference value.

The average value can be calculated as mean value, alpha-trimmed mean value or medium value for a set of pixels surrounding the selected pixel and being assigned to the same colour of the selected pixel.

For determining the alpha-trimmed mean value, the pixel values are sorted by size on the top and bottom a portion of the number of pixel values that are excluded from the mean. For example, a 2-a-trimmed mean filter deletes two pixel values for calculating the mean value, namely the lowest and highest values (i.e. brightness levels for the respective colour). Other methods for calculating the average value are also applicable, like midpoint filter or the like.

The set of pixels considered in step a) of determining the local brightness difference for a selected pixel can be located in a window of pre-determined size, preferably a 4 x 4, 5 x 5, 6 x 6 or 8 x 8 window, wherein the set of pixels is assigned to the same colour of the selected pixel.

The selected pixel can be located in the centre of this window.

By use of a window including the selected pixel, wherein the frame of the window surrounds the selected pixel, pixels of the same colour within the window are selected for the set of adjacent pixels.

The first condition can be checked to identify hot or cold pixels by determining the maximum local brightness difference of a set of pixels located adjacent to the selected in a window of a pre-determined size and identifying the selected pixel as defective hot pixel when the local brightness difference exceeds the weighted maximum local brightness difference, or determining the minimum local brightness difference of a set of pixels located adjacent to the selected pixel in the window of a pre-determined size and identifying the selected pixel as a defective cold pixel when the local brightness difference is less than the weighted minimum local brightness difference.

An improved detection result for identifying defect pixels can be achieved by considering the second maximum local difference or the second minimum local difference instead of the maximum or minimum values of the local difference. It appears that by considering not the maximum local difference value, but the lower second maximum local difference value, or for the minimum local difference value not the minimum, but the second minimum value, outliers of the local difference values are excluded. This can be compared to considering the median value for an average value instead of the mean value.

More preferably, the lower order maximum value of the local difference, e.g. the second maximum local brightness difference, is determined from the set of the positive values of the local brightness difference. Similar, the lower order minimum value of the local difference, e.g. the second minimum local brightness difference, is determined from the set of the negative values of the local brightness difference.

Another option is to determine the maximum value from the absolute values of the local brightness differences of the set of pixels located adjacent to the selected pixel. Then, the consideration of the minimum value for the second condition is not of interest.

The maximum local difference or the minimum local difference in the meaning of the present invention also include second, third, fourth and so on maximum or minimum local difference values when excluding outlier values. The upper threshold value and/or lower threshold value considered for identifying the selected pixel as a defective pixel can be pre-determined and stored in a memory for applying to the method. Alternatively, the upper and lower threshold values can be adapted as required to improve the final image quality. Preferably, the upper and/or lower threshold value can be adapted by use of a neural network.

The same applies for the weighting factors considered for the minimum and maximum local brightness difference, which can also be pre-set as a fixed value or adapted as a variable to improve the image quality. Preferably, the weighting factors considered for the minimum and maximum local brightness difference are adaptively learned and improved by use of a neural network to provide a high final image quality. The weighting factors can be set as one value for the maximum local brightness difference and the minimum local brightness difference. Optionally, each of the maximum local brightness difference and minimum of the local brightness difference is assigned to a respective weighting factor.

The step a) of determining the respective local brightness difference for a selected pixel and step b) of identifying the selected pixel as a defect pixel can be carried out repeatedly by selecting each pixel of the pixel array provided by the image camera for an image and repeating the method for each selected pixel. Alternatively, not each pixel of the pixel array can be selected when repeating the method but a variety of pixels of the pixel array, for example, only pixels of a specific selected colour within the pixel array.

After identifying a pixel as defect, the image data comprising the pixel arrays can be further processed by correcting the pixels, which are identified as defect.

The image processor unit is arranged for processing image data by carrying out the steps of the above-described method. The image processor unit can be a hardware unit, like an FPGA or suitably programed microprocessor or data signal processor.

The method can be easily carried out by use of computer program comprising instructions which, when the program is executed by a processing unit, cause the processing unit to carry out the steps of the afore-mentioned method.

The invention is explained by an example in the enclosed drawings. It shows:

Figure 1 - block diagram of the method for processing image data;

Figure 2 - exemplary pixel array for a Bayer colour filter array pattern with a selected pixel and a window of adjacent pixels;

Figure 3 - block diagram of an exemplary method for correcting pixels identified as defect.

Figure 1 shows a block diagram of the method and the functional units of the image processor unit for processing image data provided by an image sensor, wherein the image data comprising an array of pixel PAIN, which is used as input for the further steps.

The step a) determines the respective local brightness difference between a selected pixel of the pixel array PAIN and a set of pixels located adjacent to the respective pixel and assigned to the same colour of the respective pixel.

The local brightness difference LBD (x, y) for the position x, y of the selected pixel in the pixel array can be determined as the difference between the value of the selected pixel P (x, y) and the robust estimate p_est (x, y) of the selected pixel P (x, y) as follows:

LBD (x, y) = P (x, y) - p_est (x, y).

The robust estimate p_est (x, y) of the selected pixel P (x, y) can be determined by use of the pixel values of a set of pixels, which are located adjacent to the selected pixel P (x, y). For example, the mean value or the median value of all pixels of the same colour as the selected pixel, which are located within a pre-determined window, said window includes the selected pixel P (x, y), can be calculated for the robust estimate p_est (x, y). Preferably, the selected pixel P (x, y) is located in the centre of the window.

In order to detect or identify couplets of defects, the robust estimate p_est (x, y) should be robust to the presence of couplets on the raw window centred at the selected pixel P (x, y). One possibility is to calculate p_est (x, y) as the a-trimmed mean (with a = 2) of the same-colour neighbours in the raw window.

This can be better understood by an exemplary array shown in Figure 1 for a Bayer colour filter array pattern. For example, the selected pixel P (x, y) is G (2, 3). The small window 3 x 3 around this selected pixel G (2, 3) is used to select pixels of the same value for the set of adjacent pixels. In this exemplary case, the pixels G (1 , 2), G (3, 2), G (1 , 4) and G (3, 4) are selected for the set of adjacent pixels.

The average value of this set of adjacent pixels G (1 , 2), G (3, 2), G (1 , 4) and G (3, 4) is calculated and the resulting average value is used as the robust estimate p_est (x, y).

For the 2-a-trimmed mean, the highest and lowest value of the set of e.g. four pixels is disregarded and the mean value is calculated from the (with e.g. four pixels remaining two) remaining intermediate pixel values of the set of (e.g. four) adjacent pixels. Similarly, a window of a specific size is centred around the selected blue B (x, y) pixel for a selected red R (x, y) pixel. The size of the window used for the selected pixel with a respective colour can be chosen varying in dependence of the respective colour. The size of the window can be chosen for example such that it includes at least three adjacent pixels of the same colour to be assigned for the set of adjacent pixels for the selected pixel of a specific colour. The size of the window can be selected in dependence on the respective different colour filter array pattern and/or the location of the specific selected pixel P (x, y). The aim is to determine the robust estimate of the selected pixel value P (x, y) by use of a sufficient number of adjacent pixels of the same colour.

One option to safeguard a sufficient number of adjacent pixels, a local demosaicing process can be additional carried out, so that the small window around the selected pixel P (x, y) can be chosen with a sufficiently high number of pixel values of the same colour as the colour of the selected pixel.

In step b) of Figure 1 , the selected pixel Py(x, y) is identified as a defect pixel when the local brightness difference LBD (x, y) for the selected pixel P (x, y) exceeds an upper threshold value S_hot and/or is lower than the lower threshold value S_cold. This can be expressed by the following equation:

For hot pixels: LBD (x, y) > S_hot

For cold pixels: LBD (x, y) < S_cold.

Alternatively, one common threshold value S can be set for the upper and lower threshold value, when comparing the absolute value of the local brightness difference with this common upper/lower threshold value S. This can be expressed by the following equation:

|LBD (x, y)| > S.

The condition that the local brightness difference is greater or lower than the threshold value S or upper/lower threshold S_hot and S_cold includes the condition that the local brightness difference LBD (x, y) is also equal to the threshold value S, S_hot, S_cold.

By use of a weighted value and the common threshold value, it is checked within one step whether the local brightness difference LBD (x, y) exceeds an upper threshold value S and is lower than the lower threshold value S.

Further, in step b) the selected pixel P (x, y) is identified as a defect pixel in case that also the second condition applies, namely that the local brightness difference LBD (x, y) at the selected pixel P (x, y) is significantly different from the local brightness differences LBD (i, j) of the adjacent pixels. This allows to differentiate between real defective pixels and small image details.

The local brightness difference is therefore calculated not only for the selected pixel P (x, y) but also for the pixels P (i, j), which are located adjacent, i.e. as direct or indirect neighbours of the selected pixel P (x, y). As for calculating the robust estimate p_est (x, y), the pixels of the set of adjacent pixels and their respective local brightness difference LBD (x, y) are considered to determine a weighted maximum local brightness local difference LBD (i, j) and the weighted minimum local brightness difference LBD (i, j). This can be expressed by the following formula:

For hot pixels: LBD (x, y) > F_hot x max (LBD (i, j) V (i, j) e 0)

For cold pixels: LBD (x, y) < F_hot x min (LBD (i, j) V (i, j) e Q).

0 is the collection of the same-colour pixels in the pixel array within the window centred at the selected pixel P (x, y). LBD (x, y) is the local brightness difference determined for the selected pixel P (x, y). F_hot and F_cold are weighting factors which would have the same or different values. The positions i, j are related to the specific pixel positions x, y in the pixel array window related to adjacent pixel of the same colour as the selected pixel P (x, y).

In order to be robust to couplets, 0 should include a robustness strategy, for example by excluding same-colour pixels with the highest/smallest LBD values in the window. This can be achieved by consideration of the second maximum and the second minimum value of the set of local brightness differences LBD (i, j) for the set of adjacent pixels. This can be expressed by the equations:

For hot pixels: LBD (x, y) > F_hot x 2nd max (LBD_positive (i, j) V (i, j) e 0)

For cold pixels: LBD (x, y) < F_hot x 2nd min (LBD_negative V (i, j) (i, j) e Q).

LBD_positive (i, j) denotes the positive-valued local brightness differences in the set 0 at the respective pixel position i, j in the pixel matrix. LBD_negative (i, j) denotes the negative-valued local brightness differences in the set 0 at the respective pixel position i, j in the pixel matrix.

The upper and lower threshold value S, S_hot and S_cold and the weighting factors F_hot and F_cold can be physically stored in a memory or varied. They are input into step c), which includes a neural network for adapting the threshold values S and weighting factors F.

Using a small weighting factor F, F_hot and F_cold would result in more false positives. Using a big value for the weighting factor F, F_hot and F_cold would result in less fewer positives.

Preferably, S can be selected from the group of [0,2 ^A NBits-1 ] and the weighting factor F > 1 , where NBits is the bit depth of the raw data.

In step c), the threshold values, S, S_hot, S_cold and weighting factors F, F_hot, F_cold can be tuned together to achieve the best trade between defect detection and detail preservation, i.e. image quality. The adaption can be performed by use of a learning-based framework, i.e. neural network achieving an improved defect detection rate close to 100 % and reduced false positive rate, preferably close to 0 %.

The neural network can be learned by the goal to achieve the optimized image quality. The result of step b) is an identification of defect pixels D (x, y) that can be provided by a simple list of x, y positions identifying the pixel in this location as defect. Additional information on the kind of defect (hot pixel, cold pixel) can be included optionally.

In step d), a step of defect pixel correction can be carried out based on the incoming pixel array PAIN and the pixels or pixel locations D (x, y) identified as defect.

Defect pixel correction is known in the prior art. A suitable method for defect pixel correction can be selected to achieve a corrected pixel array at the output PAOUT.

Preferably, defect pixel correction seeks to preserve the image details and hence, when possible, relies on using directional correction and limiting the information used in the correction to a small neighbourhood.

Figure 3 shows an exemplary preferred embodiment for defect pixel correction using step e) of closest-neighbours correction to correct the pixel via interpolation of nearest same-colour neighbours.

In a separate path, the steps f), g), h), and i) are performed for executing a directional correction h) and a non-directional correction i), which are both based on a first step f) of non-directional pre-correction. Before carrying step h) of directional correction, a directional filtering is carried out in step g).

The defect correction mechanism decides which of the three operations is utilized for the correction depending on the colour filter array pattern and the colour channel arrangement in the colour filter array data. Specifically, when a given colour channel in the colour filter array pattern exhibits symmetry, thus facilitating directional estimation and correction, the default correction scheme can follow the three-step procedure with the steps f) of non-directional pre-correction, g) of directional filtering and h) of directional correction, or alternatively, i) of non-directional correction.

In step f) “non-directional pre-correction”, a robust estimate is calculated for all the identified defect pixels in the raw image data centred at the pixel being corrected, and all non-defect pixels are left intact. The purpose of this pre-correction (preprocessing) in step f) is to provide an intermediate clean estimate of the raw image data around the pixel being corrected, thus facilitating robust direction identification in that pixel position. The robust estimate p_est (x, y), which is defined as stated above, can be used in this step.

The step g) of directional filtering serves to identify the feature direction in the pixel position being corrected. Therefore, directional filtering is performed in the pixel position by convolving the pixel along with its same-colour neighbours with a set of directional filters. Based on the maximum absolute response of the filter set, the feature direction is identified.

The number and size and coefficients depends on several factors:

1 . The number of directions planned to be identified and supported in the correction scheme;

2. The size of the raw support centred at the pixel;

3. The colour filter array channel arrangement;

4. The type of the filters, e.g. gradient-based or otherwise.

Once a feature direction in the pixel being corrected is identified, the pixel can be corrected by interpolating clean, same-colour pixels along the identified direction in the step h) of directional correction. In order to reduce or eliminate the possibility of correction artefacts, directional correction is typically limited to a small neighbourhood of the pixel.

If no feature direction is identified, e.g. in various smooth regions, then a non- directional correction can be pursued by replacing the defect pixel with a robust estimate of it, e.g. by p_est (x, y). The step is identified as step i) in Figure 3.

When a given colour channel in the filter array pattern does not exhibit symmetry, such as R, G, or B channels in the RGBW colour filter array pattern or R and G and G channels in the Quad Bayer colour filter array pattern, then the defect correction scheme can rely solely on the step e) of closes-neighbours correction. In the step e), a robust estimate is used to correct the defective pixel. One possibility for such an estimate is, for example, the a-trimmed mean of the same-colour pixels in the neighbourhood of the pixel being corrected. Another possibility is some weighted average of those same-colour neighbours.

Of course, directional or content-aware correction would give a better image quality than non-directional correction. It is, therefore, preferred that when interpolating closest neighbours, the interpolation strategy attempts as much as possible to identify directions or takes into account the content in the raw image data in the pixel P (x, y) being corrected.

The step d) in Figure 1 of correcting pixels identified as defect can be used with various colour filter array patterns, such as Standard Bayer, Quad Bayer, RGBW and HexaDeca Bayer colour filter array patterns. The definition of the robust precorrection, directional filters, directional/non-directional correction (interpolation) and closest-neighbours correction may take different flavours. They can be changed from fixed global definitions to local ones, depending on the afforded complexity and the desired image quality.

The method for processing image data according to the present invention can correct singlets and/or couplets of hot defects, cold defects or a mixture of both types. With a proper adjustment, e.g. the size of the raw image data, it can handle higher-order defects.