Controlling an amount of filtering

The invention relates to a control method for controlling an amount of filtering, a method of filtering, a method of displaying comprising the method of filtering, a controller for controlling an amount of filtering, a system comprising the filter and the controller, and a display apparatus comprising the system.

Most existing coding artifact reduction methods apply low-pass filters on block boundaries to remove blockiness. However, grid positions are not always available, especially when videos are rescaled after decoding.

It is an object of the invention to provide a controller for a filter which does not require grid positions.

A first aspect of the invention provides a control method as claimed in claim 1. A second aspect of the invention provides a method of filtering as claimed in claim 13. A third aspect provides a method of displaying as claimed in claim 14. A fifth aspect provides a controller as claimed in claim 15. A sixth aspect provides a system as claimed in claim 16. A seventh aspect of the invention provides a display apparatus as claimed in claim 17. Advantageous embodiments are defined in the dependent claims. A control method in accordance with the first aspect of the invention controls an amount of filtering of a particular pixel of an image dependent on information content of a local region of the particular pixel.

The filtered value of the particular pixel is obtained by filtering pixel values of pixels in the local region. Usually, the filter has a footprint or kernel which covers a particular number of pixels. The values of the pixels covered by the footprint are weighted to obtain the value for the particular pixel. Usually, also the original value of the particular pixel is incorporated in the footprint. The filter may be a low-pass filter applied to remove artifacts from an image. Alternatively, the filter may be a different kind of filter to provide image enhancement, such as for example sharpness enhancement.

The information content is determined by using a probability density function which defines a probability distribution of descriptor values in the local region. The descriptors are, for example, the luminance intensity of the pixels, color values of the pixels, orientation of edges, or phase information within the local region. For simplicity, unless stated otherwise, in the following, the descriptor is thought to be the pixel luminance intensity.

The local region may, but must not, be identical to the footprint of the filter. Roughly speaking, this probability distribution indicates per range of values how many pixels in the local range have a value within this range. Such a range is often referred to as a bin, and may for example for pixels which have digital values between 0 and 255 run from 0 to 15, from 16 to 31, and so on.

The amount of filtering is controlled in response to the probability density function. For example, if the filter is a low-pass filter the amount of low-pass filtering is controlled to obtain a lower amount of low-pass filtering in a local region having a more spread out probability density function than in a local region having a less spread out and more peaked probability density function. For example, if the probability density function has a single peak, all the pixel values in the local region are identical, and a high amount of low- pass filtering is possible. On the other hand, if the probability density function has multiple peaks or is spread out, a lot of different pixel values occur in the local region and thus a lot of information is present. Consequently, the amount of low-pass filtering should be relatively low.

The use of the probability density function does not require the grid position or any other coding parameters. It is thus a more general approach which can process material of which the grid position is not known. Further, obviously no processing time is required for detecting the grid position.

Such a controlled low-pass filtering is, for example, interesting for information which was coded and after decoding has coding artifacts such as (MPEG) block structures, ringing, and mosquito noise. Conventionally, different coding artifacts are removed separately, which has the drawback that the reduction of one artifact may make another artifact more visible. In fact all coding artifacts are treated as noise, and the probability density function should differentiate noise from content. Noise in detailed regions of the content is less noticeable than in flat regions. The first reason is that a smart encoder usually allocates more bits for detailed regions than for flat regions which results in less quantization noise in detailed regions. The second reason is the masking effect of the human eye which

makes noise less visible in detailed regions. Thus, the amount of low-pass filtering is controlled to be higher in flat regions than in detailed regions. The probability density function processes the information content in the local regions to determine how much detail is present. WO03/01076 discloses how to detect edges in a block of a digital image. The block may be formed by 8 by 8 pixels. The method creates a histogram from absolute differences in pixel luminance of the pixels in the block. The entropy of this histogram is computed and compared to an entropy threshold. The entropy can be computed very efficiently using only a look-up table. The edge detection is performed by using the comparison of the entropy with the entropy threshold and by comparing the maximum absolute difference of pixel luminance in the block with a threshold.

In an embodiment as claimed in claim 2, the probability density function is a histogram of the local descriptor. Usually, it is easier to determine a histogram of the local descriptor in the local region than the probability density function in its integral form. In an embodiment as claimed in claim 3, the local descriptors are the intensity values of the pixels. The intensity values may be Y values of the signal defining the image, or may be obtained from the RGB values of the signal. Alternatively, it is possible to use one of the UV values accompanying the Y value or one of the RGB values.

In an embodiment as claimed in claim 4, the amount of filtering is varied by changing the kernel size of the filter in response to the information content. For example, for a low-pass filter, the amount of low-pass filtering is decreased by decreasing the size of the filter kernel if the information content indicates that a lower amount of low-pass filtering is desired because the probability density function is more spread-out. In fact, the filtered value of the particular pixel is determined for fewer neighboring pixels. Consequently, the low-pass filtering is performed more locally around the particular pixel and thus is less pronounced. In an embodiment as claimed in claim 5, the amount of filtering is varied by changing the weighting factors of the filter. The weighting factors are the factors allocated to the pixels in the footprint which should be used to calculate the filtered value of the particular pixel. For example, for a low-pass filter the amount of low-pass filtering is decreased by increasing the weighting factor for the pixel to be filtered and consequently decreasing the weighting factors of the other pixels contributing to the low-pass filtering.

In an embodiment as claimed in claim 6, the local entropy is calculated in the local region from the probability density function. The value of the local entropy is used to control the amount of filtering. For example, for a low-pass filter, if the local entropy has a

high value, the amount of low-pass filtering should be low. If the local entropy has a low value, the amount of low-pass filtering should be high.

In an embodiment as claimed in claim 8, the entropy is calculated for intensity values of the pixels. This calculation is relatively easy. In an embodiment as claimed in claim 9, the size of the kernel of the low-pass filter is inversely proportional to the value of the entropy.

In an embodiment as claimed in claim 10, the values of the weight factors depend on the value of the entropy. The higher the entropy is the less low-pass filtering is required thus the more the original value of the particular pixel should prevail. In an embodiment as claimed in claim 11, an amount of random noise is added dependent on the probability density function. More noise is added to flat local regions R where the artifact resides than to the detailed local regions R.

In an embodiment as claimed in claim 12, a randomizer randomizes the pixel locations in the local region, wherein the range of the randomizer depends on the probability density function. For example, the range of the randomizer is larger for regions with lower entropy, and the other way around.

These and other aspects of the invention are apparent from and will be elucidated with reference to the embodiments described hereinafter.

In the drawings:

Fig. 1 shows the pixel values of the pixels in a local region, Fig. 2 shows a footprint of the low-pass filter,

Fig. 3 shows an example of a histogram of pixel values in the local region, Fig. 4 shows another example of a histogram of pixel values in the local region,

Fig. 5 shows a block diagram of a display apparatus with a low-pass filter controlled by a filter controller for controlling the amount of low-pass filtering,

Fig. 6 shows an example of the relationship between the kernel size and the local entropy,

Fig. 7 shows an example of the weight factors of the kernel of the low-pass filter, and

Fig. 8 shows an example of the relationship between the weight factor for the to be filtered pixel and the local entropy.

It should be noted that items which have the same reference numbers in different Figures, have the same structural features and the same functions, or are the same signals. Where the function and/or structure of such an item has been explained, there is no necessity for repeated explanation thereof in the detailed description.

Fig. 1 shows the pixel values PVij of the pixels Pij in a local region R. In the example shown, the local region R is a square of 5 rows of each 5 pixels. However, the local region R need not have a square shape and may comprise any other amount of pixels Pij. The pixel values PVij are, for example, the pixel intensity values or color values.

The information content IC of the local region R is determined by using a probability density function PDF of the pixel values PVij of the pixels Pij. Such a probability density function PDF as such is well known, and in its most general form is an integral:

]f(x)dx = l

wherein the function f(x) is any function that describes the probability density in terms of the input variable x, the function f(x) is greater than or equal to zero for all values of x, and the total area under the graph formed by the function f(x) is one. The probability within a particular interval (also referred to as bin) of the input variable x is found by calculating the value of the integral over this particular interval. The bins may cover a range of possible pixel values PVij or a single pixel value Pvij. For example, for 8 bit pixel values, the first bin may cover the pixel values 0 to 15, and the last bin covers the values 239 to 255. Simply speaking, the probability density function PDF indicates how the pixel values PVij are spread over the range of possible values.

The information content IC may be the probability density function PDF in its integral form. In another embodiment, the probability density function PDF is approximated by a histogram HIS of, for example, the intensity values of the pixels Pij in the local region R. In another embodiment, the probability density function PDF is approximated by the local entropy H of the local region R. The local entropy H may be calculated from the histogram HIS. It is this information content IC which is used to control the amount of low-pass filtering of the low-pass filter LPF.

Fig. 2 shows a footprint KE of the low-pass filter LPF. The footprint is also referred to as the kernel KE. The kernel size KZ of the kernel K determines how many pixels Pij are covered by the kernel K. The low-pass filtering is obtained by calculating the

weighted sum of all or part of the samples in the kernel K. In the example shown the kernel K has a square shape and covers 5 rows of 5 pixels Pij each. The weight factors Wij are associated with the pixels Pij. The kernel K may have another shape than square and may cover a different amount of pixels Pij than the local region R. It is possible that only the pixels Pij within the kernel which fulfill a particular criterion are contributing to the weighted sum. For example, only pixels Pij which have an intensity within a predefined range are included for filtering.

The information content IC may control the kernel size KZ and/or the weight factors Wij to vary the amount of low-pass filtering of the low-pass filter LPF. If the information content IC indicates that the local region R comprises much detail, the probability density function PDF will be spread-out because many different pixel values PVij occur. With a spread-out probability density function PDF is meant that the probability has a non-zero value in many bins. However, because the total over all the bins should be one, these values are relatively low. If the information content IC indicates that the local region R comprises a low amount of detail, the probability density function PDF will have a few or even only one peaked value because all pixel values PVij are near to each other or are even identical. With a peaked probability density function PDF is meant that only one bin or a few bins have a non-zero probability. Again, due to the fact that the total over all the bins must be one, now, these non-zero probabilities are relatively large. As will be elucidated in the now following with a few examples, this probability density function PDF can be used to control the amount of low-pass filtering of the low-pass filter LPF such that more low-pass filtering is applied in local regions R which have a peaked probability density function PDF than in local regions R which have a spread out probability density function PDF.

Fig. 3 shows an example of a histogram HIS of pixel values PVij in the local region R. The histogram HIS is an approximation for the probability density function PDF. Consequently, the sum of all numbers N in the histogram should be one. In the example shown in Fig. 3, all the pixels in the local region R have a pixel value PVij of 64. This peaked histogram HIS or probability density function PDF indicates that the local region contains a low amount of information and thus a high amount of low-pass filtering is provided. Fig. 4 shows another example of a histogram HIS of pixel values PVij in the local region R. The pixels values PVij are represented by 8 bit digital words. The range of possible values is divided in bins which, for example, comprise 16 values of the range of possible values. In this example, half of the pixel values PVij are within the bin ranging from 15 to 31 and the other half of the pixel values PVij are within the bin ranging from 79 to 95.

Compared to the example shown in Fig. 3, the local region R now comprises more different pixel values PVij and thus comprises more information. Thus, the less peaked and more spread out histogram HIS indicates that the local region R contains a higher amount of information and thus a lower amount of low-pass filtering is provided. Fig. 5 shows a block diagram of a display apparatus with a low-pass filter LPF controlled by a filter controller FC for controlling the amount of low-pass filtering of the low-pass filter LPF. The information content determiner CD receives the input image II and determines for each local region R the information content IC based on the probability density function PDF. The filter controller FC receives the information content IC to control the amount of low-pass filtering of the low-pass filter LPF in response to the probability density function PDF. The filter controller FC may control the kernel size KZ and/or the weight factors Wij of the low-pass filter LPF. The low-pass filter LPF receives the input image II, and from the filter controller FC the kernel size KZ and/or weight factors Wij and supplies the filtered input image II as the output image OI to the display DP. The filter controller FC controls the kernel size KZ and/or weight factors Wij such that an amount of the low-pass filtering is higher if the probability density function PDF is more peaked or less spread out.

Fig. 6 shows an example of the relationship between the kernel size KZ and the local entropy H for the local region R. By way of example, the local entropy H may be the Shannon entropy of the probability density function PDF of the pixel value PVij distribution inside the local region R. The probability density function PDF is approximated by the histogram of pixel intensities. The distribution of the pixel values PVij in the histogram HIS is defined by the local structure of the image in the local region R. Noise and (coding) artifacts do not affect the overall distribution of the histogram HIS. The local entropy H of a local region R can be defined as

H _D,R = ^~ ∑P _D,R W )log ₂ P _n ^d ₁ )

wherein P _{D R} (d _t ) is the probability of descriptor D taking the value d _t in the local region R.

The descriptor D is in fact the type of pixel value PVij, such as, for example, the intensity value or the color value. Alternatively, the descriptor D may be a more general description of a structure in the local region. For example, the descriptor D may be an angle of edges detected in the local region R.

If the descriptor D is the luminance intensity of the pixels Pij, the local entropy H can be defined as

H _R = -∑P _R (ι)log ₂ P _R (i) , ι=l wherein i indicates the bin index of the histogram, N is the total number of bins used, and R is the local region inside which the local entropy H is calculated.

The local entropy H has a higher value for a spread histogram HIS than for a peaked histogram HIS. For example, The local entropy H for the very peaked histogram HIS shown in Fig. 3 is zero, and for the less peaked histogram HIS shown in Fig. 4 is one. If four different pixel values PVij occur the same number of times in the local region R, the histogram HIS has four peaks of height ¹ A and the local entropy H is 2. Thus, the local entropy H of a detailed local region R tends to be larger than of a smooth local region R. Thus, the local entropy H can be used as an indicator of the amount of filtering which should be applied to the local region R, either for image enhancement or coding artifact reduction. If the local region R comprises N * N neighboring pixels Pij, the local entropy H of each pixel Pij is calculated from the pixel values PVij of these N * N neighboring pixels Pij.

The aim of a good coding artifact reduction algorithm is to preserve the sharpness of details and the remove most of the artifacts. Therefore, the kernel size KZ and/or the weighting factors Wij of the low-pass filter LPF should be dependent on the local structure of the image in the local region R. The more detail the local region R has the less smoothing should be applied.

Fig. 6 shows an example of how the kernel size KZ of the low-pass filter LPF applied on the local region R may depend on the local entropy H. It is assumed that the local entropy H has a minimum value Hmin and a maximum value Hmax. The solid line indicates a linear dependence between the kernel size KZ and the local entropy H. The filter kernel KZ has a maximum value Smax at the minimum value Hmin of the local entropy H, and has a minimum value Smin at the maximum value Hmax of the local entropy H. Alternatively, the dependency of the kernel size KZ on the local entropy may be not linear as indicated by the dashed lines. What counts is that the kernel size KZ decreases for increasing local entropy H. Or said differently, the kernel size KZ should be small when the local entropy H is large, and the other way around. The kernel size KZ needs not vary over the whole range of the local entropy H, and also must not vary gradually. Fig. 7 shows an example of the weight factors of the kernel of the low-pass filter. In this particular example only two values Wl and W2 for the weight factors Wij are allocated. The value Wl is allocated to the central pixel Pc in the kernel for which the low- pass filtered value has to be determined. The value W2 is allocated to the other pixels Pij in

the footprint of the low-pass filter, as far as these pixels contribute to the to be determined low-pass filtered value. In the example shown, only the pixels Pij are included in the filtering of which the pixel values PVij are within a predefined range of the intensity. These pixels Pij contribute with the weight factor W2. The other pixels Pij, except the central pixel do not contribute or have a weight factor zero. For example, only the pixels Pij which have a pixel value PVij which differs less than a particular threshold from the value of the central pixel Pc contribute. The threshold may depend on the local entropy H, a smaller threshold is used for detailed local regions R and a larger threshold is used for flat local regions R. This algorithm removes very strong blocking artifacts in flat local regions R. Alternatively, although not shown, the low-pass filter LPF may use all the other pixel values PVij of the pixels Pij other than the central pixel Pc. In an embodiment, the filtering is normalized in that the sum of all weight factors Wij in the footprint is one.

Fig. 8 shows an example of the relationship between the weight factor for the to be filtered pixel and the local entropy. The change of the weight factors Wij of the Io w- pass filter LPF based on the local entropy H will be elucidated with respect to the same example described with respect to Fig. 7. The kernel size KZ is 2, n+1 pixels Pij are within the kernel after thresholding, and Wl + nW2 = 1. Thus if the weight factor Wl is changed, the weight factor W2 has to change accordingly to meet this equation. Because a stronger low-pass filtering is desired in flat regions than in detailed regions, the value of the weight factor Wl should be small for a small value of the local entropy H and the value of the weight factor Wl should be large for a large value of the local entropy H.

Again, it is assumed that the local entropy H has a minimum value Hmin and a maximum value Hmax. The solid line indicates a linear dependence between the weight factor Wl and the local entropy H. The weight factor Wl has a minimum value Wlmin at the minimum value Hmin of the local entropy H, and has a maximum value Wlmax at the maximum value Hmax of the local entropy H. Alternatively, the dependency of the weight factor Wl on the local entropy H is not linear as indicated by the dashed lines. What counts is that the weight factor Wl increases for increasing local entropy H. The weight factor Wl need not vary over the whole range of the local entropy H, and also must not vary gradually. It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In an embodiment, the kernel size KZ and/or the weight factors Wij change in a smooth manner to obtain a smooth transition in the processing from detailed local regions R to flat local regions R.

Algorithms with discrete modes may cause problems due to abrupt changes in the processing by slight differences in detail between the local regions R.

In flat local regions R wherein the intensity is slowly changing, a contour like or ripple like artifact may be left even after strong low-pass filtering. These artifacts are very annoying, particularly in High Definition material. A first solution to this problem is to add some random noise to the low-pass filtered signal OI to mask the residual artifact. The amount of noise added may be dependent on the local entropy H to ensure that more noise is added to flat local regions R where the artifact reside than to the detailed local regions R. Alternatively, the noise may only be added to flat local regions R. A second solution to the problem is to randomize pixel locations in flat local regions R. Particularly, the current pixel Pi is replaced by a pixel randomly selected from the neighborhood. The range of the neighborhood depends on the local entropy H. The range can be larger for regions with lower entropy H, and the other way around. Both solutions can also be used without a filter LPF at all, or with a filter LPF which is not controlled based on the probability density function. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb "comprise" and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.