The invention relates to a method of discriminating in an image between synthetic image regions and natural image regions.

The invention further relates to a natural content detector which is operative to perform said method. The invention further relates to a computer program product for performing said method.

The invention further relates to a display image processor comprising said natural content detector.

Such a method is known from the international patent application WO 03/049036. In WO 03/049036 a method of discriminating in an image between synthetic image regions and natural image regions is described. The method comprises a plurality of probability estimation steps, each step estimating for a particular pixel of the image an elementary probability of the particular pixel of being located in one of the natural image regions based on values of pixels of a group of pixels in a neighborhood of the particular pixel. A final probability value is determined by combining the elementary probability values.

One of the elementary probability values is a Separation of Values (SOV) value. This elementary probability value is calculated by means of weighted summation of differences between pixel values of pixels of a group of pixels. The differences correspond to distances between non-zero bins in a histogram of pixel values of the group of pixels. The histogram or array of pixel values is scanned and the differences between the pixel values are calculated. Once the list of pixel values that is present in the group of pixels is ordered, the distance or separation between each value of this list and the next value in the list is calculated. Each separation is then weighted and added to compute the probability value SOV.

It is a disadvantage of the known method for natural content detection that the resulting probability value is not constant within natural areas which results in parts inside

natural areas being erroneously identified as synthetic content. It is another disadvantage of the known method that it may result in faulty detections. Therefore this method may lead to synthetic regions being classified as natural regions and vice versa.

It is an object of the invention to provide a more reliable method for discriminating between synthetic and natural image regions.

According to the invention, this object is achieved by providing a method of discriminating in an image between a synthetic image region and a natural image region, the method comprising for each pixel of a group of at least three pixels in a neighborhood of a particular pixel of the image, a step of determining differences between a pixel value of the particular pixel and a pixel value of the pixel of the group, for each difference a step of weighting the difference with a weighting function, and a step of determining a summation of the weighted differences for estimating for the particular pixel a probability value representing a probability of the particular pixel of being located in the synthetic image region or in the natural image region.

With this method the single differences between the particular pixel and each one of its neighbors are calculated and weighted separately before being summed together. All differences are calculated for the particular pixel itself. With the prior art method in most differences the particular pixel is not involved, but only neighboring pixels are. With the method according to the invention each difference involves the particular pixel itself. The weighted sum of the differences provides for a more reliable measure of the probability for the particular pixel of being in a natural image region. The method according to the invention therefore results in less faulty detections.

In the easiest to build embodiment the weighting function assigns an equal weight to all differences. In a preferred embodiment the weighting function is selected in accordance with a statistical distribution of the differences for natural images. In this embodiment, a weighted difference represents a likelihood for said difference to appear in a natural image. As all weighted differences refer to the particular pixel, the sum can be interpreted as a likelihood of the particular pixel to be part of a natural content area, obtained as average of a likelihood of each of the differences.

In another embodiment the step of weighting assigns a maximum weight to one of the differences if said one of the differences equals a predetermined noise threshold which defines an upper limit of a range of differences which are interpreted as noise.

In general small differences point to natural image regions. However because most images are affected by noise, also smooth areas with pixels with equal pixel values show small differences. Synthetic image regions are characterized by sharp transitions and smooth areas with pixels with equal pixel values. In this preferred embodiment the noise threshold prevents the smooth areas of synthetic images, if being affected by noise, from being characterized as natural content. Another embodiment further comprises a step of setting the probability value to zero or near zero if at least one of the differences is above a predetermined synthetic graphics threshold which defines a lower limit of a range of differences which are interpreted as being part of the synthetic image region.

Very abrupt transitions of pixel values between two neighboring pixels do generally not occur in natural images. Therefore, in this embodiment any pixel which shows such an abrupt pixel value transition between itself and any neighboring pixel is considered to be part of a synthetic region and is assigned a probability of zero or near zero. As a consequence, a pixel will only be assigned a probability value substantially larger than zero if all pixels of the group of neighboring pixel have pixel values which differ less than the predetermined synthetic graphic threshold from the pixel value of the particular pixel.

Another embodiment further comprises a step of setting the probability value to zero or near zero if the pixel values of the particular pixel and the group of pixels together form a predetermined synthetic graphic pattern which indicates synthetic information.

In synthetic image content the pixel values of pixels of a group of neighboring pixels are often distributed in regular pattern. For example, pixels in a vertical, horizontal or diagonal straight line may have identical or nearly identical pixel values. Alternatively, when the pixels are arranged in rows and columns the pixels may be arranged in a chessboard pattern. In this embodiment a pixel which is part of such a predetermined synthetic pattern is considered to be part of a synthetic image region. A further embodiment comprises a step of determining an enhanced probability value based on a number of pixels in a neighborhood of the particular pixel of which the probability value is above a predetermined minimum natural value.

In this embodiment areas with a relatively high amount of pixels with a high probability of being part of a natural image region are considered to be natural areas. This

embodiment of the method according to the invention results in better content separation and smooth transitions from natural content areas to synthetic content areas. It prevents isolated pixels being classified different from the surrounding pixels.

According to another aspect of the invention a natural content detector and a computer program product are provided for performing the methods as described above. Furthermore a display image processor is provided, comprising such a natural content detector.

These and other aspects of the invention are apparent from and will be elucidated with reference to the embodiments described hereinafter.

Brief description of the drawings In the drawings:

Figure 1 shows a block diagram of a display image processor comprising a natural content detector,

Figures 2a and 2b show an exemplary arrangement of pixels in an image, Figure 3 shows a flow diagram of the method according to the invention, Figure 4 shows an example of a weighting function for use with the method according to the invention, Figure 5 shows a block diagram of a natural content detector according to the invention,

Figures 6a, 6b and 6c show examples of predetermined synthetic graphic patterns which indicate the presence of synthetic information,

Figure 7 shows a block diagram of a Graphic Pattern Inhibition unit, Figure 8 shows a flow diagram of a morphological filter operation,

Figure 9 schematically shows an embodiment of a luminance window and morphological window, and

Figure 10 shows an exemplary input and three exemplary outputs of a natural content detection process.

Figure 1 shows a block diagram of a display image processor 1 comprising a natural content detector 4. At the input 2 of the image processor 1, graphics information is received. The graphics information may, for example, be provided by a video card of a

computer, by an output of a TV tuner or by an output of a DVD player. A converter 3 converts the received graphics information to a pixel value format which is suitable for, e.g., display on a display screen which may be coupled to the output of the display image processor 1. The output of the converter 3 may, for example, provide RGB pixel values or YUV pixel values. The natural content detector 4 uses one or more of these pixels values for performing the method according to the invention. From now on the invention will be described with the luminance value (Y in YUV) of the pixels used as input for the natural content detection, but the skilled man could easily use other pixel values.

Purpose of the natural content detector 4 is to discriminate natural (e.g. photo, TV-like) and synthetic (e.g. graphics) areas inside an image. Such information can be exploited in a variety of situations (e.g. basic image segmentation for object recognition and retrieval, automatic optimization of the bandwidth for coding applications, etc.) and it is particularly valuable to control image processing algorithms that perform with different quality on graphics/text and natural content. Two important examples are those of video algorithms applied to computer generated images that are typically composed of mixed graphics/text and natural content (e.g. web browser windows, photo editing software, etc.) and graphic overlay on video sequences. Such algorithms, like sharpness enhancement and color boosting, are specifically designed for natural content, and produce very noticeable artefacts when run on system graphics or text. Moreover, it can be assumed that synthetic content is already optimized for the display by the system (graphic adapter + operating system), and therefore needs small or no enhancement. Therefore, the natural content estimation can be used to enable processing on photos and videos only, thus overcoming the limitation of undifferentiated processing that affects most of today's display image processors. The enhancement algorithms are executed by the enhancement module 5 which receives the pixel values from the converter 3 and probability values P from the natural content detector 4. The probability values P represent the probability of a particular pixel of being located in a natural image region.

Figure 2a shows an exemplary arrangement of pixels 9 in an image. The pixels 9 are arranged in rows and columns. Figure 2a shows part of three rows (n-1, n and n+1) of pixels 9. For the calculation of the probability value of a pixel on row n, its luminance value Y4 and the luminance values YO, Yl, Y2, Y3, Y5, Y6, Y7 and Y8 of its neighbors are considered. Figure 2b shows the differences d, for all eight neighbors which differences d, are used for calculating the probability value P. Each difference d, equals the absolute value of the difference between its luminance value Y ₁ and the luminance value I ₄ of the central pixel.

d, = \Y, - Y ₄ \ (1)

Figure 3 shows a flow diagram of the method according to the invention. In a first step 31, the natural content detector 4 receives the luminance values Y0-Y8 of the pixels in the 3 x 3 luminance window YW (Fig. 2a) of which the pixel with luminance value Y4 is the central pixel for which to calculate the probability value P has to be calculated. The skilled man will be able to apply the method according to the invention using other groups of neighboring pixels like, for example, a 5x5 window or a cross which only comprises the central pixel and its four directly adjacent neighbors.

Then in difference step 32, the luminance difference between a first neighboring pixel and the central pixel is calculated: do = \Yo - Y ₄ \ (2)

In weighting step 33 the difference do is weighted using a weighting function W(d _t ) which will be elucidated in more detail later on, with reference to Figure 4. The weighted difference W(do) is then stored in a memory in summing step 34. Then in a control step 35 it is controlled whether all 8 neighbors are already processed. If not, then the natural content detector 4 returns to the difference step 32 to calculate the luminance difference d, between the central pixel and the next neighbor to be processed, using formula (1).

Then the differences d, is also weighted in weighting step 33 and the weighted difference W(d _t ) is added to the value which is already stored in the memory. When all eight neighboring pixels are processed the probability value

P^-∑Wid,) (3)

O (=0 i≠4 is provided to, for example, the enhancement module 5 (Fig. 1) in output step 36.

The quality of the natural content detection strongly depends on the shape of the weighting function W(d _t ). In a preferred embodiment the weighting function W(d _t ) is selected in accordance with a statistical distribution of the differences d, for natural images.

The weighting function W(d _t ) may be realized by means of a lookup table wherein weights are listed for various differences d,. Alternatively the weights are calculated using analytical calculation.

Figure 4 shows an example of a weighting function W(d,) for use with the method according to the invention. Other weighting curves approximating the first order difference distribution may be adopted, but the piecewise linear function shown in Figure 4

leads to a very effective hardware implementation that does not need a lookup table. The weighting function W(d _t ) shown in Figure 4 is defined as follows:

W(d, ) = t Jim (4)

With

Sl = and 5Oe 0, (5)

Act Hm - N th JV th

Note that 51 is completely defined by the two parameters Act Jim and N Jh, while 50 can be freely chosen in the range [0, 1 /N Jh] (see below for a discussion on the value to be selected for 50).

The shape of the weighting curve is selected to approximate, in its active range [N Jh Act Jim], the statistical distribution of the absolute first order differences for natural images. In this respect, the weighted differences are interpreted as the likelihood that such differences d, belong to a natural image. As all differences d, refer to the central pixel, the sum in (3) can be interpreted as the likelihood of the central pixel to be part of a natural area, obtained as average of the likelihood of the eight radial differences d _t . It can be noted that since high gradients are less likely to be part of a natural image, their contribution to the weighting sum is lower.

The weighting function W(d _t ) comprises the following weighting curve parameters:

- The noise threshold N Jh defines the gradient range where luminance fluctuations are interpreted as noise. Perfectly constant luminance areas are very likely to be the (synthetic) background of some software application. Therefore they should be marked as non natural. As additive noise is a low amplitude signal, it generates a low gradient that would be interpreted as an indication of natural content if no noise threshold N Jh would be used. - Parameter SO is the slope of the weighting curve W(d _t ) inside the gradient range defined by N Jh. When 50 = 0, differences d, below N Jh give no contribution to the weighted sum. This allows for instance to set the natural estimation to zero whenever additive noise that affects a constant luminance area is below the noise threshold. However, a possible drawback of setting 50 = 0 is the discontinuity in W(d,) around N Jh. If differences d, around N Jh are affected by some noise, this may result in excessive variability of the weighted sum. Actually, depending on the instant value of noise, the same difference could

be just above or below N_th, thus switching its contribution to the weighted sum from 0 to about 1. Higher values for SO can be chosen to find the appropriate trade-off between noise suppression and natural estimate variability.

- The active limit Act Jim defines the upper limit of the active range. Differences d, above this limit give zero contribution to the weighted sum.

- The threshold Graph Jh defines the limit over which a very abrupt transition, in whichever direction, is assumed part of a graphic pattern. Some common image enhancement algorithms, like sharpness enhancement, perform highpass filtering. When applied to steep transitions, like those present in text patterns, the result is a noticeable artifact, usually detected as halos around the edge. It is important that in these cases the enhancement is completely suppressed. Therefore, if at least one of the radial differences d, is greater than Graph Jh, the pixel is assigned a probability value P of zero.

Assuming that the input luminance Y is given in the range [0, 255], reference ranges for the three parameters, determined experimentally, are: N Jh : 0- 10 (0 for signals not affected by noise)

Actjim: 60-120 Graph Jh: 80-150

Figure 5 shows a block diagram of a natural content detector 4 according to the invention. At the input 41 the natural content detector 4 receives graphics data concerning three pixel rows (n-1, n and n+1). The luminance window extraction unit 42 extracts a 3 x 3 luminance window YW around the pixel for which the estimation must be determined, from the three luminance lines surrounding it. The next stage is the Radial Gradient Operator (RGO) 44, which performs the basic natural content probability estimation as described above. In order to further improve the estimation two auxiliary blocks are introduced:

- A Graphic Pattern Inhibition unit (GPI) 43 detects some common graphic patterns, like horizontal and vertical lines or chessboard pixels arrangements. Whenever one of these graphic patterns is detected the input probability value P of the following morphological processing is forced to zero. The operation of the GPI 43 will further be elucidated later on with reference to the figures 6 and 7.

The Morphological Filter (MF) 45 further improves the estimation performed by RGO 44 in several ways. The operation of the MF 45 will further be elucidated later on with reference to the figures 8 and 9. At the output of the MF 45 an enhanced probability value P _e is provided. A switch 46 may be used for providing a choice of using the MF 45 or

not. Whether the output is processed or not by the MF 45, the resulting (enhanced) probability value P^) is preferably a real number in the range [0,1].

The RGO 44 defines a natural estimation based exclusively on the amplitude of the local luminance variations. Although this is satisfactory in excluding most synthetic graphic content, it does not cover some common graphic patterns characterized by mid- low contrast that are similar to natural areas as regards the resulting gradient, but different in the way luminance values are arranged in the 3 x 3 analysis window. Most recurring cases include the exemplary synthetic graphic patterns shown in Figure 6a, 6b and 6c.

Figure 6a shows a horizontal line. Horizontal and vertical lines are typically used as separator in web pages or application menus, as well as text strokes. Figure 6b shows a dithering pattern used in graphics rendered with few colors (typically GIF images in web pages). These patterns which are often referred to as 'pixel ON-pixel OFF', resemble a checkerboard, interleaving two colors to create the impression of an intermediate constant color. Figure 6c shows a uniform luminance ramp which is often found as background in web pages and in Windows title bars.

Purpose of the GPI 43 is to detect the patterns described above. Whenever this occurs its binary output is set and consequently the natural probability value P is forced to zero (see Figure 5). A block diagram summarizing the operation of GPI 43 is shown in Figure 7. Figure 7 shows a block diagram of a GPI 43. In principle, all of the above cases can be identified by checking that the same luminance is found on certain positions in the 3x3 analysis window. More specifically:

- Horizontal lines are detected by the horizontal line detector 71 when the luminance of at least one of the three rows is constant (YO=Y1=Y2 OR Y3=Y4=Y5 OR Y6=Y7=Y8).

- Vertical lines are detected by the vertical line detector 72 when the luminance of at least one of the three columns is constant (Y0=Y3=Y6 OR Y1=Y4=Y7 OR Y2=Y5=Y8).

- Pixel ON-Pixel OFF patterns are detected by the quincunx detector 73 when the same luminance is found in positions [0, 2, 4, 6, 8] or [1, 3, 5, 7] that is the quincunx and its complement, respectively.

If the output of at least one of the pattern detectors 71, 72, 73 is positive, the output of the OR gate 74 gives a positive output (1) else the output is negative (0).

Pattern detection relies on the assumption that luminance is exactly the same in all pattern-specific positions. However, if the input signal comes from an analogue source, it is likely to be affected by additive noise, and the strict equality condition is actually not applicable. A straightforward workaround is to replace equality with a constraint on the maximum luminance range. Given an estimation of the maximum noise amplitude noise est, a pattern is detected when the overall luminance variation (maximum - minimum) over the pixels belonging to the pattern is below noise est. However, hardware implementation involving calculation of maximum and minimum over a set of pixels is not trivial, as it implies many comparisons that must be repeated for all patterns to be checked. Here we adopt a simpler solution that (re)uses the radial differences, referred to as Luminance

Window Coring, which is performed by the Luminance Window Coring unit (LWC) 75. The analysis window YW is first processed with the purpose of neutralizing the noise. The equality checks are then applied to the processed luminance YC. Since most unwanted patterns include the central pixel, it is taken as reference and compared to all outer pixels of the analysis window YW. If the luminance difference d, is less than noise est, it is interpreted as noisy and the corresponding luminance Yi is made equal to the central value (YCi = Yi). On the contrary, if the d, is bigger than noise est the luminance remains the same:

[YA d, < noise est

YCi = I ' ^~ (6)

[Yi d _t ≥ noise _ est

Note that while the coring procedure effectively helps detecting patterns, it also gives rise to misdetections when the analysis window is placed over a constant area affected by noise. In this case, all pixels YCi are assigned the same value Y4 and therefore all pattern detectors 71, 72, 73 produce positive output. In order to prevent such behavior, the pattern detection output is enabled only if at least one of the radial differences is above noise est which is checked by difference checker 76. Only if the output of the OR gate 74 and the output of the difference checker 76 are both positive then the output of the GPI 43 is positive (see final AND gate 77).

The RGO 44 works on a very localized and small amount of pixels (3x3 luminance analysis window). With such limited data, the bare use of the RGO 44 could be insufficient to tell when similar pixel structures belong to different contents. For instance, smooth system graphics (like true-color icons) produce similar probability values P as details in sharp photos and videos. This and other limitations of the RGO 44 are tackled by the MF 45, which accomplishes four important functions: - better content separation,

- compact areas with sufficient natural pixel density,

- smooth natural area borders, to allow for a gradual fade in/out of the controlled enhancement algorithms, and

- remove spurious/isolated pixels in the output natural map. The idea behind the morphological filter is to evaluate probability values P around the current pixel within a relatively big window (compared to the luminance window YW) to catch more of the surrounding information, but on simple data and with minimal calculations. Looking at probability values P on a scale larger than the analysis window YW we note that, differently from graphics, natural areas are usually characterized by a considerably bigger amount of pixels with a high probability value P. Based on this observation, a better separation between graphics and natural content can be obtained defining a probability value P density index.

This enhanced probability value P _e once normalized in the range [0 1], is taken as the final output of the system, and is still interpreted as the probability that the central pixel is part of a natural area.

Figure 8 shows the operation scheme of the morphological filter, detailed below. The probability value P, is first compared with MinNatValue in threshold step 81. This operates a selection of pixels with high probability P to be natural. The resulting binary marking PJh is fed into a pipeline of MHeight line memories in storage step 82. Then a subsequent column selector, which produces a MHeight x MWidth matrix, referred to as the morphological window MW, that stores the values of P Jh for the pixels surrounding that at coordinates (i,j) for which the natural probability has to be computed in window selection step 83. Equivalently, the morphological window MW can be seen as a moving window with the same center as the luminance window YW. Satisfactory values for the size of the morphological window have been determined experimentally: MHeight = 7 Mwidth = 15.

These values are examples only. A skilled man may choose to use other values. In a counting step 84 the non-zero elements in the morphological window are counted. The count of non-zero elements in this window, NatCounter, provides the desired measure of density related to the central pixel, and is taken as the basic measure for the enhanced probability value P _e .

Better content separation, compacting areas and smooth borders in the output natural map, are realized by the definition of NatCounter. Actually, since NatCounter is a cumulative sum, it can be interpreted as a non-normalized average over the morpho logical window MW, that is a low pass filter. This ensures that the estimate around the borders of natural areas changes gradually. Moreover, as the count does not depend on the particular distribution of ones in the morphological window, non-uniform arrangements (i.e. with 'holes') are flattened, thus compacting the map inside natural areas. Cleaning the output natural map from spurious/isolated pixels, is performed by the last step 85 in Figure 8. The numerator in the expression

_ MAX{0, NatCounter-MinMorph } Mwidth • Mheight - MinMorph sets a lower positive limit to NatCounter, yielding zero output whenever it is lower than

MinMorph, which occurs when only few pixels (less than MinMorph) are marked as natural in the morpho logical window MW. For values bigger than MinMorph, P _e gradually increases, reaching one when all pixels are marked as natural. Reference value for MinMorph is 5% of the morphological window area, that is MinMorph = 0.05 x MHeigth x MWidth.

Figure 9 schematically shows an embodiment of a luminance window YW and morphological window MW. In Figure 9a the 3 x 3 luminance window YW is highlighted.

The luminance window YW comprised a central pixel (i,f) and its eight neighbors. In Figure 9b the MWidht x MHeight (7 x 5) morphological window MW for said central pixel (i,j) is highlighted. The skilled man may choose to use a luminance window YW or a morphological window MW with an other shape or size.

Figure 10a shows an exemplary input picture for the display image processor

1. The input picture comprises natural content areas 101 as well as synthetic content areas 102. Figure 10b shows an exemplary output of an RGO 43. Figure 10c shows an exemplary output of an RGO combined with a GPI 44. Figure 1Od shows an exemplary output of an

RGO 43, combined with a GPI 44 and a MF 45. Brighter pixels correspond to higher probability values Pl P _e (black=0, white=l).

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb "comprise" and its conjugations does not exclude the presence of elements or

steps other than those stated in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.