METHOD AND APPARATUS FOR MAPPING THE COLORS OF AN IMAGE ACCORDING TO THE CONTENT OF THIS IMAGE

Technical Field

The invention relates to a mapping method of the colors of an image and to the corresponding apparatus. The invention concerns notably a method and a system modifying the color gamut of images depending on the content of those images.

Background Art

Images are usually artistically produced for specific reference or target display devices. Images include photos, video content, stereo images, feature films, graphics content, multiple view content and other pictorial and textual content that is intended to be reproduced on display devices. When a photographer captures and post processes, i.e. produces an appealing photo, he might use a computer workstation for the processing with a computer monitor as reference display. In video production for movie theaters or bluray discs, the director of photography uses color grading equipment for the processing of the video in order to arrange colors, contrast, saturation, black level, white level, white temperature and hues in the images of the video, and a cinema projector will be likely used to reproduce the video production. Today, studios use generally a ITU-R BT.709 compliant monitor to adjust colors in video content when producing a video content for bluray discs.

There is content prepared with reference devices that have better color capabilities than the target display that is finally used for the reproduction of the content. For example, feature films are produced for cinematographic projectors as reference displays but might be proof viewed by the director on a PC display (as target display) that's has a smaller color gamut, a higher black level and a lower contrast that the cinematographic projectors. It is therefore a problem how high quality content can be efficiently adapted to displays with less color capabilities, notably with smaller color gamut, lower contrast ratio, lower luminance levels and smaller bit depth.

There is also legacy content prepared with legacy reference displays that have limited color capabilities with respect to more recent target displays. For example, a huge amount of content (TV broadcast, DVD, blue ray) is produced for displays according to the ITU-R BT.709 standard while today's consumer target displays, for example LCD displays, have often a larger color gamut that defined by this standard taken as a reference. Additionally, luminance levels and contrast ratios evolve in recent displays.

It is therefore a problem how legacy content can be efficiently adapted to target displays with better color capabilities than the reference display used for their production and/or transmission, notably adapted to target displays with wide color gamut, higher contrast ratio, higher luminance levels and higher bit depth.

In the patent US20070291 179, it is proposed to transmit video using two layers to display devices that have larger color capabilities. The first layer has classical color properties being sufficient for legacy display devices having color capabilities corresponding to color capabilities of CRTs taken as reference display. In a second layer, metadata about enhanced color is transmitted so that content can be actually displayed with larger color capabilities on more advanced target display devices having these larger color capabilities, such as LCD displays. Additionally, auxiliary data is transmitted to the display defining display settings concerning those color capabilities such as brightness, contrast, and color space. When using this method, the described adaptation problem is solved. Unfortunately, during content creation, the two layers have to be produced. Thus, this method cannot be used if only legacy content is available, i.e. if the second layer is not available.

Therefore, there is a need to adapt a legacy content based on the content itself.

In the patent US7903178, a method is proposed to enhance the colors of images by mapping the color coordinates of the images color region by color region, each region being defined by a common luminance value, and the mapping being a factor per region and thus enhancing the contrast.

In this document, a color aspect is defined for each color region of an image, based on a luminance histogram specific to this color region. From the definition of this color aspect, a color information can be determined for each color region of an image, namely the luminance histogram itself of this region (fig.4 of US7903178). Then, for each color aspect, namely for each luminance histogram specific to a color region, there is a specific rule that is adapted to determine an amount of change of this color aspect (i.e. amount of change of the histogram) in function of the color information (i.e. the original luminance histogram) representing this color aspect of the image. This rule is predefined and is for instance based on a greater amount of change for the luminance histogram of the "dark region" than for the luminance histograms of the other color regions (see col.4, lines 56-58). The application of the rule specific to each color region of the image leads to a global color correction based on the content itself. Such a color correction is equivalent to a color mapping.

This method of color mapping has several drawbacks. A drawback of this method is that it requires large computational effort, notably because it is applied to color coordinates of psycho-visual color spaces such as L,a,b coordinates of CIELAB such as standardized by the CIE in 1976. This is due to the highly non-linear characteristics of these color spaces. Notably in video applications, real time processing might be required and calculation of L,a,b coordinates might not be possible.

A good background for the present invention is the patent US8154560. In the method described in this document, initial color gamut and target color gamut information is used to decide about a saturation expansion coefficient that is then used to control saturation expansion of images. However, it is not specified how saturation expansion can be done with few computational effort while ensuring at the same time that only saturation.

In this document, a color aspect is defined for each color, namely a saturation expansion coefficient which is calculated from xy coordinates of this color (col. 9, lines 1 -16). From the definition of this color aspect, a color information is determined for each color of an image, namely the value of the corresponding saturation expansion coefficient. Then, for this color aspect, there is a specific rule that is adapted to determine an amount of change of this color aspect (i.e. difference between this color and the color obtained by applying the saturation expansion coefficient) in function of the color information (i.e. the saturation expansion coefficient) representing this color aspect of the image. The application of the rule specific to each color of the image leads to a color mapping of the content.

Summary of invention

An object of the invention is to avoid the aforementioned drawbacks,

For this purpose, the subject of the invention is a method of mapping the colors of an image, a series of color mapping operators being predetermined which differentiate one from another by different amount of change of at least one color aspect of an image, and, for each of at least one color aspect of an image, a rule being set that is adapted to determine an amount of change of said color aspect in function of a color information representing said color aspect of an image, comprising:

- for each of at least one color aspect, the determination of said color

information representing said color aspect of said image to map,

- the application of the rule related to said color aspect to the determined color information in order to determine an amount of change of said color aspect,

- within said series, the selection of at least one color mapping operator such that the combination of said at least one color mapping operator provides an amount of change of said color aspect corresponding to the determined amount of change,

- the application of the combination of said selected at least one color mapping operator to said image to map.

When only one color mapping operator is selected, the co-called

combination corresponds to this selected color mapping operator alone.

The series of predetermined color mapping operators and the rules adapted to determine an amount of change of the color aspects are generally stored in the apparatus used to implement this color mapping method, or sent to this apparatus before implementing this method. As these operators are predeternnined and as these rules are set in advance, the computational effort needed to implement the color mapping method is advantageously limited.

The amount of change of the color aspect - named "requested amount" - which is determined by the application of the rule in function of a color information representing the color aspect of an image may be distinguished in definition from the definition of the amounts of change of color aspect that differentiate one from the color mapping operators of the series.

Preferably, color mapping operators of said series are predetermined to change the color gamut of said image.

Preferably, said color information comprises lightness, brightness, colorfulness, saturation, contrast, white temperature, black level and/or hue.

Preferably, said color information is semantic, being at least one of shape, motion, texture, illumination or reflectance.

The method according to the invention has notably at least one of the following advantages over existing and known methods:

1 . Content can be adapted to any type of display device without producing a specific version for specific displays.

2. Highly complex and computationally demanding gamut modification operators ensuring high quality requirements can be implemented.

3. Low computational effort and high speed.

4. Adaptive to images. A subject of the invention is also a color mapping apparatus for the mapping the colors of an image, comprising,

- a module configured to receive and/or to store a series of predetermined color mapping operators which differentiate one from another by different amount of change of at least one color aspect of an image, and, to receive and/or to store, for each of at least one color aspect of an image, a predetermined rule that is adapted to determine an amount of change of said color aspect in function of a color information representing said color aspect of an image,

- a module configured to determine, for each of at least one color aspect, color information representing said color aspect of said image to map,

- a module configured to apply the rule related to said color aspect to the determined color information in order to determine an amount of change of said color aspect,

- a module configured to select, within said series, at least one color mapping operator such that the combination of said at least one color mapping operator provides an amount of change of said color aspect corresponding to the determined amount of change,

- a module configured to apply the combination of said selected at least one color mapping operator to said image to map.

Brief description of drawings

The invention will be more clearly understood on reading the description which follows, given by way of non-limiting example and with reference to the appended figure 1 which illustrates a non-limiting embodiment of implementation of the method according to the invention.

Description of embodiments

In order to implement an embodiment of the method according to the invention, a color mapping apparatus is used which comprises notably :

- a module configured to store a series of predetermined color mapping operators and to store, for a color aspect of an image, a predetermined rule that is adapted to determine an amount of change of this color aspect in function of a color information representing this color aspect of an image,

- a module configured to determine a color information representing the color aspect of an image to map,

- a module configured to apply the predetermined rule to a determined color information in order to determine an amount of change of this color aspect corresponding to this color information,

- a module configured to select, within the series above, at least one color mapping operator such that the combination of this at least one color mapping operator provides an amount of change of the color aspect corresponding to the determined amount of change, - a module configured to apply the combination of these selected at least one color mapping operator to said image to map.

The functions of the various elements shown in figure 1 which illustrates the method according to an embodiment of the invention may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. Notably, each module of the color mapping apparatus corresponds to a piece of such hardware and/or software. In the following, the non-limiting embodiment of mapping the colors of a video content according to the invention is described. The implementation of this embodiment is shown in Figure 1 . Here, the video content to map has been produced using a video production reference display with limited color capabilities. This implementation aims a color gamut expansion of the video content so that the mapped video content can be watched on a target reproduction display with better color capabilities. Aspects of colors of the video content that can be modified include for instance hue, lightness, saturation, white temperature, white level, black level and/or contrast. Here, the color aspect that will be changed through this implementation of the embodiment is color saturation. The invention is not limited to color gamut expansion; there are for instance other applications of the invention that require gamut compression, such as HDTV to SDTV conversion.

In this embodiment; the mapping method according to the invention is applied image by image, i.e. frame by frame to the video content. The image information that represents the color aspect to be changed through the color mapping and that is used for the application of the rule described below is extracted in real time from each frame or image of the video. We specifically use here color information related to the presence of saturated colors in the frames. The colors of each frame or image are represented by RGB coordinates encoded according to the standard ITU-R BT.709.

In order to determine the image information related to color saturation of a frame, for each pixel of this frame associated with a color with color coordinates R,G,B, we determine the largest and the smallest colors coordinates L=max(R,G,B) and S=min(R,G,B), hereby coordinate value of zero are excluded, calculate a saturation ratio Q for this pixel according to the formula Q=L/S and then calculate the mean saturation ratio Qjnean over all pixels of this frame. The color information representing color saturation of the frame to map is then Qjnean. In typical HDTV video, RGB coordinates are within a range of 0..255. Qjnean is then generally between 1 and 255.

Another possibility to implement the method according to the invention is to apply the above method to a group of frames instead of each frame. In this case, Qjnean is calculated over the group of frames.

In a variant, the color information of each frame may be received over a transmission channel from a specific provider of image information.

1/ series of predetermined gamut change operators:

A set or series of 1 1 gamut change operators is predetermined as follows. These operators are based on color management (CMM).

For CMM, the color characteristics of the video production reference display compliant with the ITU-R BT.709 standard and the aimed target reproduction display are measured, mathematically modeled and then compensated in a manner known per se using a color transformation.

All gamut change operators of the series are also adapted to map colors from the color gamut of the video production reference display into the color gamut of the target reproduction display. In general, such gamut mapping operators can be applied to any color values that are defined within a source color gamut in order to transform these color values such that they are included in a target color gamut. The source color gamut can be linked to a capture device such as camera or scanner. It can be linked to a reference display device such as the video production reference display. It can also be linked to a predefined color gamut for example according to a standard such as ITU-R BT.709 as stated above. The target color gamut can be linked to a targeted display device such as the mentioned target reproduction display. It can be linked also to a predefined gamut for transmission, compression or storage purpose, for example the standard ITU-R BT.2020. It can be linked to a medium such as film or paper prints. In the following we will simplify by talking about a source display with a source color gamut or reference color gamut and a target display having a target color gamut.

The gamut change operators of the series aim at the reproduction of a color on the target display such that this color as reproduced is identical or as close as possible to the color as displayed on the reference display or source display. The colors can be measured using the XYZ values of the CIE. Colors can also be measured using color appearance attributes calculated by a color appearance model, for example the CIECAM-02 of the CIE. Those models use so-called perceptually constant color spaces that have axes for intensity, hue and saturation of colors. A simple version of these spaces is CIELAB defined by the CIE in 1976.

Here, each of the 1 1 predetermined gamut change operators are composed of the following operator steps:

- A source device model transforming the ITU-R BT.709 RGB color coordinates of a color into XYZ color coordinates of this color,

- An appearance attribute calculator calculating Lab color coordinates of a color in CIELAB color space from the XYZ color coordinates of this color,

- A gamut boundary representation means adapted to represent the gamut boundary of the video production reference display that corresponds to ITU- R BT.709 (i.e. the reference color gamut) and to represent the gamut boundary of the target display (i.e. the target color gamut) in CIELAB color space,

- A color mapper for the implementation of a percentage of a gamut mapping algorithm adapted to map, in the Lab color space, colors from the reference color gamut into the target color gamut resulting into mapped colors, wherein these colors and mapped colors are represented by Lab color coordinates (the implementation of a percentage of a gamut mapping algorithm is explained below),

- An inverse appearance attribute calculator that calculates XYZ color coordinates of a color from Lab color coordinates of this color,

- A target device model adapted to transform XYZ color coordinates of a color into RGB color coordinates of this color. As gamut mapping algorithm, we use an algorithm close to the CARISMA algorithm proposed by Green and Luo in their paper entitled "Extending the CARISMA gamut mapping model" published in the Journal of Imaging Science and Technology volume 46 number 1 in 2002. However, the invention is not limited to this gamut mapping algorithm and other algorithms can be used. In general, gamut mapping algorithm is computationally demanding and therefore the color mapper requires usually high computational effort.

The original CARISMA algorithm changes notably the hue of a color and applies different mapping methods for different hue sections of the color space, depending on the relative shapes of the reference color gamut and of the target color gamut. In our implementation of the embodiment, we apply simply a linear chroma/saturation expansion from the reference color gamut to the target gamut along lines with constant hue and lightness. An additional hue shift is defined - similar to the CARISMA algorithm - for those hue planes containing either a primary or a secondary color of the reference color gamut - i.e. red, green, blue for primary colors and magenta, cyan, yellow for secondary colors - half way towards the hue of the corresponding primary or secondary color, respectively, of the target gamut. Hue shift of intermediate colors between primary and secondary colors are determined by interpolation between two adjacent primary and/or secondary colors.

Each of the 1 1 gamut change operators are characterized by an amount A of modification of the color aspect of an image, here an amount of change of the color saturation of this image. The 1 1 different values of A corresponding to the 1 1 different operators could be 0, 0.1 , 0.2, ... 1 , and represent a percentage 0%, 10%, 20%, 100% of application of the described modified CARISMA gamut mapping method when these operators are applied to a color. The value of A characterizing an operator means how far the aimed color aspect is changed by this operator after a color is mapped by this operator. A describes thus the actual amount of modification of the aimed color aspect by this operator. The first operator of the series with A=0%, when applied to an image, does not change hues and saturation of the colors of this image. The eleventh operator of the series with A=100%, when applied to an image, changes hues and saturation of the colors of this image as proposed by the modified CARISMA method described above. It is possible that an operator changes other color aspects, here hue, than the aimed color aspect, here saturation. Each of the other operators with 0<A<100% :

- first map a color C onto a mapped color C according to the modified CARISMA algorithm, C and C being vectors of Lab coordinates,

- then calculates the mapped color C" obtained by this operator by the equation C" = (1 -A) C + A C, C" being a vector with Lab color coordinates.

Instead of using the modified CARISMA algorithm to define the different operators of a series, other gamut mapping methods could be used, for example a modification of the SGCK algorithm proposed by Central Bureau of the CIE, Vienna in the "Guidelines for the Evaluation of Gamut Mapping Algorithms" publication no. 156 published in 2004. Originally, this algorithm keeps hue constant, compresses lightness by a sigmoidal function in chroma- dependent manner, and compresses colors along trajectories in color space in direction of a so-called anchor point that is dependent on the shape of source and target gamuts. This algorithm would need to be modified in order to allow expansion beside compression, depending on whether the target gamut is larger or smaller than the source gamut, respectively. By this way, contrast but also saturation can be increased, not only reduced, i.e. the reduced gamut of a reference display can be expanded to the larger gamut of a target display.

All gamut change operators defined above are pre-calculated in form of RGB Look Up Tables (LUT) having LUT entry values and LUT output values. For any given LUT entry corresponding to a triple of RGB image color coordinates representing a color to map, the described operator steps above defining a gamut operator are applied and the result target RGB coordinates representing a color mapped according to this operator are stored in the LUT as the corresponding LUT output value. The RGB input color coordinates are preferably regularly sampled in the RGB space. Instead, a non-regular sampling might be used in order to account for non-linear encoding of RGB coordinates such as the case in ITU-R BT.709 standard.

21 Rule to determine the amount of modification RA in function of the color information Q mean: A rule is set that is adapted to determine a requested amount of change RA of the color aspect (here : color saturation) in function of the color information Qjnean (as defined above) representing the color saturation of an image. Here, we set up one color change rule stating that the required amount RA of modification of saturation and hue of the colors of an image shall be RA=0 - representing 0% - in case the image information Qjnean related to color saturation of an image is 255 and shall be RA=1 - representing 100% - in case the image information Qjnean related to color saturation of this image is 1 , and, in the intermediate cases, RA is determined according to the equation : RA=(255-Qjnean)/255, expressed in percent. As explained above, the modified CARISMA algorithm changes the aimed aspect of colors of an image, here saturation, and other aspects of colors of this image, here hue.

3/ Implementation of the method according to the invention:

Again, in this embodiment, the mapping method according to the invention is applied image by image, i.e. frame by frame to the video content.

Then, for each image of the video content to map, within the series of 1 1 predetermined gamut change operators as defined above, we select the LUT corresponding to the gamut change operator of the series that provides, within this image, a minimum color distance between the amount of modification A provided by this LUT when applied to all colors of this image and the required amount of modification RA of the color saturation of this image by min(| RA - A|).

In a variation of the method, we choose two LUTs of the series and interpolate in a manner known per se between these LUTS . For example, when RA=0.55, we select within the series of 1 1 operators, two LUTs with amounts of modification A1 =0.5 and A2=0.6, and interpolate a final LUT using linear interpolation between these two LUTs. For instance, for a given entry value having an output value R1 B1 G1 through the LUT having an amount of modification A1 and an output value R2B2G2 through the LUT having an amount of modification A2, the red target coordinate R_final of the output value is calculated according to the equation:

RJinal = ( (RA-A1 ) R2 + (A2-RA) R1 ) / (A2-A1 ). The green and blue coordinates of the output value are interpolated accordingly. This interpolation can be multi-dimensional if requested amounts of change RA exist for more than one aspect of color and therefore more than one color change rules and therefore more than eleven color change operators and corresponding LUTs exist.

In another case where we have only two gamut change operators and only 2 LUTs, the interpolation simplifies into the following equation:

RJinal = RA R2 + (1 -RA) R1 = R1 + RA (R2-R1 ).

The above mapping method is applied image after image, such as to get a mapped video content with a color aspect (here, color saturation) which is changed to better fit the color characteristics of the target display with better color capabilities, namely such that colors as reproduced on this target display are as close as possible to colors as displayed on the reference display.

Since the LUTs corresponding to the 1 1 different gamut change operators contain complex change operators able to precisely control psycho-visual color aspects and since the color change rule and LUT interpolation is of low computation complexity, thank to the invention, a very high quality gamut expansion can be advantageously applied in real time to a video sequence. The Lab color space in which percentages A of the gamut mapping algorithm are performed is defined by the CIE in 1976. In L ^* a ^* b ^* space, a constant a-b angle is assumed to correspond to identically perceived hue. The L coordinate represents the intensity. Unfortunately, this space was shown to not well represent hues, notably in blue tones. In a variation, instead of using Lab color space to apply the gamut mapping algorithms, other color spaces can be used, for example the JCh color space defined in the CIECAM-02 standard defined by the CIE in 2002. In JCh space, the h coordinate is assumed to correspond to perceived hue by the human eye and the J coordinate is assumed to correspond to perceived intensity. JCh space was shown to better represent hues and intensity than Lab. However, all these color spaces require highly computational effort for the appearance attribute calculator and for the inverse appearance attribute calculator. It is to be understood that the invention may be implemented in various forms of hardware, software, firmware, special purpose processors, or combinations thereof. The invention may be notably implemented as a combination of hardware and software. Moreover, the software may be implemented as an application program tangibly embodied on a program storage unit. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units ("CPU"), a random access memory ("RAM"), and input/output ("I/O") interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit.

While the present invention is described with respect to a particular example and a preferred embodiment, it is understood that the present invention is not limited to this examples and embodiment.