TECHNICAL FIELD

The present principles relate generally to the field of high dynamic range imaging and expanding the dynamic range of low dynamic range content to prepare such content for display devices having notably high peak luminance.

BACKGROUND

Recent advancements in display technology are beginning to allow an extended range of chrominance, luminance and contrast values to be displayed. Technologies allowing for extensions in the range for luminance or brightness in image content are known as high dynamic range imaging, often shortened to HDR. HDR technologies focus on capturing, processing and displaying content of a wider dynamic range.

Although a number of HDR display devices have appeared, and cameras capable of capturing images with an increased dynamic range are being developed, there is still very limited HDR content available. While recent developments promise native capture of HDR content in the near future, they do not address existing content.

To prepare conventional, herein referred to as LDR for low dynamic range, content for HDR display devices, reverse or inverse tone mapping operators (ITMO) can be employed. Such methods process the luminance information of colored areas in the image content with the aim of recovering or recreating the appearance of the original scene. Typically, ITMOs take a conventional (LDR) image as input, expand the luminance range of the colored areas of the image in a global manner, and subsequently process highlights or bright regions locally to enhance the HDR appearance of colors in the image.

Although several ITMO solutions exist, they focus at perceptually reproducing the appearance of the original scene and rely on strict assumptions about the content. Additionally, most expansion methods proposed in the literature are optimized towards extreme increases in dynamic range. Typically, HDR imaging is defined by an extension in dynamic range between dark and bright values of luminance in colored areas combined with an increase in the number of quantization steps. To achieve more extreme increases in dynamic range, many methods combine a global expansion with local processing steps that enhance the appearance of highlights and other bright regions of images. Known global expansion steps proposed in the literature vary from inverse sigmoid, to linear or piecewise linear.

To enhance bright local features in an image, it is known to create a luminance expansion map, wherein each pixel of the image is associated with an expansion value to apply to the luminance of this pixel. In the simplest case, clipped regions in the image can be detected and then expanded using a steeper expansion curve, however such a solution does not offer sufficient control over the appearance of the image. SUMMARY

These and other drawbacks and disadvantages of the prior art are addressed by various described embodiments, which are directed to methods and apparatus for expanding the dynamic range of low dynamic range content to prepare such content for display devices having notably high peak luminance.

According to one general aspect, a method is provided for expanding the dynamic range of low dynamic range content to prepare such content for display devices having notably high peak luminance. The method comprises low pass filtering a luminance component of an image, and expanding a value range for at least one component in the image in order to reproduce the image on a display device having a lower peak luminance, by using the low pass filtered luminance component.

According to another general aspect, an apparatus is provided for expanding the dynamic range of low dynamic range content to prepare such content for display devices having notably high peak luminance. The apparatus comprises a low pass filter operating on at least one component in an image and, a display device having a lower peak luminance value than the at least one image component prior to the low pass filtering, thereby expanding a value range of the at least one component in the image having a lower peak luminance. According to another general aspect, a second method is provided for expanding a value range for at least one component in the image in order to reproduce said image on a display device having a lower peak luminance. The method comprises scaling a low pass filtered luminance component exponentially using an exponent map, and weighting the scaled low pass filtered luminance component by a luminance component of the image scaled exponentially by a function of the luminance component.

According to another general aspect, a third method is provided for expanding a value range for at least one component in the image in order to reproduce said image on a display device having a lower peak luminance. The method includes the features of the first method, and further comprises generating an exponent map computed on a single pixel basis, and computing an enhancement map, wherein the enhancement map is a function of the luminance component and the low pass filtered luminance component. The aforementioned expanding comprises applying the exponent map exponentially on the low pass filtered luminance component, and scaling the expanded value range for at least one component with the enhancement map.

According to another general aspect, a second apparatus is provided for expanding a value range for at least one component in the image in order to reproduce said image on a display device having a lower peak luminance. The apparatus comprises a first lookup table storing values of a low pass filtered luminance component scaled exponentially using an exponent map. The apparatus further comprises a second lookup table storing values of the luminance component scaled exponentially by a scaling parameter. The apparatus further comprises a first multiplier that combines output of the first and second lookup tables, a third lookup table storing an exponent map to be applied to the low pass filtered luminance component, and a second multiplier that combines chrominance components with the output of the third lookup table. The apparatus further comprises a third multiplier that combines the luminance component with the output of the first multiplier to generate an expanded luminance component. The apparatus further comprises fourth and fifth multipliers operating on each of the chrominance components that combine output of the first multiplier with output of the second multiplier to generate expanded chrominance components.

According to another general aspect, a third apparatus is provided for expanding a value range for at least one component in the image in order to reproduce said image on a display device having a lower peak luminance. The apparatus comprises a first processor configured to generate an exponent map computed on a single pixel basis. The apparatus further comprises a second processor configured to compute an enhancement map, wherein the enhancement map is a function of the luminance component and the low pass filtered luminance component, and also comprises a third processor configured to expand the low pass filtered luminance component exponentially using the exponent map and scaling the result with the enhancement map.

The details of one or more implementations are set forth in the accompanying drawings and the description below. Even if described in one particular manner, it should be clear that implementations can be configured or embodied in various manners. For example, an implementation can be performed as a method, or embodied as an apparatus, such as, for example, an apparatus configured to perform a set of operations or an apparatus storing instructions for performing a set of operations, or embodied in a signal. Other aspects and features will become apparent from the following detailed description considered in conjunction with the accompanying drawings and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The present principles can be better understood in accordance with the following exemplary figures, in which:

Figure 1 shows one embodiment of an implementation of the present principles.

Figure 2 shows two embodiments of an implementation of the processing of a luminance and two chrominance components.

Figure 3 is a revised version of the embodiment of Figure 2.

Figure 4 shows an embodiment of a method under the present principles.

Figure 5 shows a second embodiment of a method under the present principles. Figure 6 shows an embodiment of an apparatus under the present principles.

Figure 7 shows a second embodiment of an apparatus under the present principles.

DETAILED DESCRIPTION

As mentioned, to enhance bright local features in an image, it is known to create a luminance expansion map, wherein each pixel of the image is associated with an expansion value to apply to the luminance of this pixel. In the simplest case, clipped regions in the image can be detected and then expanded using a steeper expansion curve; however such a solution does not offer sufficient control over the appearance of the image.

A more controllable luminance expansion solution is given in the disclosure 'Scene Adaptive Inverse Tone Mapping for LDR to HDR conversion". One of the inventors of that disclosure is also one of the present inventors. The solution described in that disclosure is the closest to the principles described herein, as the expansion function proposed in both cases is exponential.

However, in the present case, the exponential expansion function is applied to a low-pass filtered version of the image luminance, rather than the luminance itself. The expansion is then a function of one variable (namely Y _exp = f (Xi _ow )) instead of two variables. Because of this difference, the current invention may be implemented using small size look-up tables, which makes it more suitable for hardware applications.

The described embodiments are directed to methods and apparatus for expanding the dynamic range of low dynamic range content to prepare such content for display devices having notably high peak luminance.

The aim of the present methods is to expand the luminance channel Y of an image /, obtaining an expanded luminance Y _exp such that noise and other artifacts are not amplified, and such that the expansion can be approximated through look-up table (LUT) operations, making it suitable for hardware implementations. The luminance channel Y can be obtained by transforming the image to a color space separating chrominance from luminance, such as the YUV color space, for example. To obtain the expanded luminance Y _exp , a low-pass filtered version of the luminance Y is first computed, denoted as Y _low . Using Y _low , a per-pixel exponent map E can be computed. Additionally, an enhancement map Y _en hance ^can be obtained as a function of both the luminance Y and the low-pass version Y _low . This enhancement map Y _en hance encodes grain and other details in the image and can be used to recover and enhance the visibility of such details after expansion. The final expanded luminance Y _exp is obtained by applying the per pixel exponent E to the low-pass version Y _low and finally multiplying by Y _enha nce-

To reconstruct the expanded color image I _exp , the chromatic channels of the luminance-chrominance representation of / (e.g. U and V) are scaled according to the relation between Y and Y _exp , obtaining accordingly U _exp and V _exp and then the inverse color space transform is applied, to convert from the luminance-chrominance representation back to the original color space of the image (for example, RGB).

Because the exponent map E is computed from Y _low and applied to Y _low , the expansion may be approximated through LUTs that simplify processing in hardware applications. This is described as an alternative embodiment below.

To compute the low-pass version of luminance, in one embodiment, a 2- dimensional Gaussian filter is used to compute the low-pass version of the luminance, which is denoted as y _iow . The standard deviation σ of the Gaussian filter can be set for example to a value of 2, while the kernel size is set to a small value, for example, 3x3, 3x5, 5x5 etc.

In another embodiment, the low-pass version of the luminance may be computed using a separable Gaussian filter, whereby, for example, the horizontal dimension of the luminance Y is filtered first with a 1 D Gaussian filter, and its output is then filtered in the vertical dimension with another 1 D Gaussian filter. Or, the order could be reversed so that vertical filtering is done first followed by horizontal filtering.

To compute the exponent map E, in one embodiment, for a given pixel i follows the same process as in the disclosure 'Scene Adaptive Inverse Tone Mapping for LDR to HDR conversion". It is given by the following equation:

E = a(Y _low ) ² + bY _low + c (1) where the parameters a, b and c are computed depending on the maximum desired output luminance value of the expanded image, which we denote by L _mai . For example, L _max can be set to a value between 500 and 2000.

In particular, for this embodiment, for each pixel p of the image having its luminance value Y(p), an intermediate pixel expansion exponent value E(p) is obtained, by low pass filtering luminance values of pixels in a spatial neighborhood of the pixel p, and optionally also in a neighborhood of the luminance value Y(p). This low pass filtering step preferably uses Gaussian functions. E(p) is for instance obtained through the following equation :

where f _s is first a Gaussian function applied on the spatial domain of the image, and f _r a second Gaussian function applied on the luminance range domain, where Ω is the size of a window of the image centered at the pixel p, where p, is a pixel in this window. The window size can be, for example, 5 or 7. Smaller values of window size are preferred for computational efficiency.

In this embodiment of the method according to the invention, the low pass filtering is then bilateral. The word "bilateral" refers to the fact that the filtering is performed here both in spatial and luminance range domains.

Preferably, the value of the standard deviation o _s of the first Gaussian function f _s is superior or equal to 2.

The value of the standard deviation o _r of the second Gaussian function f _r should be preferably high enough to smooth texture and noise in the original image, but low enough to avoid crossing edges between objects of this image. The value of the standard deviation o _r is chosen preferably between 0.1 max(Y) and 0.5 max(Y), where max(Y) is the maximum luminance value over all pixels of the original image.

In a specific implementation of this embodiment, the standard deviation for the spatial Gaussian function f _s is set at o _s = 3, and the standard deviation for the luminance range Gaussian function f _r is set at σ _Γ = 0.3 x max(Y).

All intermediate pixel expansion exponent values that are obtained through this low pass filtering form an intermediate expansion exponent map

E(P). To compute the enhancement map Yenhance. the following equation is used:

Yenhance

r low' ) (2) where the parameter d can be set for instance to d = 1.5. Y _enhance in this case is a function of both the image luminance Y and the low-pass version Y _low .

As the enhancement map is used to recover details lost after the low-pass filtering, the parameter d is closely related to the standard deviation σ of the Gaussian filter.

The expanded luminance Y _exp is obtained as follows:

^exp ~ (¾ ) X ^enhance ?)

By replacing Y _enha nce in the above with the right-hand side of equation (2), we can also reformulate equation (3) as:

Yexp = (Xiow) ^E x ( ^r lo-w-f' (4)

or:

Yexp = (Ylo _W ^iE - ^d) X y ^(d_1) X Y (5)

Note that equations (3), (4) and (5) are mathematically equivalent.

Figure 1 shows one embodiment of an implementation of the present principles.

After obtaining the expanded luminance Y _exp following equations (3-4-5), the chromatic channels of the image are also scaled accordingly. In one embodiment, where the luminance-chrominance representation used is the YUV color space, the input chromatic channels are given by U and V. Their scaled counterparts, denoted U _exp and V _exp are obtained as follows:

Uexp = U (ψ) (6)

V _exp = V ( ) (7)

Similar to the earlier disclosure, the chromatic channels can be additionally scaled according to the exponent map E to enhance the saturation of the image. In this case, the expanded chromatic channels U _exp and V _exp are computed as follows instead of using equations (6-7): U, = u (8)

exp = V (9)

The chromatic channels U _exp and V _exp are finally recombined with the expanded luminance channel Y _exp to form the final image. Figure 2(a) shows an embodiment of an implementation of the processing of a luminance and two chrominance components. Figure 2(b) shows how reversing the order of multipliers in the chroma channel can actually save one multiplier because a single multiplier is used to form (Y _exp /Y)E which is then multiplied by each chroma channel with two additional multipliers.

In another implementation, the expanded chromatic channels U _exp and V _exp are computed as follows instead of using equations (8-9):

Figure 3(a) is a revised version of Figure 2, updated to follow this implementation. Figure 3(b) shows how reversing the order of multipliers in the chroma channel can actually save one multiplier because a single multiplier is used to form ( Yexp/Y)E which is then multiplied by each chroma channel with two additional multipliers.

In Equation (4), the parameter d can be set to a constant, e.g. 1.5 like in

Equation (2), or can vary depending on the value of Y _low : d— d _max — (d _max — dp _j i _jj ) ^ ^— d(Yi _ow ) (12) where in one implementation d _mln = 1.3 and d _max = 1.7. That means that, in equation (4), the details applied on _iow and on Y depend on Y _low .

In Equation (4), the parameter d can also vary depending on the value of

Y:

d = d ^■ .max max dmin) X = d (Y) (13)

where in one implementation d _min = 1.3 and d _max = 1.7. That means that, in equation (4), the details applied on _iow and on Y depend on Y.

We can also reformulate equation (3) as:

Which can be rewritten as:

Yexp = (Ylo _W ) ^E Ylo _W - ^d{Yl0w) (1 )

Which is the same as:

Yexp = (Ylo _W ) ^iE - ^diYlow) X Y ^d(Y) (15)

where the function dQ can follow equation (12). In one implementation d _min = 1.3 and d _max = 1.7. That means that in equation (15), the details applied on Y _low depend on Y _low , and the details applied on Y depend on Y.

Note that in all the above equations, d can vary in a non-linear way instead of varying linearly.

In low cost hardware applications, complex computations (as power functions are) are very often replaced by Look Up Tables (LUTs). In one embodiment of the present invention dedicated to hardware, the three function blocks {Y ι _ον ,) ^{Ε~ά~ ι} ^ Y ^d and E described in Figure 2 are replaced by look-up tables. As memory size is also an important criterion in the choice of a hardware architecture, the process described above is particularly well-adapted to hardware applications as the expansion of the luminance is a function of one unique variable, namely F _iow . This reduces dramatically the memory size of a LUT (256 entries in LUT if Y _low is 8 bits data) compared to a function of two variables Yexp = Y ^E{Ylow) (64k entries in LUT if both F _iow and Y are 8 bits data). The LUT size can be reduced if an interpolation method is used, however, more calculations are needed in that case.

Among the three different implementations proposed when the detail function dQ is not a constant, the third one is particularly adapted to hardware applications as it minimizes the memory size by using functions of a unique variable: a) (7iow) ^{(£ ' ~d(yiow))} which depends uniquely on Y _low

b) Y ^{d x} ^ which depends uniquely on Y As shown in a prior disclosure by one of the present inventors, only three LUTs and five multipliers are needed to implement the expansion and details recovery / enhancement once the filtering is done. Note that the filtering step itself is something known in hardware implementations.

a) The first LUT provides (Yi _ow ) ^{(E ~d(} - ^Yi ° when addressed by Y _low b) The second one provides K ^{(d( )_1)} when addressed by Y

c) The third one provides E when addressed by Y _low

Among the advantages of the present approach relative to prior methods, this invention has been optimized to ease the implementation in HW. It can use a 2D separable symmetrical filter with a reasonable aperture, a few look-up tables and a few multiplications.

The main enabler to ease the HW implementation is that the dynamic expansion is applied on the "filtered luminance" and not on the luminance itself, while keeping the same expansion factor, a filtered luminance signal. Thus instead of having a function of two different variables (luminance and filtered luminance) for the expansion, this invention uses only one variable (filtered luminance).

In order to keep the same quality after the expansion, the sharpening of details has also been modified to recover details on the expanded filtered luminance. This modification has also made it possible to lower the number of operations.

Figure 4 shows one embodiment of a method 400 for implementing the present principles. The method begins with Start block 401 and proceeds to a block 410 for low pass filtering of a luminance component of an image. Control proceeds from block 410 to block 420 for expanding a component value range of at least one image component in order to reproduce the image on a display device having a lower peak luminance by using the low pass filtered luminance to produce an expanded image component value.

Figure 5 shows one embodiment of a method 500 for expanding at least one component of an image. The method commences at Start block 501 and proceeds to block 510 for low pass filtering a luminance component. The method also proceeds to block 520 for generating an exponent map, computed on a per pixel basis. Control proceeds from block 510 for low pass filtering to block 530 for generating an enhancement map, wherein the enhancement map is a function of the luminance component and the low pass filtered luminance component. Control also proceeds from block 510 to block 540 for expanding an image component value range, by applying the exponent map exponentially on the low pass filtered luminance component from block 520. From block 540, control proceeds to block 550 for scaling the expanded image component value range using the enhancement map generated from block 530.

Figure 6 shows an embodiment of an apparatus 600 for reproducing an image on a display having a lower peak luminance value than the image. The apparatus comprises a low pass filter 610, operating on an image luminance component. The output of low pass filter 610 is in signal connectivity with an input of Processor 620. Processor 620 expands a value range of at least one component of the image. The output of Processor 620 is in signal connectivity with the input of Display 630. Display 630 displays the image with a component having expanded value range.

Figure 7 shows an embodiment of an apparatus 700 for expanding a value range of at least one component in an image in order to reproduce the image on a display device having a lower peak luminance. The apparatus comprises a low pass filter (not shown) and a first look up table (LUT 1 ) 710. The input to LUT 710 is a low pass filtered luminance component from the output of low pass filter (not shown). LUT 710 stores values of a low pass filtered luminance component scaled exponentially using an exponent map. The output of LUT 710 is in signal connectivity with one input of a first multiplier 740. The other input to multiplier 740 comes from the output of a second look up table (LUT 2) 720. The input to LUT 720 is the luminance component of an image. LUT 720 stores values of this luminance component scaled exponentially by a scaling parameter. The output of multiplier 740 is sent to a first input of another multiplier 760. The second input of multiplier 760 is the luminance component of the image. The output of multiplier 760 is the expanded values of the luminance component of an image.

Also comprising apparatus 700, a third look up table (LUT 3) 730 takes the low pass filtered luminance component as its input. LUT 730 stores an exponent map to be applied to the low pass filtered luminance component. The output of LUT 730 is sent to one input of multiplier 750. The second input of multiplier 750 are the chrominance components of the image. The chrominance components can be multiplexed in time so that only one multiplier is needed to apply the exponent map to both chrominance components. The output of multiplier 750 is then sent to multiplier 770 and multiplier 780, which operate on the respective chrominance components and apply the output of multiplier 740 to each chrominance component. The output of multiplier 750 can be alternate chrominance components, so that multipliers 770 and 780 are alternately receiving their respective inputs. The output of multipliers 770 and 780 are the expanded chrominance components, U _exp and V _exp , for example.

The present description illustrates the present principles. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the present principles and are thereby included within the present principles.

All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the present principles and the concepts contributed by the inventor(s) to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions.

Moreover, all statements herein reciting principles, aspects, and embodiments of the present principles, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein represent conceptual views of illustrative circuitry embodying the present principles. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudocode, and the like represent various processes which can be substantially represented in computer readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

The functions of the various elements shown in the figures can be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions can be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which can be shared. Moreover, explicit use of the term "processor" or "controller" should not be construed to refer exclusively to hardware capable of executing software, and can implicitly include, without limitation, digital signal processor ("DSP") hardware, read-only memory ("ROM") for storing software, random access memory ("RAM"), and non-volatile storage.

Other hardware, conventional and/or custom, can also be included. Similarly, any switches shown in the figures are conceptual only. Their function can be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.

In the claims hereof, any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a) a combination of circuit elements that performs that function or b) software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to perform the function. The present principles as defined by such claims reside in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which the claims call for. It is thus regarded that any means that can provide those functionalities are equivalent to those shown herein.

Reference in the specification to "one embodiment" or "an embodiment" of the present principles, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment of the present principles. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment", as well any other variations, appearing in various places throughout the specification are not necessarily all referring to the same embodiment.

It is to be appreciated that the use of any of the following ", "and/or", and "at least one of", for example, in the cases of "A/B", "A and/or B" and "at least one of A and B", is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of "A, B, and/or C" and "at least one of A, B, and C", such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This can be extended, as readily apparent by one of ordinary skill in this and related arts, for as many items listed.

These and other features and advantages of the present principles can be readily ascertained by one of ordinary skill in the pertinent art based on the teachings herein. It is to be understood that the teachings of the present principles can be implemented in various forms of hardware, software, firmware, special purpose processors, or combinations thereof.

Most preferably, the teachings of the present principles are implemented as a combination of hardware and software. Moreover, the software can be implemented as an application program tangibly embodied on a program storage unit. The application program can be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units ("CPU"), a random access memory ("RAM"), and input/output ("I/O") interfaces. The computer platform can also include an operating system and microinstruction code. The various processes and functions described herein can be either part of the microinstruction code or part of the application program, or any combination thereof, which can be executed by a CPU. In addition, various other peripheral units can be connected to the computer platform such as an additional data storage unit and a printing unit.

It is to be further understood that, because some of the constituent system components and methods depicted in the accompanying drawings are preferably implemented in software, the actual connections between the system components or the process function blocks can differ depending upon the manner in which the present principles are programmed. Given the teachings herein, one of ordinary skill in the pertinent art will be able to contemplate these and similar implementations or configurations of the present principles.

Although the illustrative embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the present principles are not limited to those precise embodiments, and that various changes and modifications can be effected therein by one of ordinary skill in the pertinent art without departing from the scope of the present principles. All such changes and modifications are intended to be included within the scope of the present principles as set forth in the appended claims.