Login| Sign Up| Help| Contact|

Patent Searching and Data


Title:
DEVICE AND METHOD FOR CONTROLLING OUTPUT LEVELS OF CONTENT
Document Type and Number:
WIPO Patent Application WO/2021/009104
Kind Code:
A1
Abstract:
In a processing device (110) a content signal, a signal characteristic is measured (S202), an average is calculated (S204) for it and the difference between them is calculated (S206) to obtain an input to which a discomfort function is applied (S208) to determine a user discomfort signal that is used to modify (S210) the signal characteristic of the content signal, whereafter the modified content signal is output (S212). The processing device (110) can be content consumption device, but the general principles can also be applied during content preproduction.

Inventors:
STAUDER JURGEN (FR)
REINHARD ERIK (FR)
MORVAN PATRICK (FR)
BLONDE LAURENT (FR)
Application Number:
PCT/EP2020/069719
Publication Date:
January 21, 2021
Filing Date:
July 13, 2020
Export Citation:
Click for automatic bibliography generation   Help
Assignee:
INTERDIGITAL CE PATENT HOLDINGS SAS (FR)
International Classes:
H04N21/422; G06K9/46; G06N3/063; H04N9/68; H04N17/04; H04N21/439; H04N21/44; H04N21/442
Domestic Patent References:
WO2017086950A12017-05-26
Foreign References:
JP2014153834A2014-08-25
JPH11234583A1999-08-27
US20100164975A12010-07-01
EP2658264A12013-10-30
EP3352467A12018-07-25
US20110175925A12011-07-21
Attorney, Agent or Firm:
STAHL, Niclas et al. (FR)
Download PDF:
Claims:
CLAIMS

1. A method in a device for processing content, the method comprising in at least one hardware processor of the device:

measuring a signal characteristic of an input signal corresponding to the content to obtain a measurement;

obtaining a value representative of discomfort by application of a discomfort function to a difference between the measurement and an average for the measurements of the signal characteristic;

modifying the signal characteristic of the input signal based on the value representative of discomfort resulting in a modified signal; and

outputting the modified signal.

2. The method of claim 1 , wherein the modified signal is output for rendering on a renderer.

3. The method of claim 2, wherein the renderer is comprised in the device.

4. The method of claim 1 , wherein the at least one hardware processor outputs the modified signal to a storage unit or a broadcasting unit for provision to an end user.

5. The method of claim 1 , wherein the input signal comprises at least one an image signal, a video signal, an audio signal and a haptic signal.

6. The method of claim 1 , wherein the average is a spatial average.

7. The method of claim 1 , wherein the average is a temporal average.

8. The method of claim 1 , wherein the discomfort function includes a threshold operator.

9. The method of claim 1 , wherein the modifying comprises attenuating the signal characteristic.

10. A device for processing content comprising:

an input interface configured to receive an input signal representative of a content; and at least one hardware processor configured to:

measure a signal characteristic of an input signal corresponding to the content to obtain a measurement;

obtain a value representative of discomfort by application of a discomfort function to a difference between the measurement and an average for the measurements of the signal characteristic;

modify the signal characteristic of the input signal based on the value representative of discomfort resulting in a modified signal; and

output the modified signal.

1 1. The device of claim 10, wherein the at least one hardware processor is configured to output the modified signal is output for rendering on a renderer.

12. The device of claim 10, wherein the input signal comprises at least one an image signal, a video signal, an audio signal and a haptic signal.

13. The device of claim 10, wherein the at least one hardware processor is configured to modify the signal characteristic by attenuating the signal characteristic.

14. The device of claim 10, wherein the discomfort function further takes as input a threshold value in the device, the threshold value configurable by a user.

15. A non-transitory computer readable medium storing program code instructions that, when executed by a processor, implement the steps of a method according to at least one of claims 1 to 9.

Description:
DEVICE AND METHOD FOR CONTROLLING OUTPUT LEVELS OF CONTENT

TECHNICAL FIELD

The present disclosure relates generally to management of output levels of content for rendering to a user. BACKGROUND

This section is intended to introduce the reader to various aspects of art, which may be related to various aspects of the present disclosure that are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

Signals that are rendered to humans include image signals, video signals, audio signals and haptic signals. For example, video signals are watched by human observers on display devices, while audio signals are listened to using loudspeakers. In response to being exposed to such signals, users can experience comfort and discomfort that depend on the signal.

A well-known hypothesis is that discomfort in general arises from signal presentations that are far removed from the current level of adaptation of the human visual system, or components thereof, including individual cells. For example, photoreceptors (rods and cones) can be thought of as adapting to the amount of light they receive as input. If, up to a given moment, this amount has been small, but suddenly increases significantly, then a given photoreceptor is not well adapted to the new quantity, and can therefore signal an overload condition to cell layers downstream in the visual pathway. The cell is then said to be maladapted. During adaptation to the new level, the photoreceptors output slowly normalizes and it will once again be able to produce signals consistent with variations in its input. If, during adaptation, the difference between current adaptation level and the current signal level is sufficiently large, a sense of discomfort can arise, especially if enough photoreceptors are signalling an overload condition. It is well-known that this adaptation process is not unique to photoreceptors, but occurs for all neurons in the human brain.

The fifth layer of neuronal cells in the retina, retinal ganglion cells, aggregates the information received from the fourth layer, amacrine cells, which in turn receives information from the third layer, bipolar cells, and the second layer, half cells, and indirectly from photoreceptors in the first layer. A subclass of retinal ganglion cells is formed by ipRCGs (intrinsically photosensitive retinal ganglion cells). These cells are directly sensitive to light, but they also receive neuronal input from preceding layers of cells. They are implicated in a variety of non-image forming and image forming tasks, including regulation of the circadian clock, regulation of pupil size, regulation of sleep and arousal, melatonin production and dopamine production.

The class of ipRCGs can be subdivided into 5 subclasses, labelled M1 to M5. In particular the M1 ipRCGs are thought to be active in the spatially varying detection of absolute luminance. While each individual M1 cells has a limited dynamic range, different M1 cells are tuned to different ranges of light intensity. As a population, they are therefore able to sense absolute light levels. However, because they also receive neuronal input, including spatial center-surround signals, it can be hypothesized that M1 ipRCGs are sensitive to spatio-temporal contrast as well as absolute light levels. This means that it should be possible for such a cell to be driven into a state of maladaptation simultaneously in multiple different ways. For example, extreme spatial contrast may be present at a light level near the edge of its dynamic range for direct light sensitivity. As an alternative example, at a given moment, the centre-surround contrast signaled to its neuronal inputs may change dramatically.

In turn, the state of maladaptation of a given neuron in the visual pathway may be an indicator for the potential for discomfort.

As an example, an audio signal that is too loud typically lowers comfort for the human listening to it. At the same time, an audio signal that is too low and may also lower the comfort for the listener owing to the necessity for increased attention to be able to follow the transmission. Another source of discomfort result from too much glare in a video signal, for example caused by light scattering within the display device or light scattering in the ocular media of the human eye. Detection of contrast in the video signal can be limited by such glare. Another source of discomfort can be caused by the presence of too much blue light, especially when emitted by LED light sources as this can result in eye strain.

Some discomfort situations, such as the audio examples above, can be detected using well-known thresholding on the audio signal. However, such conventional thresholding cannot detect more complex discomfort situations such as blackouts, noise, glare or human adaptation.

It will thus be appreciated that there is a desire for a solution that addresses at least some of the shortcomings controlling output levels for content. The present principles provide such a solution.

SUMMARY OF DISCLOSURE

In a first aspect, the present principles are directed to a method in a device for processing content. At least one hardware processor of the device measures a signal characteristic of an input signal corresponding to the content to obtain a measurement, obtains a value representative of discomfort by application of a discomfort function to a difference between the measurement and an average for the measurements of the signal characteristic, modifies the signal characteristic of the input signal based on the value representative of discomfort resulting in a modified signal, and outputs the modified signal.

In a second aspect, the present principles are directed to a device for processing content comprising an input interface configured to receive an input signal representative of a content and at least one hardware processor configured to measure a signal characteristic of an input signal corresponding to the content to obtain a measurement, obtain a value representative of discomfort by application of a discomfort function to a difference between the measurement and an average for the measurements of the signal characteristic, modify the signal characteristic of the input signal based on the value representative of discomfort resulting in a modified signal, and output the modified signal.

In a third aspect, the present principles are directed to a computer program product which is stored on a non-transitory computer readable medium and includes program code instructions executable by a processor for implementing the steps of a method according to any embodiment of the second aspect.

BRIEF DESCRIPTION OF DRAWINGS

Features of the present principles will now be described, by way of non-limiting example, with reference to the accompanying drawings, in which:

Figure 1 illustrates a system according to a first embodiment of the present principles; Figure 2 illustrates a method according to a first embodiment of the present principles; Figure 3 illustrates a system according to a second embodiment of the present principles; and

Figure 4 illustrates a method according to a second embodiment of the present principles.

DESCRIPTION OF EMBODIMENTS

Figure 1 illustrates a system 100 according to a first embodiment of the present principles. The system 100 includes a presentation device 1 10 and a content source 120; also illustrated is a non-transitory computer-readable medium 130 that stores program code instructions that, when executed by a processor, implement steps of a method according to the present principles. The system can further include a rendering device 140.

The presentation device 1 10 includes at least one input interface 1 1 1 configured to receive content from at least one content source 120, for example a broadcaster, a storage device, an OTT provider and a video server on the Internet. It will be understood that the at least one input interface 1 1 1 can take any suitable form depending on the content source 120; for example a cable interface or a wired or wireless radio interface (for example configure for Wi-Fi or 5G communication).

The presentation device 1 10 further includes at least one hardware processor 1 12 (“processor”) configured to, among other things, control the presentation device 1 10, process received content for display and execute program code instructions to perform the methods of the present principles. The presentation device 1 10 also includes memory 1 13 configured to store the program code instructions, execution parameters, received content - as received and processed - and so on. The presentation device 1 10 can further include a rendering interface 1 14 configured to output processed content to an external rendering device 140 and/or an internal rendering unit 1 15 for rendering processed content. The rendering interface 1 14, the rendering units 1 15, and the rendering device 140 can be configured to render one or more of for example images, video, sounds and haptics. In addition, the presentation device 1 10 typically includes a control interface (not shown) configured to receive instructions, directly or indirectly (such as via a remote control) from a user. The control interface can be used to select, with the aid of a user interface, different settings for the presentation device 1 10. The settings can for example be for the“discomfort function” (to be described) and determine a threshold operator to use and also associated threshold values. In other words, this enables the user to personalise the discomfort levels, such as lowering the threshold if the user is comparatively sensitive to a certain stimulus and increase the threshold in case the user is comparatively insensitive to the stimulus.

In an embodiment, the presentation device 1 10 is configured to receive a plurality of content items simultaneously, for example as a plurality of broadcast channels.

The presentation device 1 10 can for example be embodied as a television, a set-top box, a decoder, a smartphone or a tablet.

Figure 2 illustrates a method 200 of the first embodiment of the present principles. The method 200 can be performed by the processor 1 12.

In step S202, at least one signal characteristic of a received input signal is measured to obtain at least one measurement.

There are various conventional methods to measure a characteristic of a signal, usually depending on the characteristic to be measured. As a first example, one characteristic of an image signal is the brightness perceived by a human observer. Brightness is the sensation of absolute intensity of light. One conventional way of measuring brightness is to calculate the mean luminance of an image signal.

As a second example, one characteristic of a video signal is the potential glare that will be generated in the display device or in the human eye. Potential glare can be measured by analysis of the luminance of the video signal.

A third example of a characteristic is blue light emitted from a display showing an image based on an image signal. It is well-known that blue light can be calculated from the image signal, for example from the blue channel of the image signal.

A fourth example of a characteristic is shrill, loud sounds that can be measured using thresholding techniques, possibly in a given frequency range.

A fifth example is the combined measurement of multiple signal characteristics. For example, spatial and temporal characteristics can each contribute to a sense of discomfort, independently or in tandem. Video-related examples include a bright spot on a dark background, the sudden emergence of a bright object in an otherwise dark video, an overall high luminance, high contrast.

It is noted that in case of more than one signal characteristic, these can be treated separately afterwards to, as will be described, control the output of these signal characteristics independently of one another. This can be the case if one characteristic relates to sound and another to image, but also if more than one are related to, for example, video to control blue light and high contrast separately.

In step S204, an average is calculated, preferably for each of the at least one measurement. The average calculation may be temporal for video and audio signals or spatial for image signals. In case of temporal calculation, a sliding window can be used so that the average is calculated over the most recent, e.g. a few seconds worth of measurements.

For example, the average calculation can be formulated so that the adaptation state of a human consumer of the signal is represented. For a video signal, the video signal controls a display that generates light to be watched by a human observer. One perception of the human observer is the brightness of the image signal. If the measured signal characteristics is brightness, the average calculation can be formulated such the resulting average represents the adaptation stage of the human observer to brightness.

The average can for example be calculated as the average frame average light level corresponding to the average frame light level for the most recent L frames (or equivalently K seconds).

In step S206, a difference is calculated between the measurements and and their corresponding average.

As an example, for a video signal with brightness as the measured image signal characteristic and the calculated average representing the human adaptation state to brightness, the difference between current adaptation level and current signal brightness level has a great impact on the comfort of the observer. If the difference is sufficiently large, a sense of discomfort can arise, especially if enough photoreceptors are signaling an overload condition.

In step S208, the calculated difference is processed by a“discomfort function” that for example can be a threshold operator that indicates discomfort if the absolute difference is larger than a given threshold. In step S210, one or more characteristics for which the discomfort function indicates discomfort can be modified in the input signal. For example, if a sound is determined to be too shrill, its volume can be lowered, possibly in a given frequency interval.

In step S212, the input signal, possibly modified in step S210, is output through the rendering interface 1 14, to one or more rendering units 1 15, 140.

In a first illustrative example, the input signal is a video signal and the characteristic of the video signal is the brightness perceived by a human observer. As is well known, brightness is the sensation of absolute intensity of light. The brightness is measured in a straight-forward well-known way by calculating the mean luminance for each frame of the video signal.

In the example, the use of the present method allows calculation of a discomfort signal that represents the state of mal-adaptation of a given neuron in the visual pathway of the human observer of an image shown by a display device that is controlled by a video signal. The discomfort signal is adapted to indicate the potential for discomfort.

Synaptic connections between neurons can be either additive or multiplicative (respectively subtractive or divisive). This suggests that the state of mal-adaptation can be modeled either as the ratio between the current input and the current state of adaptation, or as the linear difference between the current input and the current state of adaptation.

If ratio modelling is used, the difference calculation includes the calculation of the difference between the logarithm of the signal characteristic and the logarithm of the average.

If linear difference modelling is used, the difference calculation includes the calculation of the arithmetic difference between the signal characteristic and the average.

In the first example, the difference calculation and the discomfort function are based on the visual processing of ipRCGs cells outlined above as well as the hypothesis that discomfort arises from signals that are far removed from the current level of adaptation of the human visual system, or components thereof, including individual cells. The idea is a model for determining the discomfort signal in a sense that if the signal characteristic deviates from the average signal characteristic by an amount that is greater than a (possibly signal characteristic-dependent) threshold, then the discomfort signal is large and discomfort arises.

As mentioned, in the first example, the signal characteristic is the brightness of the video signal, but it will be appreciated that other characteristics of the video signal and indeed characteristics of other type of signals can be processed in the same or in a similar way.

In the following, it is assumed that L(x, t) represents a signal characteristic at time t, and represents the spatial average at time t. Further, L(x) represents a

temporal average at location x, and represents the spatio-temporal average of a signal.

In its simplest form, a measure of discomfort, such as may be experience by photoreceptors, is then the difference between input and adaptation level, where the adaptation level may be represented by a temporal average resulting from

average calculation. The difference calculation then results in the following difference:

In the simplest case, a unity discomfort function can be applied to the difference leading to the following discomfort signal:

DS(x) = D(x).

If current light levels are below the adaptation level, then it is possible that no discomfort is experienced, so the model may be extended using the maximum operator as the discomfort function as follows:

For certain values of DS(x), discomfort will arise. A threshold may be applied to divide the range of DS(x) into a comfort and a discomfort range. Such a threshold can for example be determined through tests on a group of viewers.

In an alternative, discomfort measures can be obtained as follows. First, the discomfort signal DS(x ) is normalised by the current input level, to arrive at a measure that incorporates Weber’s law into the discomfort function:

Second, a divisive measure of discomfort may be created using the following difference calculation:

that may be implemented as linear difference as follows:

As in previous formulations, large values of DS(x) indicate a level of discomfort, and therefore thresholding is appropriate for this case as well. The calculation of ratios can always be implemented in the difference calculation step as described using logarithmic representations.

Third, a spatio-temporal model of discomfort may be created by combining spatial and temporal measures of discomfort, for example according to:

or

This model may serve as a basic representation of M1 ipRCG behaviour.

In a second illustrative example, the discomfort signal is derived from a blue light characteristic of an image signal.

In the example, it is assumed that the signal is an image signal L(x, c) with x being the position in the image and c Î ( R , G, B ) the color channels R,G,B for red, green blue. The image characteristic is the blue light emitted by a display controlled by the image signal. Blue light is notably controlled by the blue channel of the image signal, thus the measurement of the signal characteristic can be carried out by averaging the blue channel of the image spatially over the image according to:

In the example, the average calculation is not directly responsive to the signal characteristic but is calculated from the signal directly. Assuming that a human observer is adapted to the mean luminance of the image signal, the average is calculated according to:

with w(c,x) a color channel and spatially dependent weight such that

The weights may be spatially uniform w(x, c) = w(c) and may be chosen in a well-known manner according to the specific RGB color space in which the signal is defined. This may be according to, for instance, ITU-R BT.709 or ITU-R BT.2020.

The difference calculation then processes the measured amount of blue light against the adaptation level according to D = C— A.

Assuming a non-linear, logarithmic sensitivity of the human eye to blue light, the discomfort signal can be calculated according to

Depending on the discomfort signal for blue light, the signal may be modified automatically in order to lower the discomfort by modifying the blue channel of the image signal according to:

In this example, the discomfort signal DS applies globally to the image but the image signal modification is applied locally at positions x in order to preserve the overall perception of hue.

Figure 3 illustrates a system 300 according to a second embodiment of the present principles. A main difference between system 100 in Fig. 1 and system 300 in Fig. 3 is that the former is typically used by an end-user consuming content, while the latter is typically used during post-processing of the content for subsequent consumption.

The system 300 includes a processing device 310 and a content source 320; also illustrated is a non-transitory computer-readable medium 330 that stores program code instructions that, when executed by a processor, implement steps of a method according to the present principles. The system can further include a rendering device 340.

The processing device 310 includes at least one input interface 31 1 configured to receive content from at least one content source 320, for example a storage device or a video server. It will be understood that the at least one input interface 31 1 can take any suitable form depending on the content source 320; for example a cable interface or a wired or wireless radio interface (for example configure for Wi-Fi or 5G communication).

The processing device 310 further includes at least one hardware processor 312 (“processor”) configured to, among other things, control the processing device 310, process received content for display and execute program code instructions to perform the methods of the present principles. The processing device 310 also includes memory 313 configured to store the program code instructions, execution parameters, received content - as received and processed - and so on.

The processing device 310 can further include a rendering interface 314 configured to output processed content to an external rendering device 340 and/or an internal rendering unit 315 for rendering processed content. The rendering interface 314, the rendering unit 315, and the rendering device 340 can be configured to render one or more of for example images, video, sounds and haptics.

In addition, the processing device 310 typically includes a control interface (not shown) configured to receive instructions, directly or indirectly (such as via a keyboard, mouse and/or touchpad) from a user.

The processing device 310 can for example be embodied as a computer, a workstation, a smartphone or a tablet.

Figure 4 illustrates a method 400 of the second embodiment of the present principles. The method 400 can be performed by the processor 312.

Steps S402-S408 can be the same as steps S202-S208 in Figure 2. In one variant, the discomfort signal is rendered according to well-known principals of user interfaces, in step S410, to a human operator as a discomfort meter is showing the level of the discomfort signal. For example, a horizontal bar could be shown on a screen where the length of the bar corresponds to the discomfort signal. In another example, colours could be shown: green for a low discomfort signal, yellow for a medium DS and red for high DS.

In step S412, the input signal is modified depending on the discomfort meter indicating discomfort. The modification can be made in response to input from the operator or directly by the processor in response to the discomfort signal. Upon modification, it is possible to iterate, with the modified signal replacing the most recent original signal in order to determine if the modified signal results in discomfort.

In step S414, the input signal, possibly modified in step S412, is output through the rendering interface 314 to a storage unit, a broadcasting unit or the like with the intent of providing the content to at least one end user.

It will thus be appreciated that the present principles can be used to provide content that can remove or reduce discomfort for a consumer of the content, which can prevent hurt or fatigue of the consumer. In addition, the complexity of the present principles can be made independent of the complexity of the signal characteristic that is processed.

It should be understood that the elements shown in the figures may be implemented in various forms of hardware, software or combinations thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory and input/output interfaces. The present description illustrates the principles of the present disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the disclosure and are included within its scope.

All examples and conditional language recited herein are intended for educational purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions.

Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e. , any elements developed that perform the same function, regardless of structure.

Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, and the like represent various processes which may be substantially represented in computer readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term“processor” or“controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, read only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage. Other hardware, conventional and/or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.

In the claims hereof, any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a) a combination of circuit elements that performs that function or b) software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to perform the function. The disclosure as defined by such claims resides in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which the claims call for. It is thus regarded that any means that can provide those functionalities are equivalent to those shown herein.