CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of WO International Application No.

PCT/CN2012/080632, filed August 27, 2012.

TECHNICAL FIELD

This invention relates to video quality measurement, and more particularly, to a method and apparatus for determining a video quality metric at a bitstream level. The determined video quality metric can then be used, for example, to adjust encoding parameters, or to provide required video quality at the receiver side.

BACKGROUND

Generally, the more textured or complex the video content is, the more artifacts in the video content can be tolerated by human eyes. That is, when a video content is viewed by human eyes, visual artifacts may be masked by the video content itself. This property of human eyes is known as masking property or masking effect. Thus, content complexity may be considered in visual quality assessment.

In our previous work, we estimate a content unpredictability (CU) parameter to indicate content complexity. In a commonly owned PCT application, entitled

"Method and apparatus for video quality measurement" by F. Zhang, N. Liao, K. Xie, and Z. Chen (PCT/CN1 1/002096, Attorney Docket No. PA1 10061 , hereinafter "Zhangl "), the teachings of which are specifically incorporated herein by reference, we disclosed a method for predicting video quality using a quantization parameter (QP), which is adjusted by a correction function that depends on content

unpredictability. In another commonly owned PCT application, entitled "Video quality measurement" by F. Zhang, N. Liao, K. Xie, and Z. Chen (PCT/CN201 1/082870, Attorney Docket No. PA1 10050, hereinafter "Zhang2"), the teachings of which are specifically incorporated herein by reference, we disclosed a method for estimating a compression distortion factor, a slicing distortion factor, and a freezing distortion factor using parameters (for example, quantization parameters, content

unpredictability parameters, ratios of lost blocks, ratios of propagated blocks, error concealment distances, motion vectors, durations of freezing, and frame rates) derived from a bitstream.

SUMMARY The present principles provide a method for assessing video quality of a video included in a bitstream, comprising the steps of: determining a content

unpredictability parameter responsive to prediction residuals of a block of a picture in the video; scaling the content unpredictability parameter responsive to the encoding configuration of the block; and determining a quality metric representative of video quality of the video included in the bitstream in response to the scaled content unpredictability parameter as described below. The present principles also provide an apparatus for performing these steps.

The present principles also provide a method for assessing video quality of a video included in a bitstream, comprising the steps of: determining a content unpredictability parameter responsive to prediction residuals and encoding

configuration of a block of a picture in the video, wherein the content unpredictability parameter is indicative of a degree of difficulty of predicting the block; scaling the content unpredictability parameter responsive to the encoding configuration, wherein a value of the scaled content unpredictability parameter is substantially same as a value of a second content unpredictability parameter for another block that has similar content; and determining a quality metric representative of video quality of the video included in the bitstream in response to the determined content unpredictability parameter as described below. The present principles also provide an apparatus for performing these steps.

The present principles also provide a computer readable storage medium having stored thereon instructions for assessing video quality of a video included in a bitstream, according to the methods described above.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1A is a pictorial example depicting three original pictures to be encoded as I, P, and B pictures, respectively, FIG. 1 B is a pictorial example depicting raw CU parameters corresponding to FIG. 1A, and FIG. 1 C is a pictorial example depicting aligned CU parameters, generated in accordance with an embodiment of the present principles. FIG. 2A is another pictorial example depicting another three original pictures to be encoded as I, P, and B pictures, respectively, FIG. 2B is a pictorial example depicting raw CU parameters corresponding to FIG. 2A, and FIG. 2C is a pictorial example depicting aligned CU parameters, generated in accordance with an embodiment of the present principles. FIG. 3 is a pictorial example depicting raw CU parameter distributions for I, P, and B pictures, respectively.

FIG. 4 is a flow diagram depicting an example for estimating video quality based on content complexity, in accordance with an embodiment of the present principles.

FIG. 5 is a block diagram depicting an example of a video quality

measurement apparatus that may be used with one or more implementations of the present principles.

FIG. 6 is block diagram depicting an example of a video processing system that may be used with one or more implementations of the present principles.

DETAILED DESCRIPTION

It is known that variance and entropy may be used to measure complexity of natural images. For a general signal sequence, variance quantifies contrast or energy, while entropy evaluates randomness or uncertainty. The work described in A. B. Watson, R. Borthwick, and M. Taylor, "Image quality and entropy masking," SPIE Proceedings, vol. 3016, 1997, suggests that another complexity-related characteristic apart from contrast and randomness, called learn ability, may be used to measure content complexity.

Learn ability may be considered to be similar to predictability. That is, the more likely a signal can be represented by neighborhood, the higher its predictability and learn ability are. In video coding, intra prediction and inter prediction are commonly used to remove redundancy in the video. Specifically, a current block may be predicted from a previously encoded block, and the difference between the current block and the predicted block (i.e., prediction residual), rather than the current block itself, may be encoded. In general, when intra prediction or inter prediction work well, that is, when the current block can be predicted well from a previously encoded block, the prediction residual becomes small and/or sparse. Therefore, energy of prediction residuals in video coding may be a good indicator of how difficult a video signal can be predicted (i.e., content unpredictability). In the present embodiments, we use content unpredictability to represent content complexity.

When the content predictability parameter is defined based on prediction residuals, it can be computed as parameter to measure the degree of difficulty of predicting a video signal. Because the prediction residuals are available in the compressed domain, the content predictability parameter can be measured without full decoding or accessing decoded image pixels, and thus promote the

computational efficiency of applications using CU parameters, such as QoE (quality of experience) assessment and video indexing.

Most existing video compression standards, for example, H.264 and MPEG-2, use a macroblock (MB) as the basic encoding unit. Thus, the following embodiments use a macroblock as the basic processing unit. However, the principles may be adapted to use a block at a different size, for example, an 8x8 block, a 16x8 block, a 32x32 block, or a 64x64 block.

In one embodiment, for each macroblock, the CU parameter can be computed using the variance or energy of prediction residuals, which is theoretically equal to the sum of the squared de-quantized DCT coefficients minus the squared sum of de- quantized DC coefficients. When no prediction is performed on a macroblock, the CU parameter can be computed as energy of pixels. That is, inverse DCT (IDCT) is not necessary for calculating the CU parameter. Given the QP and quantized DCT coefficients which are parsed from a bitstream, the de-quantized DCT coefficients can be estimated with sufficient precision in order to calculate CU parameters. In an example, in Zhangl , when H.264/AVC is used for encoding, the CU parameter for the r-th macroblock can be calculated as:

∑icr∑ _fcE i OC¾ - (∑ _icr ^) ² ] , (1 )

^K MB

where constant K _M B is the total number of pixels in a macroblock, i.e., 256 for H.264/AVC; 0.625 x 2 ^QPr/6 is an approximated formula of calculating a quantization step with respect to a QP value for H.264/AVC; / indexes the partitions in the r-th macroblock; | / \ _r denotes the total number of partitions therein, for example, 16 partitions for 4x4 block partitioning and 4 partitions for 8^8 block partitioning; non- negative k indexes the quantized DCT coefficients in the /-th partition of the r-th macroblock; and DCT _0,/ refers to the quantized DC coefficient in the /-th partition.

The above CU calculation can be applied to blocks (or Coding Unit/Prediction Unit in other standards, for example, HEVC) of any size and any partitions within a block. In the present application, a content unpredictability parameter calculated based on energy of prediction residuals, for example, as calculated in Eq. (1 ), is denoted as a raw CU parameter.

Prediction technique varies with encoding picture types. For example, I pictures support only intra-frame prediction and may have larger prediction residuals than P and B pictures. Due to inter-frame prediction, P pictures usually have smaller prediction residuals than I pictures. B pictures usually have smaller prediction residuals than both I and P pictures because of bi-directional inter-frame prediction. That is, different types of pictures contain prediction residuals of different numerical scales, even if the corresponding pictures may have similar contents (for example, the pictures belong to the same scene, or a similarity measure between the pictures is smaller than a threshold). Therefore, the raw CU parameters, calculated as prediction residual energy over pictures, depend on not only the video content but also the picture type, and thus may not accurately reflect the complexity of pictures.

FIG. 1A shows three exemplary original pictures to be encoded as I, P, and B pictures, respectively. Corresponding raw CU parameters are illustrated in FIG. 1 B. For visualization, CU parameter values are logarithmically mapped to pixel brightness (i.e., the brightness of a pixel in FIG. 1 B corresponds to the value of a CU parameter at a macroblock in FIG. 1 A). The brighter a pixel is, the larger the CU parameter is. While the three original pictures are very similar in image content and thus very similar in perceived content complexity, the I picture has the largest raw CU parameters and the B picture has the smallest raw CU parameters. Similar observations are also made in another example shown in FIGs.2A and 2B.

FIG. 3 illustrates exemplary distributions of macroblocks' raw CU parameter values for different picture types. The probability distribution functions are approximated by normalized histograms with a bin width of 15 (i.e., intervals of raw CU parameter values). The raw CU parameters are calculated on all the

macroblocks from I, P and B pictures, respectively, in consecutive 250 frames of a 576-line video. Overall, the raw CU parameters are the largest for I pictures, and smallest for B pictures. From I, P to B picture, the tail of the probability distribution curve gets thinner. The legend of FIG. 3lists Γ distribution parameters, which are fitted for each type of pictures separately. In this example, we observe that the shape parameters (0.033 for I pictures, 0.041 for P pictures, and 0.049 for B pictures) are quite similar, while the scale parameters (4916 for I pictures, 1086 for P pictures, and 324 for B pictures) differ a lot. From the examples shown in FIGs.1 -3, we observe that even when the perceived content complexity are very similar, the raw CU parameters may vary significantly with picture types. In order to more accurately measure content complexity, the present principles are directed to a method and apparatus for improving the estimated content unpredictability parameters taking into account the perceived variations described above. In particular, the raw CU parameters are scaled to align with CU parameters obtained in another encoding configuration such that the aligned CU parameters reflect the perceived content complexity more closely. After CU parameter alignment, the value of the aligned CU parameter of a current block in one encoding configuration is substantially the same as the value of the aligned CU parameter of another block in another encoding configuration, when the another block has similar content as the current block.

The proposed CU parameter alignment method is based on two general assumptions as discussed in the following.

Assu m ption 1 Raw CU parameter values of macroblocks conform to independent gamma distribution^ distribution). A random variable Y that is -distributed with shape parameter k and scale parameter Θ, denoted asY~r(k, 9), has a probability distribution function: fiy- ^k ' ^e = for y > O and /c, 0 > 0. (2) Assumption 1 is based on the observation that the raw CU parameter of a macroblock is defined as the variance of pixels or prediction residuals, as well as the following lemma:

Lem ma: if xi , X2, x _n are i.i.d. Ν(μ, σ ² ) random variables, their variance, x„) ² , conforms to a special case of the Gamma distribution,

Γ(^, 2σ ² ).

Strictly speaking, neither adjacent pixels nor residuals are independently Gaussian distributed, therefore we make a careful approximation that their variance conforms to a generalized distribution r(/c, 2σ ² ) with a relaxed shape parameter k, where k no longer equals (n-1 )/2 exactly. Generally, the correlation and sparsity of adjacent pixels make k much smaller than (n-1 )/2. Nevertheless, such correlation should be relatively consistent for the same visual content, despite of different encoding configurations. Therefore, we make the following assumption.

Assu m ption 2 Raw CU parameter distributions (of same or similar content), under different configurations, have a constant shape parameter k and possibly different scale parameters©.

Based on the above assumptions, the problem of CU alignment becomes estimation of scale parameters efor different configurations. Assuming there are two configurations X and Y, the estimation problem can be solved by the maximum likelihood estimator of the distribution parameters on CU training sets {x} _x and {y} _Y :

arg max _¾ e _} f( ^x 0 ^k > ^θ χ) f O ^k > ^Θ Υ) (3) where x and y denote the raw CU parameter values of the MBs under configurations X and Y, respectively, and n _x and n _Y are the total number of sets {x}x and y}y, respectively.

Subsequently, we can align the MB-wise CU for each configuration and obtain the aligned MB-wise CU (denoted as CU), for example, as:

^ _ (ευβγ/θχ if MB; is configured with X

^£ ( Uiif MB; is configured with Y ^' ' '

In the following, we provide an exemplary embodiment for solving the estimation problem as described in Eq. (3). Referring to Eqs. (2) and (3), the log- likelihood is

{0)) = (fc - 1) In xt + ^ In y _f ) - (n _x + η _γ )1ηΓ(Λ)

- (∑ Χΐ/θ _χ + ^ yi/βγ) - H¾ln6>x + η _γ \ηθ _γ ). Setting the derivative of log-likelihood with respect to θχίο zero yields:

and leads to the maximum likelihood estimator for θ _χ :

Similarly, the maximum likelihood estimator for θ _γ can be derived as:

Thus, without estimating k, we have: nY Li %i

That is, Eq. (5) provides a ratio between scale parameters efor configurations X and Y, wherein the ratio is also referred to as an alignment scaling factor, to scale the raw CU parameters in order to align them between configurations X and Y. In one example, when ITU-T P.NBAMS (Parametric Non-intrusive Bitstream Assessment of video Media Streaming quality) databases are used for training, we derive alignment scaling factors and estimate the content complexity of a video sequence as an average value of aligned CU parameters for all available MBs in all pictures of the video sequence. In this example, the content complexity is refined as a weighted combination of CUs:

Zc = 7 (∑t£{I}∑r£t _{20 6x} | _r | _t +∑tE{P}∑rEt _{3 52 x r} | _t +∑tE{B}∑rEt ^) + C, (6) where te{\},te{P} and te{B} represent I pictures, P pictures and B pictures,

respectively, CU _r is the raw content unpredictability parameter value of the r-th MB, T is the total number of pictures, c is a constant, and | r| _t denotes the total number of partitions in the r-th MB. The constant 20.6 is the alignment scaling factor between I pictures and B pictures, and the constant 3.52 is the alignment scaling factor between P pictures and B pictures.

For the examples of FIGs.1 A and 2A, the aligned CU parameters are illustrated in FIGs.l C and 2C, respectively. Compared with raw CU parameters shown in FIGs. 1 B and 2B, the aligned CU parameters are in a more similar scale across different picture types. Thus, the aligned CU parameters may better reflect the content complexity.

In addition to picture types, other encoding configurations, for example, the compression standard in use, the maximum consecutive B pictures, GOP (group of picture) structure, instantaneous refresh rate, and the total number of prediction directions of intra prediction, may affect prediction efficiency and thereby affect the average raw CU parameter values of B, P or I pictures, especially when I frames are used infrequently (i.e., a long GOP is used). Moreover, raw CU parameter values also depend on the video resolution and frame rate. For example, a higher resolution or frame rate often causes lower raw CU parameter values.

Consequently, it is necessary to determine the alignment across different

configurations. In one embodiment, depending on the encoding configurations, different training sets can be used and thus alignment scaling factors can be estimated accordingly. For example, a power function with respect to the resolution and a power function with respect to the frame rate can be used for alignment, which may keep aligned CU parameters independent with the configurations and only related to the content. The estimated content unpredictability parameter for the video sequence, for example, z _c in Eq. (6), can be used to replace Q _c in Eq. (1 ) of Zhangl

(PCT/CN1 1/002096) for measuring video quality, or CU _T in Eq. (3) of Zhang2

(PCT/CN201 1 /082870) for calculating the compression distortion factor. That is, the compression distortion factor may be calculated as:

(51 - ρρ _τ ) ^δ (7) where QP _T is an average QP for the video clip, and bi and £>2 are constants.

Combining the compression distortion factor and other distortion factors (for example, slicing distortion factor and freezing distortion factor), an overall video quality metric can be obtained for the video sequence. Since prediction residuals are available in a bitstream, the video quality measurement according to the present principles may be implemented on a bitstream level.

Using z _c and other parameters, a quality metric may be calculated as: where output variable q is the predicted quality score; constants MOS _U b and are the upper bound and lower bound of MOS (Mean Opinion Score), i.e., 5 and 1 , respectively; a, β, {a} and {b} are model parameters (a _c = 1 constantly); subscripts c, f and s indicate compression, freezing and slicing impairments respectively;

variables {x} and {z} are model factors and also generally termed as features, which are extracted from video data; R is width ^* height/256; and F is the frame

rate. Specifically, {x} and {z} are respectively the key factor and the co-variate associated with each type of impairment, for example, x _c is the key factor for compression impairment and z _s is the co-variate for slicing impairment. FIG. 4 illustrates an exemplary method 400 for measuring video quality based on content complexity according to an embodiment of the present principles.

Method 400 starts at step 410, wherein prediction residuals are accessed, for example, from a bitstream. At step 420, raw CU parameters are calculated, for example, as energy of the prediction residuals as shown in Eq. (1 ). At step 430, alignment scaling factors for different encoding configurations are estimated, for example, using Eq. (5). Based on the raw CU parameters and alignment scaling factors, aligned CU parameters are estimated at step 440 to estimate the content complexity. At step 450, a compression distortion factor may be calculated based on the aligned CU parameters and other encoding parameters (for example,

quantization parameters). Combining the compression distortion factor with slicing and/or freezing distortion factors, an overall video quality metric can be estimated at step 460, for example, using Eq. (8).

In addition to video quality measurement, the content unpredictability parameter can also be used in texture recognition, object recognition, or scene recognition.

FIG. 5 depicts a block diagram of an exemplary video quality measurement apparatus 500 that can be used to generate a video quality metric. The input of apparatus 500 includes a transport stream that contains the bitstream. The input may be in other formats that contains the bitstream. A receiver at the system level determines packet losses in the received bitstream.

Demultiplexer 510 parses the input stream to obtain the elementary stream or bitstream. It also passes information about packet losses to decoder 520. The decoder 520 parses necessary information, including QPs, picture types, transform coefficients, and motion vectors for each block or macroblock, in order to generate parameters for estimating the quality of the video. The decoder also uses the information about packet losses to determine which macroblocks in the video are lost. Decoder 520 is denoted as a partial decoder to emphasize that full decoding is not performed, i.e., the video is not reconstructed. Using the MB level QPs parsed from decoder 520, a QP parser 533 obtains average QPs for pictures and for the entire video clip. Using transform coefficients obtained from decoder 520, a transform coefficients parser 532 parses the coefficients. A content unpredictability parameter calculator 534 calculates the content unpredictability parameter for individual pictures and for the entire video clip, based on transform coefficients, quantization parameters, and encoding

configurations. Using information about which macroblocks are lost, a lost MB tagger 531 marks which MB is lost. Further using motion information, a propagated MB tagger 535 marks which MBs directly or indirectly use the lost blocks for prediction (i.e., which blocks are affected by error propagation). Using motion vectors for blocks, an MV parser 536 calculates a motion parameter for MBs, pictures, and the entire video clip. Other modules (not shown) may be used to determine error concealment distances, durations of freezing, and frame rates.

A compression distortion predictor 540 estimates the compression distortion factor, a slicing distortion predictor 542 estimates the slicing distortion factor, and a freezing distortion predictor 544 estimates the freezing distortion factor. Based on the estimated distortion factors, a quality predictor 550 estimates an overall video quality metric.

When extra computation is allowed, a decoder 570 decodes the pictures. The decoder 570 is denoted as a full decoder and it will reconstruct the pictures and perform error concealment if necessary. A mosaic detector 580 performs mosaic detection on the reconstructed video. Using the mosaic detection results, the lost MB tagger 531 and the propagated MB tagger 535 update relevant parameters, for example, the lost block flag and the propagated block flag. A texture masking estimator 585 calculates texture masking weights. The texture masking weights can be used to weigh the distortions.

The video quality measurement apparatus500 may be used, for example, in ITU-T P.NBAMS standard, which works on video quality assessment models in two application scenarios, namely, IPTV and mobile video streaming, also called HR (High Resolution) scenario and LR (Low Resolution) scenario respectively. The difference between the two scenario ranges from the spatio-temporal resolution of video content and coding configuration to transport protocols and viewing conditions.

The input to the P.NBAMS VQM (Video Quality Model) is coded video bitstream with all transmission packet headers (UDP/IP/RTP or UDP/IP/RTP/TS). The output is an objective MOS score. A major target application of P.NBAMS work is to monitor video quality in a set-top box (STB) or gateway. P.NBAMS mode 1 model only uses bitstream information, and mode 2 model may decode parts or all of the video sequence, and the pixel information is used for visual quality prediction in addition to parsing the bitstream information in order to improve the prediction accuracy.

Referring to FIG. 6, a video transmission system or apparatus 600 is shown, to which the features and principles described above may be applied. A processor 605 processes the video and the encoder 610 encodes the video. The bitstream generated from the encoder is transmitted to a decoder 630 through a distribution network 620. A video quality monitor or a video quality measurement apparatus, for example, the apparatus 500, may be used at different stages.

In one embodiment, a video quality monitor 640 may be used by a content creator. For example, the estimated video quality may be used by an encoder in deciding encoding parameters, such as mode decision or bit rate allocation. In another example, after the video is encoded, the content creator uses the video quality monitor to monitor the quality of encoded video. If the quality metric does not meet a pre-defined quality level, the content creator may choose to re-encode the video to improve the video quality. The content creator may also rank the encoded video based on the quality and charges the content accordingly.

In another embodiment, a video quality monitor 650 may be used by a content distributor. A video quality monitor may be placed in the distribution network. The video quality monitor calculates the quality metrics and reports them to the content distributor. Based on the feedback from the video quality monitor, a content distributor may improve its service by adjusting bandwidth allocation and access control.

The content distributor may also send the feedback to the content creator to adjust encoding. Note that improving encoding quality at the encoder may not necessarily improve the quality at the decoder side since a high quality encoded video usually requires more bandwidth and leaves less bandwidth for transmission protection. Thus, to reach an optimal quality at the decoder, a balance between the encoding bitrate and the bandwidth for channel protection should be considered.

In another embodiment, a video quality monitor 660 may be used by a user device. For example, when a user device searches videos in Internet, a search result may return many videos or many links to videos corresponding to the requested video content. The videos in the search results may have different quality levels. A video quality monitor can calculate quality metrics for these videos and decide to select which video to store. In another example, the decoder estimates qualities of concealed videos with respect to different error concealment modes. Based on the estimation, an error concealment that provides a better concealment quality may be selected by the decoder.

The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed may also be implemented in other forms (for example, an apparatus or program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable/personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" of the present principles, as well as other variations thereof, mean that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment of the present principles. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation", as well any other variations, appearing in various places throughout the specification are not necessarily all referring to the same embodiment.

Additionally, this application or its claims may refer to "determining" various pieces of information. Determining the information may include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

Further, this application or its claims may refer to "accessing" various pieces of information. Accessing the information may include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information. Additionally, this application or its claims may refer to "receiving" various pieces of information. Receiving is, as with "accessing", intended to be a broad term. Receiving the information may include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, "receiving" is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information. As will be evident to one of skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry the bitstream of a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The

formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.