TECHNICAL FIELD

This invention relates to video quality measurement, and more particularly, to a method and apparatus for determining an objective video quality metric. The determined video quality metric can then be used, for example, to adjust encoding parameters, or to provide required video quality at the receiver side.

BACKGROUND

A video bitstream may suffer from transmission losses when it is transmitted over unreliable channels such as wireless network or Internet. Consequently, in addition to the quality loss from video compression, the video quality is further degraded when a video is transmitted through unreliable networks. A successful video quality modeling tool needs to rate the quality degradation caused by network transmission impairment (for example, packet losses, transmission delays, and transmission jitters), in addition to quality degradation caused by video compression.

SUMMARY

The present principles provide a method for estimating visual quality of a video sequence, comprising the steps of: accessing a first set of pixels that is used to conceal a block of a frame in the video sequence; accessing a second set of pixels that is used to approximate the block; determining differences responsive to the first and second sets of pixels; and estimating the visual quality of the video sequence responsive to the determined differences as described below. The present principles also provide an apparatus for performing these steps.

The present principles also provide a method for estimating visual quality of a video sequence, comprising the steps of: accessing a first set of pixels that is used to conceal a block of a frame in the video sequence; accessing a second set of pixels that is used to approximate the block; determining differences responsive to the first and second sets of pixels; estimating a respective set of differences for each of a plurality of spatial neighboring blocks of the block; and estimating the visual quality of the video sequence responsive to the determined differences and the respective sets of differences for the plurality of spatial neighboring blocks as described below. The present principles also provide an apparatus for performing these steps.

The present principles also provide a computer readable storage medium having stored thereon instructions for estimating visual quality of a video sequence according to the methods described above.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a pictorial example depicting that a macroblock (EC-MB B) in frame t+k is used to conceal a macroblock (Error MB A) in current frame t.

FIG. 2A is a pictorial example depicting artifacts caused by spatial error concealment, FIG. 2B is a pictorial example depicting artifacts caused by temporal error concealment, and FIGs. 2C and 2D are pictorial examples depicting artifacts caused by error propagation. FIG.3 is a flow diagram depicting an example of estimating initial visible artifact level, in accordance with an embodiment of the present principles.

FIG. 4 is a pictorial example depicting an error macroblock and its four neighboring macroblocks. FIG. 5 is a pictorial example depicting a decoded frame after error

concealment.

FIG. 6 is a block diagram depicting an example of a video quality monitor, in accordance with an embodiment of the present principles.

FIG. 7 is a block diagram depicting an example of a video processing system that may be used with one or more implementations.

DETAILED DESCRIPTION

Most existing video compression standards, for example, H.264 and MPEG-2, use a macroblock (MB) as the basic encoding unit. Thus, the following embodiments use a macroblock as the basic processing unit. However, the principles may be adapted to use a block at a different size, for example, an 8x8 block, a 1 6x8 block, a 32x32 block, or a 64x64 block.

When some portions of a coded video bitstream are lost during network transmission, macroblocks corresponding to the lost portions are denoted as lost macroblocks and they are undecodable. Some macroblocks, even if correctly received, may also become undecodable, for example, because the syntax for the macroblock is out of synchronization and cannot be properly parsed by the decoder. Undecodable macroblocks, either because the corresponding compressed data are lost or the syntax is out of synchronization, are denoted as error MBs in the present application. Decodable macroblocks, if using undecodable macroblocks as references, may not be reconstructed properly either, and thus causes visual artifacts (usually known as error propagation). To reduce the perceived artifacts, a decoder may adopt error concealment techniques to conceal macroblocks that are not properly reconstructed. The goal of error concealment is to restore undecodable macroblocks in order to reduce perceptual quality degradation. The perceived strength of artifacts in restored macroblocks depends heavily on employed error concealment techniques. A spatial or temporal approach may be used for error concealment. In a spatial approach, spatial correlation between pixels is exploited, and undecodable macroblocks can be recovered from neighboring pixels using interpolation techniques. In a temporal approach, both temporal coherence and spatial smoothness of pixels can be exploited to estimate an undecodable macroblock or pixel. For example, undecodable pixels can be estimated using collocated pixels in a previously decoded frame (i.e., using pixels in the same positions as the

undecodable pixels in the previously decoded frame), or can be concealed using reference pixels in previous frames.

In the present application, a macroblock used to conceal an error macroblock is denoted as an EC-MB, and terms "picture" and "frame" are used interchangeably. FIG. 1 illustrates that an EC-MB B in a previous decoded frame t+k can be used to conceal an Error MB A in the current frame t in a temporal error concealment approach. Visual artifacts may still be perceived after error concealment. FIGs. 2A and 2B illustrate exemplary decoded frames, where some packets of the coded bitstream are lost during transmission. Specifically, for FIG. 2A, spatial error concealment is used and obvious artifact can still be noticed. For FIG. 2B, temporal error concealment is used, and some macroblocks are restored properly and some other macroblocks still have visible artifacts. FIGs. 2C and 2D illustrate artifacts caused by error propagation.

As discussed above, visual artifacts may still be perceived after error concealment. The visual artifacts may occur at an error MB because error concealment is not effective therein. Such visual artifacts caused by undecodable macroblocks are denoted as initial visible artifacts. If a block having initial visible artifacts is used as a reference, for example, for intra prediction or inter prediction, the initial visible artifacts may propagate spatially or temporally to other blocks in the same or other frames through prediction. Such propagated artifacts are denoted as propagated visible artifacts. The overall artifacts, caused by initial and propagated visible artifacts, are denoted as overall visible artifacts.

The present principles are directed to measuring initial visible artifact level, which can then be used to measure the overall visible artifact level.

FIG. 3 illustrates an exemplary video quality modeling method 300 for estimating initial visible artifact levels. Method 300 starts at step 305. At step 310, we locate error macroblocks in the current frame, for example, from the bitstream information. At step 320, we estimate pixels that are used for performing error concealment of a current error MB. At step 325, we locate pixels that can be used to approximate the current error MB. At step 330, we estimate the error (also referred to as residual) caused by transmission errors which remains after error concealment in the current error MB. Based on the estimated error, at step 340, we calculate an initial visible artifact level for the current error MB. At step 350, it checks whether more error MBs need to be processed. If yes, the control returns to step 325.

Otherwise, method 300 ends at step 399.

In the following, the steps of estimating pixels used for performing error concealment for the current error MB (320), locating pixels to approximate the current error MB (325), estimating residual for the current error MB (330), and calculating the initial visible artifact level(340)are discussed in further detail.

Estimate pixels used for error concealment

Various error concealment methods are available. In one exemplary embodiment, an error MB can be the restored by copying its collocated MB in a previous decoded frame as shown in FIG. 1 , wherein a collocated MB in a previous decoded frame refers to a macroblock in the same position as the error MB in the previous decoded frame. Considering this exemplary error concealment method, we discuss in the following how we can estimate pixels that are used for performing error concealment for a current error MB. The method can be applied when other error concealment methods are used. For an error MB of current frame t, we calculate the difference between a pixel in the error MB and its collocated pixel in a neighboring frame t+k:

dij ( ^k ) = (t ^~ j (t + /c) | , pixel (i, j) e error MBs, k =—3,—2,—1,1,2,3, (1 ) where Ijj(t) is the luminance intensity of the pixel in the i ^th row and f ^h column of current frame t, j(t+k) is the luminance intensity of the pixel in the / row and / column of neighboring frame t+k, and d,j- (k) is the difference.

Then we average the differences for the pixels in the error MBs in the current frame with respect to the corresponding MBs in neighboring frame t+k: d(/c) = average (d _i; (/c)) , pixel(ij) e error MBs, k =—3, -2, -1,1,2,3. (2)

We consider the neighboring frame with the minimum d(/c)as the frame used for performing error concealment on the error MBs of the current frame:

k _EC = argmin _fc (d(/c)), (3) where t+k _E c denotes the detected error concealment frame.

In addition to detecting error concealment frame t+k _E c, there may be other neighboring frames whose d(/c)are close or equal to d(k _EC ). In one embodiment, we select the frame nearest to current frame t as the error concealment frame. In another embodiment, if a preceding neighboring frame and a subsequent frame are both close or equal to d k _EC ), the preceding frame is selected as the error

concealment frame. For example, if the average difference value of frame t-2, t-1 , and t+1 are all zero, then frame t-1 is selected as the frame used for error

concealment.

In the above, we consider neighboring six frames (i.e., frames t-3, t-2, t-1 , t+1 , t+2, t+3) to detect error concealment frame t+k _E c, which are generally sufficient for common GOP (group of picture) structures (for example, IBBPBBP, IBPBP). In other embodiments, we may consider more or fewer neighboring frames.

Locate pixels to approximate an error MB As discussed above, when an error MB is detected, an EC-MB is used to conceal the error MB. For ease of notation, we denote the error MB after error concealment as the restored error MB (that is, the restored error MB is identical to the corresponding EC-MB if an EC-MB is correctly detected and copied for error concealment), and denote the error MB as if it had been correctly decoded as the original error MB.

When the restored error MB is not similar to the original error MB, the difference between the original error MB and the restored error MB may cause visible artifacts. In order to estimate the artifact after error concealment, we first estimate the difference between the original error MB and the restored error MB.

Because an error MB is not correctly decoded (i.e., some or all bits

representing the error MB are not correctly received or parsed) and it may be recovered using error concealment, taking a difference directly between the EC-MB and the restored error MB would not reflect the distortion caused by transmission losses and remains after error concealment. In fact, if our estimation of the error concealment frame is accurate, a difference between the EC-MB and the restored error MB may become zero if an EC-MB is copied for error concealment, and thus, would not provide any meaningful indication about whether artifacts will be visible at the error MB. To more effectively estimate distortion at an error MB, that is, to more accurately estimate the difference between the original error MB and the restored error MB, we use a corresponding macroblock in temporal neighboring frames to estimate the original error MB. That is, a macroblock in a temporal neighboring frame is used to approximate what should have been correctly decoded at the error MB. We consider the video characteristics, for example, but not limited to, scene cut around current frame t, the distance between the current frame and the error concealment frame, and choose a representative macroblock in the temporal neighborhood that can be used to approximate the error MB.

In one embodiment, we use a macroblock in neighboring frame t+1 or t-1 to approximate the original error MB. In one embodiment, to determine whether a macroblock in frame t+1 or t-1 is used to approximate the original error MB (i.e., whether m is set to be 1 or -1 ), we use the following rules:

- if k _E c<0 and frame t+1 is not a scene cut frame and the corresponding MB in frame t+1 is not an error MB; or if k _E c>1 and frame t is a scene cut frame; or if k _E c>1 and the corresponding MB in frame t-1 is an error MB; we set m=1 .

- if k _E c>0 and frame t is not a scene cut frame and the corresponding MB in

frame t-1 is not an error MB; or if k _EC <-1 and frame t+1 is a scene cut frame; or if k _EC <-1 and the corresponding MB in frame t+1 is not an error MB; we set m=-1 .

- if the corresponding MB in frame t+k _EC is an error MB, or if the above listed conditions are not satisfied, the error MB is marked as ineffective and it will not be used in estimating the initial visible artifact level. Otherwise, the error MB is marked as effective.

That is, we try to select a frame that is further away from the error

concealment frame and is in the same scene as the current frame. In addition, we choose the temporal frame wherein the corresponding macroblock is not an error MB. Estimate residual for an error MB

After we decide which frame can be used to approximate the original error MB, the difference between a pixel in the original error MB and a corresponding pixel in the restored error MB (i.e., the corresponding EC-MB in frame t+k _E c) can be estimated as: d-ij = \ j (t + τη)— lij t + /c _£C ) | , pixel(i, j) e error MB; m = 1 or— 1. (4)

In addition to the pixel differences calculated for the error MB, we may also estimate the differences at spatial neighboring MBs of the error MB (for example, MBu, MB|_, MB _R , and MB _D as shown in FIG. 4). neighboring macroblock is an error MB, the pixel differences of the neighboring macroblock is calculated as discussed above, for example, as described in Eq. (4). Otherwise, the pixel difference can be calculated as: dij = \ j ) - j + k _EC ) \ .{5)

If the corresponding MB in frame t+k _E c for the neighboring macroblock is an error MB, the neighboring macroblock is marked as ineffective and will not be used to estimate the initial visible artifact level. A neighboring MB is considered effective if it is not marked as ineffective.

Based on the pixel differences calculated for the error MB itself and its four spatial neighboring macroblocks (i.e., upper, lower, left, and right immediate neighboring macroblocks), we calculate a weighted difference, denoted as residual R, for the error MB:

lij _t ≠ 0, i, j e error MB, MB _U; MB _L , MB _R , or MB D

(6)

NULL , N = 0 where N is the number of effective macroblocks among the error ΜΒ,ΜΒυ, MB _L ,MBR, and MB _D . If all five MBs are marked as ineffective (i.e., N = 0), R is set to NULL.

Other variations can be used to calculate residual R, such as applying different weights to different macroblocks based on their correlation with the error MB, or applying luminance masking or texture masking to adjust d , or count the number of dy whose value is higher than a threshold to set the value of N. We may also use the pixel differences of the error MB only or the differences of neighboring macroblocks only when calculating residual R. These variations could be more effective for some special cases. In the above, we consider spatial and temporal neighboring pixels to calculate residual R. When considering temporal neighboring pixels, immediate temporal neighboring frames are considered; when considering spatial neighboring pixels, the upper, lower, left, and right immediate neighboring macroblocks are considered. In other embodiments, we may expand or reduce the neighboring pixels that are considered, for example, we may also use pixels in upper-left, upper-right, lower-left and lower-right neighboring macroblocks.

In another embodiment, we may estimate the original error MB by using its spatio-temporal neighboring macroblocks, and calculate residual R based on pixel differences between the restored error MB and the estimated original error MB. Estimate the initial visible artifact level for an error MB

If the residual value R for an error MB is NULL, we consider that there will be visible artifact at the error MB, for example, we set the initial visible artifact level of the error MB to be 1 . Otherwise, the initial visible artifact level of the error MB may be calculated as:

where T1 and T2 are two thresholds, and V denotes the initial visible artifact level. If residual value R is smaller than threshold T1 , we consider that the difference between the original error MB and the restored error MB is low and no visible artifact can be noticed. If residual value R is higher than threshold T2, we consider that the difference is high and visible artifact can be noticed. ThresholdsTI and T2 can be trained, for example, based on the content, user requirement, and the application. In one example, we set T1 = 5, and T2 = 1 0. Other monotonic increasing functions can be used to map residual value R to the initial visible artifact level, for example, a logarithmic or exponential function can be used.

After initial visible artifact levels for individual error MBs are estimated, the initial visible artifact level of the image/video can be estimated, for example, by averaging the initial visible artifact levels of error MBs. Because of intra or inter prediction used in video compression, initial visible artifacts of error MBs may propagate spatially and temporally to other macroblocks or other frames.

Propagated visible artifact levels can be estimated, for example, using the method and apparatus described in a commonly owned PCT application, entitled "Video quality assessment at a bitstream level" by N. Liao, Z. Chen, and K. Xie

(PCT/CN1 2/078766, Attorney Docket No. PA120027), the teachings of which are specifically incorporated herein by reference. Based on initial visible artifact levels and propagated visible artifact levels, an overall artifact level caused by transmission losses can be estimated. Subsequently, a quality metric can be estimated, for example, based on the overall artifact level caused by transmission losses and video compression artifact.

Advantageously, by considering both spatial and temporal neighboring macroblocks to measure the artifact level for an error MB, the present principles can not only detect spatial artifact, but also temporal artifact. In addition, by considering the difference of the macroblocks themselves, rather than the difference or discontinuity at block boundaries, the present principles can be more efficient in detecting artifacts. For example, in a decoded frame as shown in FIG. 5, the spatial discontinuity is not very high. But when the video is played frame by frame, obvious temporal discontinuity can be observed. Our method can provide a high artifact level which indicates that there exist spatial/temporal artifact in the frame.

The present principles may be used in a video quality monitor to measure video quality. FIG. 6 depicts a block diagram of an exemplary video quality monitor 600. The input of apparatus 600 may include a transport stream that contains the bitstream. The input may be in other formats that contain the bitstream.

Demultiplexer610 obtains packet layer information, for example, which packets are lost, from the bitstream. Decoder 620 decodes the input stream to reconstruct the video and perform error concealment if necessary. Decoder 620 also determines which macroblock is an error MB. In other embodiments, the decoder may perform the functions of the demultiplexer.

Using the decoded frames, initial visible artifact levels for error macroblocks are estimated at initial visible artifact level estimator 630, for example, using method 300. Considering the initial visible artifacts and error propagation, the overall artifact levels for individual macroblocks are estimated at overall artifact level estimator 640. A quality predictor 650 then pools the macroblock level artifact levels into a quality score. The quality predictor 650 may consider other types of artifacts and the property of human visual property.

The video quality monitor 600 may be used, for example, in ITU-T P.NBAMS (parametric non-intrusive bitstream assessment of video media streaming quality) standard, which works on video quality assessment models in two application scenarios, namely, IPTV and mobile video streaming, also called HBR scenario and LBR scenario respectively. The difference between the two scenario ranges from the spatio-temporal resolution of video content and coding configuration to transport protocols and viewing conditions.

The input to the P.NBAMS VQM (Video Quality Model) is coded video bitstream with all transmission packet headers (UDP/IP/RTP or UDP/IP/RTP/TS). The output is an objective MOS (Mean Opinion Score) score. To be embedded in a set-top box (STB) or gateway to monitor video quality is a major target application of P.NBAMS work. P.NBAMS mode 1 model only uses bitstream information by parsing the H.264 syntax, and mode 2 model may decode parts or all of the video sequence, and the pixel information is used for visual quality prediction in addition to parsing the bitstream information in order to improve the prediction accuracy.

Referring to FIG. 7, a video transmission system or apparatus 700 is shown, to which the features and principles described above may be applied. A processor 705 processes the video and the encoder 710 encodes the video. The bitstream generated from the encoder is transmitted to a decoder 730 through a distribution network 720. A video quality monitor, for example, the video quality monitor 600, may be used at different stages.

In one embodiment, a video quality monitor 740 may be used by a content creator. For example, the estimated video quality may be used by an encoder in deciding encoding parameters, such as mode decision or bit rate allocation. In another example, after the video is encoded, the content creator uses the video quality monitor to monitor the quality of encoded video. If the quality metric does not meet a pre-defined quality level, the content creator may choose to re-encode the video to improve the video quality. The content creator may also rank the encoded video based on the quality and charges the content accordingly.

In another embodiment, a video quality monitor 750 may be used by a content distributor. A video quality monitor may be placed in the distribution network. The video quality monitor calculates the quality metrics and reports them to the content distributor. Based on the feedback from the video quality monitor, a content distributor may improve its service by adjusting bandwidth allocation and access control.

The content distributor may also send the feedback to the content creator to adjust encoding. Note that improving encoding quality at the encoder may not necessarily improve the quality at the decoder side since a high quality encoded video usually requires more bandwidth and leaves less bandwidth for transmission protection. Thus, to reach an optimal quality at the decoder, a balance between the encoding bitrate and the bandwidth for channel protection should be considered. In another embodiment, a video quality monitor 760 may be used by a user device. For example, when a user device searches videos in Internet, a search result may return many videos or many links to videos corresponding to the requested video content. The videos in the search results may have different quality levels. A video quality monitor can calculate quality metrics for these videos and decide to select which video to store. In another example, the user device may have access to several error concealment techniques. A video quality monitor can calculate quality metrics for different error concealment techniques and automatically choose which concealment technique to use based on the calculated quality metrics. The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed may also be implemented in other forms (for example, an apparatus or program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable/personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" of the present principles, as well as other variations thereof, mean that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment of the present principles. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation", as well any other variations, appearing in various places throughout the specification are not necessarily all referring to the same embodiment.

Additionally, this application or its claims may refer to "determining" various pieces of information. Determining the information may include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory. Further, this application or its claims may refer to "accessing" various pieces of information. Accessing the information may include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

Additionally, this application or its claims may refer to "receiving" various pieces of information. Receiving is, as with "accessing", intended to be a broad term. Receiving the information may include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further,

"receiving" is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

As will be evident to one of skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry the bitstream of a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The

formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.