BACKGROUND

[0001] To encode video for streaming over a network or a wireless channel, it has become possible to perform different types of encoding on different slices of a same video frame. For example, the ITU's (International Telecommunication Union) H.264 standard allows for a frame to have some slices that are independently encoded ("ISlices"). An ISlice has no dependency on other parts of the frame or on parts of other frames. The H.264 standard also allows slices ("PSlices") of a frame to be encoded based on other slices of a preceding frame.

[0002] When a stream of frames encoded in slices is transmitted on a lossy channel, if an individual Nth slice of one frame is corrupted or dropped, it is possible to recover from that partial loss by encoding the Nth slice of the next frame as an ISlice. However, when an entire frame is dropped or corrupted, a full encoding recovery becomes necessary. Previously, such a recovery would be performed by transmitting an entire Iframe (as used herein, an "Iframe" will refer to either a frame that has only ISlices or a frame encoded without slices, and a "Pframe" will refer to a frame with all PSlices or a frame encoded with some non-intra-frame encoding blocks). However, as observed only by the present inventors, the transmission of an Iframe can cause a spike in frame size relative to Pframes or frames that have mostly PSlices. This spike can create latency problems, jitter, or other artifacts that can be problematic, in particular for interactive applications such as games.

[0003] Techniques related to recovering from corrupt or dropped Pframes are discussed below.

SUMMARY

[0004] The following summary is included only to introduce some concepts discussed in the Detailed Description below. This summary is not comprehensive and is not intended to delineate the scope of the claimed subject matter, which is set forth by the claims presented at the end.

[0005] Embodiments relate to encoding and decoding frames of a video stream.

Video frames are encoded as intra-coded frames (Iframes) and predictive coded frames (Pframes) or bi-predictive coded frames (Bframes) and transmitted. When a receiver of the encoded frames is unable to decode a frame, due to transmission problems or otherwise, the encoded video stream can be recovered without requiring a full Iframe to be generated at one time. Instead, intra-coded data is provided by the transmitter in slices. Specifically, frames with only portions of intra-coded data (Islices) are transmitted in sequence until enough intra-coded data is provided to the receiver to recover a frame and resume decoding. The intra-refresh frames may also contain slices predictively encoded (Pslices) based on restricted search spaces of preceding intra-refresh frames.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The present description will be better understood from the following detailed description read in light of the accompanying drawings, wherein like reference numerals are used to designate like parts in the accompanying description.

[0007] Figure 1 shows a host transmitting a video stream to a client.

[0008] Figure 2 shows a timeline of processing by a frame-by-frame pipeline architecture.

[0009] Figure 3 shows a timeline where video frames are processed in incremental portions.

[0010] Figure 4 shows how a framebuffer, an encoder, and a

transmitter/multiplexer (Tx/mux) can be configured to process portions of frames concurrently.

[0011] Figure 5 shows a sequence of encoded video frames transmitted from the host to the client.

[0012] Figure 6 shows how a video stream can be recovered when a Pframe becomes unavailable for decoding.

[0013] Figure 7 shows a process for performing an intra-refresh when encoded video data is unavailable.

[0014] Figure 8 shows an example of a computing device.

[0015] Many of the attendant features will be explained below with reference to the following detailed description considered in connection with the accompanying drawings.

DETAILED DESCRIPTION

[0016] Figure 1 shows a host 100 transmitting a video stream to a client 102. The host 100 and client 102 may be any type of computing devices. An application 104 is executing on the host 100. The application 104 can be any code that generates video data, and possibly audio data. The application 104 will generally not execute in kernel mode, although this is possible. The application 104 has logic that generates graphic data in the form of a video stream (a sequence of 2D frame images). For instance, the application 104 might have logic that interfaces with a 3D graphics engine to perform 3D animation which is rendered as 2D images. The application 104 might instead be a windowing application, a user interface, or any other application that outputs a video stream.

[0017] The application 104 is executed by a central processing unit (CPU) and/or a graphics processing unit (GPU), perhaps working in combination, to generate individual video frames. These raw video frames (e.g., RGB data) are written to a framebuffer 106. While in practice the framebuffer 106 may be multiple buffers (e.g., a front buffer and a back buffer), for discussion, the framebuffer 106 will stand for any type of buffer arrangement, including a single buffer, a triple buffer, etc. As will be described, the framebuffer 106, an encoder 108, and a transmitter/multiplexer (Tx/mux) 108 work together, with various forms of synchronization, to stream the video data generated by the application 104 to the client 102.

[0018] The encoder 108 may be any type of hardware and/or software encoder configured to implement a video encoding algorithm (e.g., H.264 variants, or others) with the primary purpose of compressing video data. Typically, a combination of inter-frame and intra-frame encoding will be used.

[0019] The Tx/mux 108 may be any combination of hardware and/or software that combines encoded video data and audio data into a container, preferably of a type that supports streaming. The Tx/mux 108 may interleave video and audio data and attach metadata such as timestamps, PTS/DTS durations, or other information about the stream such as a type or resolution. The containerized (formatted) media stream is then transmitted by various communication components of the host 100. For example, a network stack may place chunks of the media stream in network/transport packets, which in turn may be put in link/media frames that are physically transmitted by a

communication interface 111. In one embodiment, the communication interface 111 is a wireless interface of any type.

[0020] As will be explained with reference to Figure 2, in previous devices, the type of pipeline generally represented in Figure 1 would operate on a frame-by-frame basis. That is, frames were processed as discrete units during respective discrete cycles. Although the devices in Figure 1 have similarities to such prior devices, they also differ from prior devices in ways that will be described herein.

[0021] Figure 2 shows a timeline of processing by a frame-by-frame pipeline architecture. With prior graphics generating devices, a refresh signal that corresponds to a display refresh rate drives the graphics pipeline. For example, for a 60 Hz refresh rate, a vsync (vertical-sync) signal is generated for every 16 ms refresh cycle 112 (112A-112D refer to individual cycles). Each refresh cycle 112 is started by a vsync signal and begins a new increment of parallel processing by each of (i) the capturing hardware that captures to the framebuffer 106, (ii) the encoder 108, and (iii) the Tx/mux 110. In Figure 2, it is assumed that a new video stream is starting, for example, in response to a user input. As will be explained, a graphics pipeline corresponding to the example of Figure 2 requires two refresh cycles 1 12 before the corresponding video stream can begin transmitting to the client 102.

[0022] At the beginning of the first refresh cycle 112A after the user input, each component of the graphics pipeline is empty or idle. During the first refresh cycle 112A, the framebuffer 106 fills with the first frame (Fl) of raw video data. During the second refresh cycle 112B, the encoder 108 begins encoding the frame Fl (forming encoded frame El), while at the same time the framebuffer 106 begins filling with the second frame (F2), and the Tx/mux 110 remains idle. During the third refresh cycle 112C, each of the components is busy: the Tx/mux 110 begins to process the encoded frame El

(encoded Fl, forming container frame Ml), the encoder 108 encodes frame F2 (forming a second encoded frame E2), and the framebuffer 106 fills with a third frame (F3). The fourth refresh cycle 112D and subsequent cycles continue in this manner until the framebuffer 106 is empty. This is assumes that the encoder takes 16ms to encode a frame. However, if the encoder is capable to encoding faster, the Tx/mux can start as soon as the encoder is finished. Due to power considerations, the encoder can be typically run so that it can encode a frame in 1 vsync period.

[0023] It is apparent that a device configured to operate as shown in Figure 2 has an inherent latency of approximately two refresh cycles between the initiation of video generation (e.g., by a user input or other triggering event) and the transmission of the video. For some applications such as interactive games, this delay to prime the graphics pipeline can be noticeable and the experience of the user may not be ideal. As will be explained with reference to Figures 1, 3, and 4, this latency can be significantly reduced by configuring the host 100 to process frames in piecewise fashion where portions of a same frame are processed in parallel at different stages of the pipeline.

[0024] Figure 3 shows a timeline where video frames are processed in incremental portions. In the example of Figure 3, each frame has 4 portions (N=4). However, any number greater than two may be used for N, with the consideration that larger values of N may decrease the latency but the video fidelity and/or coding rate may be impacted due to smaller portions being encoded. The frames in Figure 3 will be referred to with similar labels as in Figure 2, but with a sub-index number added. For example, the first unencoded frame Fl has four portions that will be referred to as F 1-1, F 1-2, F 1-3, and Fl- 4. Similarly, the first encoded frame, for example, has portions El-1 through El-4, and the first Tx/mux frame has container portions Ml-1 to Ml -4.

[0025] Figure 1 shows unencoded frame portions 120 passing from the

framebuffer 106 to the encoder 108. Figure 1 also shows encoded frame portions 122 passing from the encoder 108 to the Tx/mux 110. Figure 1 further shows container portions outputted by the Tx/mux 110 for transmission by the communication facilities (e.g., network stack and communication interface 111) of the host 100. The frame portions 120 may be any of the frame portions FX-Y (e.g., Fl-1) shown in Figure 3. The encoded portions 122 may be any of the encoded portions EX-Y (e.g., E2-4), and the container portions 124 may be any of the container portions MX-Y (e.g., Ml-3).

[0026] Figure 4 shows how the framebuffer 106, the encoder 108, and the Tx/mux 110 can be configured to process portions of frames concurrently, possibly even before a video frame is completely generated and fills the framebuffer 106. Initially, as in Figure 2, the application 104 begins to generate video data, which starts to fill the framebuffer 106. At step 130, the video capture hardware is monitoring the framebuffer 106. At step 132 the video capture hardware determines that the framebuffer 106 contains a new complete portion of video data, and, at step 134, signals the encoder 108.

[0027] At step 136 the encoder 108 is blocked (waiting) for a portion of a video frame. At step 138 the encoder 108 receives the signal that a new frame portion 120 is available. In this example, the first frame portion will be frame Fl-1. At step 140 the encoder 108 signals the Tx/mux 110 that an encoded portion 122 is available. In this case, the first encoded portion is encoded portion El-1 (the encoded form of frame portion Fl-

1).

[0028] At step 142 the Tx/mux 110 is block-waiting for a signal that data is available. At step 144 the Tx/mux 110 receives the signal that encoded portion El-1 is available, copies or accesses the new encoded portion, and in turn the Tx/mux 110 multiplexes the encoded portion El-1 with any corresponding audio data. The Tx/mux 110 outputs the container portion 124 (e.g., Ml-1) for transmission to the client 102.

[0029] It should be noted that the aforementioned components operate in parallel.

When the capture hardware has finished a cycle at step 134 the capture hardware continues at step 130 to check for new video data while the encoder 108 operates on the output from the framebuffer 106 and while the Tx/mux 110 operates on the output from the encoder 108. Similarly, when the encoder 108 has finished encoding one frame portion it begins a next, and when the Tx/mux 110 has finished one encoded portion it begins a next one, if available.

[0030] As can be seen in Figure 3, by reducing the granularity of processing from frames to portions of frames, it is possible to reduce the latency between the initiation of video generation and the transmission of the appropriately processed generated video. Synchronization between the pipeline components can be accomplished in a variety of ways. As described above, each component can generate a signal for the next component. Timers can be used to assure that each component does not create a conflict by failing to finish processing a portion in sufficient time. For example, if frames are partitioned into four portions, and the refresh cycle is 16 ms, then each component might have a 4 ms timer. In practice, the time will be a small amount less to allow for overhead such as interrupt handling, data transfer, and the like. In another embodiment, the graphics pipeline is driven by the vsync signal and each component has an interrupt or timer appropriately offset from the vsync signal (e.g., ~4ms). Different components can generate interrupts as a mechanism to notify the next component in pipeline that the data is ready for their consumption. Any combination of driver signals, timers, and inter- component signals, implemented either in hardware, firmware, or drivers, can be used to synchronize the pipeline components.

[0031] Details about how video frames can be encoded by portions or slices are available elsewhere; many video encoding standards, such as the H.264 standard, specify features for piece-wise encoding. In addition, the client 102 need not be modified in order to process the video stream received from the host 100. The client 102 receives an ordinary containerized stream. An ordinary decoder at the client 102 can recognize the encoded units (portions) and decode accordingly. In one embodiment, the client 102 can be configured to decode in portions, which might marginally decrease the time needed to begin displaying new video data received from the host 100.

[0032] In a related aspect, latency or throughput can be improved in another way. Most encoding algorithms create some form of dependency between encoded frames. For example, as is well understood, time-variant information, such as motion, can be detected across frames and used for compression. Even in the case where a frame is encoded in portions, as described above, some of those portions will have dependencies on previous portions. The embodiments described above can end up transmitting individual portions of frames in different frames or packets. A noisy channel that causes intermittent packet loss or corruption can create problems because loss/corruption of a portion of a frame can cause the effective loss of the entire frame or a portion thereof. Moreover, a next

Pframe/Bframe (predicted frame) may not be decodable without the good reference. For convenience, wherever the terms "Pframe" and "Pslice" are used herein, such terms are intended to represent predictively encoded frames/slices, or bi-directionally predicted frames/slices (Bframes/Bslices), or both. In other words, where the context permits, "PFrame" refers to "Pframe and/or Bframe", and "Pslice" refers to "Pslice and/or Bslice". Described next are techniques to refresh (allow decoding to resume) a disrupted encoded video stream without requiring transmission of a full Iframe (intracoded frame).

[0033] Figure 5 shows a sequence 160 of encoded video frames transmitted from the host 100 to the client 102. As is known in the art of video encoding, frames can be encoded based on changes between frames (Pframes 164 A -164C) or based only on the intrinsic content of one frame (Iframes 162). An Iframe can be decoded without needing other frames, but Iframes are large relative to Pframes and Bframes. Pframes, on the other hand, depend on and require other frames to be decoded cleanly. As shown in Figure 5, when Pframe 164B is not available for decoding, perhaps due to packet loss or corruption during transmission, the next Pframe 164C cannot be decoded. Prior approaches would require a new Iframe each time a Pframe was effectively not available for decoding.

Embodiments described next allow an encoded video stream to be recovered with low latency and with near-certainty and reasonable fidelity.

[0034] As is also known and discussed above, many video encoding algorithms and standards include features that allow slice-wise encoding. That is, a video frame can have intra-encoded (self-decodable data) portions or slices, as well as predictively encoded portions or slices. The former are often referred to as Islices, and the latter are often referred to as Pslices. As shown in Figure 5, a Pframe can be encoded as set of Pslices 170, and an Iframe can be encoded as a set of Islices 172. It is also possible for an encoded frame to have a mix of Islices 172 and Pslices 170, with the Pslices of one frame being dependent on Pslices and/or Islices of the previous frame. Slice-based encoding can be helpful for a pipeline that works with portions of frames rather than whole frames, as described above. In addition, smaller pieces of encoded data such as Pslices and Islices can be individually transmitted across a wireless link or other potentially lossy medium, which can help with data retransmission. If a slice is unavailable for decoding, only that slice might need to be retransmitted in order to recover. Nonetheless, in some situations, an entire frame might be unavailable for decoding.

[0035] Figure 6 shows how a video stream can be recovered when a Pframe becomes unavailable due to packet loss, corruption, misordering, etc. When the client 102 provides feedback to the host 100 that a frame has been corrupted or lost, the host 100 transmits a sequence of frames that together include sufficient Islices to refresh the video stream. Supposing that Pframe 164B has been dropped, a first refresh-frame 180A is encoded with a corresponding Islice 182 and a remainder of Pslices br. A next refresh- frame, second refresh-frame 180B, is then encoded with a second Islice in the next slice position. The third refresh-frame 180C is similarly encoded with an Islice at the next slice position (the third slice position). The fourth refresh-frame 180D is encoded with an Islice at the fourth and last slice position (partitions other than four slices may be used).

[0036] The other slices of each refresh-frame are encoded as Pslices. However, because only portions of a previous refresh-frame may be valid, the encoding of any given Pslice may involve restrictions on the spatial scope of scans of the previous frame. That is, scans for predictive encoding are limited to those portions of the previous frame that contain valid encoded slices (whether Pslices or Islices). In one embodiment where the encoding algorithm uses a motion vector search for motion-based encoding, the motion vector search is restricted to the area of the previous refresh-frame that is valid (i.e., the intra-refreshed portion of the previous frame). In the case of the second refresh-frame

180B, predictive encoding is limited to only the Islice of the first refresh-frame 180A. In the case of the third refresh-frame 180C, predictive encoding is limited to the first two slices of the second refresh-frame 180B (a Pslice and an Islice). For the fourth refresh- frame 180D, predictive encoding is performed over all but the last slice of the third refresh-frame 180C. After the fourth refresh-frame 180D, the video stream has been refreshed such that the current frame is a complete validly encoded frame and encoding with mostly Pframes may resume.

[0037] While different patterns of Islice positions may be used over a sequence of refresh-frames, the staggered approach depicted in Figure 6 may be preferable because it provides a contiguous searchable frame area that increases in size with each refresh-frame; the first refresh-frame has a one-slice searchable area, the next has a two-slice searchable area, and so forth. Moreover, the searchable area grows with the addition of predictively encoded slices (Pslices) and therefore is encoded with a minimal amount of intra-encoded data in any given intra-refresh frame. [0038] Figure 7 shows a process for performing an intra-refresh when encoded video data is unavailable. At step 200, the host 100 is transmitting primarily Pframes, each dependent on the previous for decoding. At step 202, the client 102 receives the Pframes and decodes them using the previous Pframes. While receiving the Pframes, the client 102 detects a problem with a Pframe (e.g., missing, corrupt, out of sequence, etc.). Missing encoded data can be detected at the network layer, at the encoding layer, at the decoding layer or any combination of these. In response to the missing Pframe, at step 204 the client 102 transmits a message to the host 100 indicating which frame was not able to be decoded by the client 102. At step 206 the host 100 begins sending intra-refresh frames. A loop can be used to incrementally shift the slice to be intra-encoded (encoded as an Islice) down after each frame. At step 208, the current intra-refresh frame is encoded. For the i-th refresh-frame, the i-th slice is encoded as an Islice. The slices above the i-th slice (if any) are predictively encoded as Pslices. Moreover, when encoding any Pslices, the predictive scanning for those Pslices (in particular, a search for a motion vector) is limited in scope to the refreshed portion of the previous frame (an Islice and any Pslices above it). After an i-th refresh-frame has been encoded it is transmitted at step 210 and the iteration variable i is incremented until a refresh-frame with N (e.g., four) valid slices has been transmitted, such as the fourth refresh-frame 180D shown in Figure 6.

[0039] As the refresh-frames are transmitted, at step 212 the client receives the refresh-frames and decodes them in sequence until a fully valid frame has been

reconstructed, at which time the client 102 resumes receiving and decoding primarily ordinary Pframes at step 202.

[0040] In some implementations, the use of slices that are aligned from frame to frame can create striations artifacts; seams may appear at slice boundaries. This effect can be reduced with several techniques. Dithering with randomization of the intra-refresh slices can be used for smoothening. Put another way, instead of using ISlices, an encoder may encode different blocks as intra blocks in a picture. The spatial location of these blocks can be randomized to provide a better experience. To elaborate on the dithering technique, the idea is that, instead of encoding I-macroblocks consecutively upon a transmission error or the like, spread out the I-macroblocks across the relevant slice. This can help avoid the decoded image appearing to fill from top to bottom. Instead, with dithering, it will appear that the whole frame is getting refreshed. To the viewer it may look like the image is recovered faster. [0041] To optimize performance, conditions of the channel between the host 100 and the client 102 can be used to inform the intra-refresh encoding process. Parameters of intra-refresh encoding can be targeted to appropriately fit the channel or to take into account conditions on the channel such as noise, packet loss, etc. For instance, the compressed size of Islices can be targeted according to estimated available channel bandwidth. Slice QP (quantization parameter), and MB (macro-block) delta can be adjusted adaptively to meet the estimated target.

[0042] Figure 8 shows an example of a computing device 300. One or more such computing devices are configurable to implement embodiments described above. The computing device 300 comprises storage hardware 302, processing hardware 304, networking hardware 306 (e.g. network interfaces, cellular networking hardware, etc.). The processing hardware 304 can be a general purpose processor, a graphics processor, and/or other types of processors. The storage hardware can be one or more of a variety of forms, such as optical storage (e.g., compact-disk read-only memory (CD-ROM)), magnetic media, flash read-only memory (ROM), volatile memory, non-volatile memory, or other hardware that stores digital information in a way that is readily consumable by the processing hardware 304. The computing device 300 may also have a display 308, and one or more input devices (not shown) for users to interact with the computing device 300.

[0043] The embodiments described above can be implemented by information in the storage hardware 302, the information in the form of machine executable instructions (e.g., compiled executable binary code), source code, bytecode, or any other information that can be used to enable or configure the processing hardware to perform the various embodiments described above. The details provided above will suffice to enable practitioners of the invention to write source code corresponding to the embodiments, which can be compiled/translated and executed.