TRICK PLAY VIDEOS

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application is a non-provisional of, and claims benefit under 35 U.S.C. §119(e) from, U.S. Provisional Application Serial No. 62/279,587 filed January 15, 2016 titled "SYSTEM AND METHOD FOR OPERATING A VIDEO PLAYER DISPLAYING TRICK PLAY IN ADAPTIVE BIT RATE VIDEO," and U.S. Provisional Application Serial No. 62/279,569 filed January 15, 2016, titled "SYSTEM AND METHOD FOR TRICK PLAY IN ADAPTIVE BIT RATE VIDEO," both of which are incorporated herein by reference.

BACKGROUND

[0002] Digital video signals are commonly characterized by parameters of i) resolution (luma and chroma resolution or horizontal and vertical pixel dimensions), ii) frame rate, and iii) dynamic range or bit depth (bits per pixel). The resolution of the digital video signals has increased from Standard Definition (SD) through 8K-Ultra High Definition (UHD). The other digital video signal parameters have also increased, with frame rate increasing from 30 frames per second (fps) up to 240 fps and bit depth increasing from 8 bit to 10 bit. To transmit digital video signal over a network, MPEG/ITU standardized video compression has undergone several generations of successive improvements in compression efficiency, to include MPEG2, MPEG4/H.264, and HEVC/H.265. The technology to display the digital video signals on a consumer device, such as a television or mobile phone, has also increased correspondingly.

[0003] In capturing digital video, a camera (or other capture device) may have initially captured the streams at a much higher frame rate than the final playback rate. Currently, very high- rate capture systems are available. For example, there are capture cameras that can record streams at 240 frames per second (fps). However, most consumer delivery systems use playback frame rates of 60 fps (50 fps in Europe). With an increased frame-rate dimension, it is possible to play back the frames at a slower rate and create different slow motion effects. For example, if a stream captured at 240 fps is played back at a rate of 60 fps, a 4x slow motion effect results. With high frame-rate capture, it is possible to create zooming effects in the frame dimension.

[0004] A normal high frame rate conversion back to a normal distribution frame rate at the studio results in a loss of frame rate fidelity at the consumer. Today, the available high frame rate capture is used mostly to create special effect (like slow motion replays) at the studio. Studios can generate slow-motion content by using video captured at a high frame rate (e.g. 240 fps) and playing it back at a lower playback frame rate (e.g. 60 fps). However, such slow motion options are not available to consumers, who do not have access to the high-frame-rate video stream.

SUMMARY

[0005] Described herein are methods and systems for operating a video player, wherein a method includes the steps of playing a first stream of video content at a first playback frame rate, the first stream having a first captured frame rate, receiving a user selection of a slow-motion play mode, in response to the user selection, retrieving a second stream of the video content, the second stream having a second captured frame rate higher than the first captured frame rate, and playing the second stream at the first playback frame rate.

[0006] In some embodiments, retrieving the second stream includes selecting the second stream from among a plurality of available streams having different bitrates; wherein the first playback frame rate is the same as the first captured frame rate; the user selection of a slow-motion mode is a user selection of a selected one of a plurality of slow-motion modes, further comprising selecting the second stream from among a plurality of available streams based on the selected slow-motion mode; the plurality of slow-motion modes is presented to the video player via a manifest file.

[0007] In one such embodiment, the first stream is played back at a reduced playback frame rate in response to receiving the user selection while the second stream of the video content is being received. In such an embodiment, the second stream is played at the first playback frame rate subsequent to a predetermine amount of content being received for the second stream. The amount of content may correspond to a number of packets of the second stream, a number of segments of the second stream, or an amount of playback time of the second stream.

[0008] In another embodiment the method comprises, playing a first stream of video content at a first playback frame rate, the first stream having a first captured frame rate; receiving a user selection of a slow motion play mode; in response to the user selection: playing the first stream of video at a second playback frame rate lower than the first playback frame rate; retrieving a second stream of the video content, the second stream having a second captured frame rate higher than the first captured frame rate; and playing the second stream at the first playback frame rate.

[0009] In one such embodiment, the user selection of a slow-motion mode is a user selection of a selected one of a plurality of slow-motion modes, the method further comprising: selecting the second stream from among a plurality of available streams based on the selected slow-motion mode. In another embodiment, a switchover from playing the first stream to playing the second stream is a synchronized switchover. In one such embodiment, the synchronized switchover occurs subsequently to determining an amount of content received for the second stream exceeds a threshold. The amount of content corresponds to a number of frames of the second stream; a number of packets of the second stream, a number of segments of the second stream, or an amount of playback time of the second stream.

[0010] Another embodiment takes the form of a method comprising: playing a first stream of video content at a first playback frame rate, the first stream having a first captured frame rate; receiving a user selection of a slow motion replay mode, where the user selection is received at a first playback position in the video content; in response to the user selection: starting from a second point in the video content earlier than the first point, playing the first stream of video at a second playback frame rate slower than the first playback frame rate; retrieving a second stream of the video content, the second stream having a second captured frame rate higher than the first captured frame rate; and switching playback of the video content to playing of the second stream at a third playback frame rate.

[0011] In one such embodiment, the third playback frame rate is equal to the first playback frame rate. In another embodiment, the first playback frame rate and the third playback frame rate are different.

[0012] In another such embodiment, the user selection of a slow-motion mode is a user selection of a selected one of a plurality of slow-motion modes, further comprising: selecting the second stream from among a plurality of available streams based on the selected slow-motion mode.

[0013] In another such embodiment switching playback to the second stream is performed once a sufficient amount of the second stream has been received to perform a synchronized switch to the second stream.

[0014] In another such embodiment, the second point is a predetermined number of frames before the first playback position the second point is a predetermined number of seconds before the first playback position; a predetermined significant event in the video content, or an automatically determined significant event in the video content.

[0015] In another such embodiment, the first playback frame rate is the same as the first captured frame rate.

[0016] In another such embodiment, a switchover from playing the first stream to playing the second stream is a synchronized switchover.

[0017] Another embodiment takes the form of a method comprising: playing a first stream of video content at a first playback frame rate, the first stream having a first captured frame rate; receiving a user selection of a slow motion play-forward mode, where the user selection is received at a first point in the video content; in response to the user selection: starting from the first point in the video content, playing the first stream of video content at a second playback frame rate slower than the first playback frame rate; retrieving a second stream of the video content, the second stream having a second captured frame rate higher than the first captured frame rate; and switching play of the video content to playing of the second stream at a third playback frame rate.

[0018] In one such embodiment, the third playback frame rate is equal to the first playback frame rate. In another embodiment, the first playback frame rate and the third playback frame rate are different. In another embodiment, a switchover from playing the first stream to playing the second stream is a synchronized switchover. In another embodiment the user selection of a slow- motion mode is a user selection of a selected one of a plurality of slow-motion modes, further comprising: selecting the second stream from among a plurality of available streams based on the selected slow-motion mode.

[0019] In another such embodiment, switching playback to the second stream is performed once a sufficient amount of the second stream has been received to perform a synchronized switch to the second stream. In another embodiment, the first playback frame rate is the same as the first captured frame rate.

[0020] Another embodiment takes the form of a method comprising: receiving a manifest file, wherein the manifest file comprises playback speed factor information for a plurality of video streams, each of the video streams having a respective captured frame rate; identify a selected playback speed factor; and requesting one of the plurality of video streams associated with the selected playback speed factor.

[0021] In one such embodiment, the method further comprises playing the requested video stream at a first playback speed factor. In another such embodiment, the first playback speed factor is equal to the respective captured frame rate of the requested video stream. In another embodiment, the first playback speed factor is less than the respective captured frame rate of the requested video stream.

[0022] In another such embodiment, identifying the selected playback speed factor comprises receiving a user selection. In such an embodiment, the user selection corresponds to a user input via a user interface (UI) or an automatic selection.

[0023] In another such embodiment, identifying the selected playback factor comprises receiving an automatic selection. In such an embodiment, the automatic selection corresponds to identifying a significant event. Identifying a significant event comprises identifying an increased audio level at a venue, sentiment tracking of an end user, sentiment tracking at a venue, social media tracking, is based on identifying biometric measurements of an end user, or is based on crowd-sourced feedback.

[0024] Another embodiment takes the form of a method comprising: requesting a first stream having a first sampled frame-rate, the first stream part of a plurality of streams having respective sampled frame-rates; and receiving a manifest file comprising playback speed factors for the plurality of streams; selecting a stream having a slow-motion sampled frame-rate from the plurality of streams.

[0025] In such an embodiment, selecting the stream occurs in response to a user request, or in response to an automatic request. In another such embodiment, the selected stream begins playback at the point of selection.

[0026] Another embodiment take the form of a method of stream playout at preset speeds including a regular playout speed and other than regular playout speed, the method comprising: receiving information regarding availability of a plurality of representations of a portion of a video presentation, wherein the plurality of representations varies in at least frame rate; causing display of video derived from a first representation of the portion of a video presentation at a first normal frame rate associated with the first representation; and in response to receiving indication of user interaction with a user interface element associated with rewind-and-play-again-in-slow-motion functionality: requesting from the server, a second representation of the portion of a video presentation wherein the second representation is associated with a second normal frame rate higher than that of the first representation; and causing display of video derived from a second representation of the portion of a video presentation at a third normal frame rate lower than the second normal frame rate associated with the second representation.

[0027] In such an embodiment, the user may be presented with the first representation of the portion of a video presentation at the second normal frame rate until the second representation of the portion of a video presentation is available.

[0028] Another embodiment takes the form of a method of stream playout at preset speeds including a regular playout speed and other than regular playout speed, the method comprising: receiving information regarding availability of a plurality of representations of a portion of a video presentation, wherein the plurality of representations varies in at least frame rate; displaying video derived from a first representation of the portion of a video presentation at a first normal frame rate associated with the first representation; and in response to receiving an indication of user interaction with a user interface element associated with rewind-and-play-again-in-slow-motion functionality, the indication received at a current playback time: requesting from the server, a second representation of the portion of the video presentation wherein the second representation is associated with a second normal frame rate higher than that of the first representation; causing display of video derived from the second representation of the portion of the video presentation at a third normal frame rate lower than the second normal frame rate associated with the second representation.

[0029] In one such embodiment, the requested second representation of the portion of the video presentation begins at an earlier playback time with respect to the current playback time. In such an embodiment, the earlier playback time may be provided by user or determined based on a past significant event, associated with last time instance at which one or more significant events occur. The one or more significant events includes detecting an audio level above a known threshold, or detecting user sentiment is above a known threshold.

[0030] In another such embodiment, the request for the second representation of the portion of the video presentation is associated with a request for pause and further comprising the step of requesting the portion of the video presentation corresponding to the second normal frame rate while in the paused state.

[0031] In another such embodiment, the user may be presented with the first representation of the portion of the video presentation at a second normal frame rate until a buffer receiving the second representation of the portion of the video presentation is available.

[0032] Another embodiment takes the form of a method of serving video content, the method comprising: providing a manifest file to a client, wherein the manifest file identifies a plurality of streams representing a video presentation at least two of the streams having different sampled frame rates, and wherein the manifest file identifies at least one stream associated with a slow motion mode.

[0033] In such an embodiment, the manifest file is a DASH Media Presentation Description (MPD) file, or an HLS manifest file. In another embodiment, the method further comprises providing segments of frames of the plurality of streams.

[0034] In another such embodiment, the manifest file provides slow-motion factors for each stream of the plurality of streams or recommended playback rates for slow-motion factors for each stream of the plurality of streams.

[0035] Another embodiment takes the form of a method for stream playout at different speeds using streams of different frame rates, the method comprising: encoding a plurality of versions of a portion of a video program, wherein each version has a different version base encoding frame rate; creating, for each version of the encoded plurality of versions of the portion of the video program with different version base encoding frame rate, a plurality of entries in a manifest file, wherein each entry in the manifest file of the plurality of entries corresponding to the version comprises information regarding a client playout frame rate; and a client relative presentation speed, such that the ratio of the client playout frame rate and the client relative presentation speed is the base encoding frame rate for the version; receiving from a client a request for a portion associated with a manifest entry; and sending the requested portion.

[0036] In one such embodiment, the client plays the frames of the version at the playout frame rate. In another such embodiment, client relative presentation speed is one of l/2x, lx, and 2x. In another such embodiment, the client playout frame rate is one of 30 fps and 60 fps.

[0037] In another such embodiment, the manifest file identifies a plurality of streams associated with the slow motion mode, the plurality of streams associated with the slow motion mode having different bitrates. In another such embodiment, the version with a lowest version base encoding frame rate corresponds to a normal playback version.

[0038] Another embodiment takes the form of a method for stream playout at different speeds using streams of different frame rates, the method comprising: encoding a plurality of video streams with different versions, where each version has different sampled frame rate; creating, for each version with different sampled frame rate, a plurality of entries in a manifest file; wherein each entry in the manifest file is tagged with supported sampled frame rate representation and the playout frame rate, such that each entry may correspond to a combination of supported sampled frame rate representation and playout frame rate; responsive to a user request, selecting a stream from among the plurality of streams to service the request, wherein the selection is made from among different bit rate versions of video streams dependent on network bandwidth conditions.

[0039] Another embodiment takes the form of a method comprising: providing a manifest file, wherein the manifest file comprises playback speed factor information for one or more streams. The method may further comprise providing a stream in response to a stream-request made by a client device

[0040] Another embodiment takes the form of a method comprising: creating a plurality of entries in a manifest file, each entry corresponding to a respective stream of a plurality of streams, the respective stream having a respective sampled frame-rate, and wherein each entry comprises playback speed factor information for the respective stream; and transmitting the manifest file to a client device.

[0041] In one such embodiment, each stream of the plurality of streams has a corresponding frame rate, and wherein each stream of the plurality of streams is separately encoded. In such an embodiment, the encoding corresponds to adaptive bitrate (ABR) encoding. In another such embodiment the method further comprises transmitting all of the encoded streams to a client device. [0042] In another such embodiment, a full rate stream is encoded, and subsampled frame rate streams are generated by selecting frames of the full rate stream. In such an embodiment, the subsampled frame rate streams are generated in accordance with a hierarchical group of pictures (GOP). In another such embodiment, the method further comprises transmitting the encoded full rate stream. In another such embodiment, the method further comprises transmitting a subsampled frame rate stream generated from the encoded full rate stream.

BRIEF DESCRIPTION OF THE DRAWINGS

[0043] A more detailed understanding may be had from the following description, presented by way of example in conjunction with the accompanying drawings, wherein:

[0044] FIG. 1A depicts an example communications system in which one or more disclosed embodiments may be implemented.

[0045] FIG. IB depicts an example client device that may be used within the communications system of FIG. 1A.

[0046] FIG. 2 depicts a flowchart of an independent-encoding technique, in accordance with some embodiments.

[0047] FIG. 3 depicts a flowchart of an independent-encoding technique, in accordance with some embodiments.

[0048] FIG. 4 depicts a flowchart of adaptive bitrate (ABR) encoding, in accordance with some embodiments.

[0049] FIG. 5 depicts a flowchart of a DASH messaging flow, in accordance with some embodiments.

[0050] FIGs. 6A-6C depict various representations (also referred to herein as profiles) received in a Media Presentation Description (MPD) file, including recommended playback rates and various bitrates BitRate-A, BitRate-B, and BitRate-C, in accordance with some embodiments.

[0051] FIG. 7A depicts an example of encoding utilizing a hierarchical group of pictures (GOP), in accordance with some embodiments.

[0052] FIG. 7B depicts an example of a hierarchical GOP, in accordance with some embodiments.

[0053] FIG. 8 depicts an example of encoding using SVC techniques, in accordance with some embodiments.

[0054] FIG. 9 depicts a flowchart of a modified delivery mechanism to account for the slow motion streams that may be additionally requested by the client, in accordance with some embodiments. [0055] FIG. 10 depicts an example user interface (UI), in accordance with some embodiments.

DETAILED DESCRIPTION

[0056] A detailed description of illustrative embodiments will now be provided with reference to the various Figures. Although this description provides detailed examples of possible implementations, it should be noted that the provided details are intended to be by way of example and in no way limit the scope of the application. The systems and methods relating to video compression may be used with the wired and wireless communication systems described with respect to FIGS. 1 A- IB. As an initial matter, these wired and wireless systems will be described.

[0057] FIG. 1 A is a diagram of an example communications system 100 in which one or more disclosed embodiments may be implemented. The communications system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, and the like, to multiple wireless users. The communications system 100 may enable multiple wired and wireless users to access such content through the sharing of system resources, including wired and wireless bandwidth. For example, the communications systems 100 may employ one or more channel-access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), and the like. The communications systems 100 may also employ one or more wired communications standards (e.g.: Ethernet, DSL, radio frequency (RF) over coaxial cable, fiber optics, and the like.

[0058] As shown in FIG. 1A, the communications system 100 may include client devices 102a, 102b, 102c, and/or 102d, Radio Access Networks (RAN) 103/104/105, a core network 106/107/109, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, and communication links 115/116/117, and 119, though it will be appreciated that the disclosed embodiments contemplate any number of client devices, base stations, networks, and/or network elements. Each of the client devices 102a, 102b, 102c, 102d may be any type of device configured to operate and/or communicate in a wired or wireless environment. By way of example, the client device 102a is depicted as a tablet computer, the client device 102b is depicted as a smart phone, the client device 102c is depicted as a computer, and the client device 102d is depicted as a television.

[0059] The communications systems 100 may also include a base station 114a and a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the core network 106/107/109, the Internet 110, and/or the networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and/or network elements.

[0060] The base station 114a may be part of the RAN 103/104/105, which may also include other base stations and/or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, and the like. The base station 114a and/or the base station 114b may be configured to transmit and/or receive wireless signals within a particular geographic region, which may be referred to as a cell (not shown). The cell may further be divided into sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one for each sector of the cell. In another embodiment, the base station 114a may employ multiple- input multiple output (MIMO) technology and, therefore, may utilize multiple transceivers for each sector of the cell.

[0061] The base stations 114a, 1 14b may communicate with one or more of the client devices 102a, 102b, 102c, and 102d over an air interface 115/116/117, or communication link 119, which may be any suitable wired or wireless communication link (e.g., radio frequency (RF), microwave, infrared (IR), ultraviolet (UV), visible light, and the like). The air interface 115/116/117 may be established using any suitable radio access technology (RAT).

[0062] More specifically, as noted above, the communications system 100 may be a multiple access system and may employ one or more channel-access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 103/104/105 and the client devices 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 115/116/117 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and/or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink Packet Access (HSDPA) and/or High-Speed Uplink Packet Access (HSUPA).

[0063] In another embodiment, the base station 114a and the client devices 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 115/116/117 using Long Term Evolution (LTE) and/or LTE- Advanced (LTE- A). [0064] In other embodiments, the base station 114a and the client devices 102a, 102b, 102c may implement radio technologies such as IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 IX, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0065] The base station 114b in FIG. 1 A may be a wired router, a wireless router, Home Node B, Home eNode B, or access point, as examples, and may utilize any suitable wired transmission standard or RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, and the like. In one embodiment, the base station 114b and the client devices 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, the base station 114b and the client devices 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the client devices 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, and the like) to establish a picocell or femtocell. In yet another embodiment, the base station 114b communicates with client devices 102a, 102b, 102c, and 102d through communication links 119. As shown in FIG. 1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not be required to access the Internet 110 via the core network 106/107/109.

[0066] The RAN 103/104/105 may be in communication with the core network 106/107/109, which may be any type of network configured to provide voice, data, applications, and/or voice over internet protocol (VoIP) services to one or more of the client devices 102a, 102b, 102c, 102d. As examples, the core network 106/107/109 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, and the like, and/or perform high-level security functions, such as user authentication. Although not shown in FIG. 1A, it will be appreciated that the RAN 103/104/105 and/or the core network 106/107/109 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 103/104/105 or a different RAT. For example, in addition to being connected to the RAN 103/104/105, which may be utilizing an E-UTRA radio technology, the core network 106/107/109 may also be in communication with another RAN (not shown) employing a GSM radio technology.

[0067] The core network 106/107/109 may also serve as a gateway for the client devices 102a,

102b, 102c, 102d to access the PSTN 108, the Internet 110, and/or other networks 112. The PSTN 108 may include circuit-switched telephone networks that provide plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and IP in the TCP/IP Internet protocol suite. The networks 112 may include wired and/or wireless communications networks owned and/or operated by other service providers. For example, the networks 112 may include another core network connected to one or more RANs, which may employ the same RAT as the RAN 103/104/105 or a different RAT.

[0068] Some or all of the client devices 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities, i.e., the client devices 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wired or wireless networks over different communication links. For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0069] FIG. IB depicts an example client device that may be used within the communications system of FIG. 1 A. In particular, FIG. IB is a system diagram of an example client device, or access terminal, 102. As shown in FIG. IB, the client device 102 may include a processor 118, a transceiver 120, a transmit/receive element 122, a speaker/microphone 124, a keypad 126, a display/touchpad 128, a non-removable memory 130, a removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and other peripherals 138. It will be appreciated that the client device 102 may represent any of the client devices 102a, 102b, 102c, and 102d, and include any sub-combination of the foregoing elements while remaining consistent with an embodiment. Also, embodiments contemplate that the base stations 114a and 114b, and/or the nodes that base stations 114a and 114b may represent, such as but not limited to transceiver station (BTS), a Node-B, a site controller, an access point (AP), a home Node-B, an evolved home Node-B (eNodeB), a home evolved Node-B (HeNB), a home evolved Node-B gateway, and proxy nodes, among others, may include some or all of the elements depicted in FIG. IB and described herein.

[0070] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application

Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 1 18 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the client device 102 to operate in a wired or wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit/receive element 122. While FIG. IB depicts the processor 118 and the transceiver 120 as separate components, it will be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0071] The transmit/receive element 122 may be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 114a) over the air interface 115/116/117 or communication link 119. For example, in one embodiment, the transmit/receive element 122 may be an antenna configured to transmit and/or receive RF signals. In another embodiment, the transmit/receive element 122 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, as examples. In yet another embodiment, the transmit/receive element 122 may be configured to transmit and receive both RF and light signals. In yet another embodiment, the transmit/receive element may be a wired communication port, such as an Ethernet port. It will be appreciated that the transmit/receive element 122 may be configured to transmit and/or receive any combination of wired or wireless signals.

[0072] In addition, although the transmit/receive element 122 is depicted in FIG. IB as a single element, the client device 102 may include any number of transmit/receive elements 122. More specifically, the client device 102 may employ MFMO technology. Thus, in one embodiment, the

WTRU 102 may include two or more transmit/receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 115/116/117.

[0073] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit/receive element 122 and to demodulate the signals that are received by the transmit/receive element 122. As noted above, the client device 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the client device 102 to communicate via multiple RATs, such as UTRA and IEEE 802.11, as examples.

[0074] The processor 118 of the client device 102 may be coupled to, and may receive user input data from, the speaker/microphone 124, the keypad 126, and/or the display/touchpad 128

(e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker/microphone 124, the keypad

126, and/or the display/touchpad 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and/or the removable memory 132. The non-removable memory 130 may include random-access memory

(RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SFM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the client device 102, such as on a server or a home computer (not shown).

[0075] The processor 118 may receive power from the power source 134, and may be configured to distribute and/or control the power to the other components in the client device 102. The power source 134 may be any suitable device for powering the WTRU 102. As examples, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel- zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li -ion), and the like), solar cells, fuel cells, a wall outlet and the like.

[0076] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the client device 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 115/116/117 from a base station (e.g., base stations 114a, 114b) and/or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the client device 102 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment. In accordance with an embodiment, the client device 102 does not comprise a GPS chipset and does not acquire location information.

[0077] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, and the like.

[0078] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer- readable medium for execution by a computer or processor. Examples of computer-readable media include electronic signals (transmitted over wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

[0079] FIG. 2 depicts a flowchart of an independent-encoding technique, in accordance with an embodiment. In an exemplary embodiment, an independent-encoding technique may be used to support multiple slow motion (or trick mode) options. In some embodiments, multiple subsample frame rates may be derived from a source video 202 captured high frame-rate (also referred to herein as a full-rate stream). In the various embodiments described herein, the source video may be stored on any suitable device, such as a network, a cache server, a streaming server, a client device, or the like. As shown in FIG. 2, the system derives a plurality (e.g. a number N) of subsampled (e.g. temporally subsampled) streams 204, with each subsampled stream having a different sampled frame rate. In some embodiments, at least one of the N subsampled streams is adaptive bit rate (ABR) encoded into a plurality of different streams 2016 with different bitrates, and the different encoded streams are made available via an IP network 208, the respective communication paths 210A-C to end user's client devices 212A-C. In some embodiments, FIG. 4 illustrates a method of ABR encoding. As shown, the ABR encoder 404 receives an input video stream 402 (which may be the full-rate video stream, or any of the subsampled streams), and generates various representations encoding those streams, depicted as representations 1, 2, ... n with each representation having various resolutions, frame rates, and bitrates. As shown, the different streams may have various different resolutions, frame rates, and bit rates associated with them (see FIGs. 6A-6C for an exemplary embodiment of profiles). The streams encoded into different representations are presented to a streaming server 406 which may deliver the streams to a client 412A-C (such as a compatible video player) via an internet protocol (IP) network 408 via respective communication paths 410A-C.

[0080] In some embodiments, each of the subsampled streams is a different channel that could be presented to the client device as a different playback speed factor (e.g. ½ speed slow motion, ¼ speed slow motion, and the like). In some embodiments, the client device may be configured to retrieve and decode a video stream with a sampled frame rate that corresponds to a selected slow motion playback speed factor in response to receiving a user selection of that playback speed factor. In some embodiments, the client may be configured to select from the N available sampled frame rates. In some embodiments, a user may return to the normal playback rate by simply selecting a normal playback rate option via the client player (e.g., via a user interface, see FIG.

10). At the normal playback rate, the sampled frame rate is substantially equal to the playback frame rate. At, for example, slow motion playback with a playback speed factor of ½, the sampled frame rate is substantially twice as great as the playback frame rate. FIG. 3 depicts an exemplary embodiment utilizing independent-encoding techniques. In a first example, the camera captures a full-rate stream 302 at 240 fps. In some embodiments, a plurality of additional subsampled streams is generated by subsampling the full-rate stream. In FIG. 3, the additional subsampled streams 304 are generated with sampled frame rates including 120 fps and 60 fps, and any other varying frame rate. In some embodiments, each subsampled stream is ABR encoded (at 306) to support adaptive delivery over an IP network 308 to client devices 312A-C via respective communication paths

3 lOA-C. (For example, a given subsampled stream may be encoded to a plurality of bit rates at the already subsampled frame rate in order to support bit rate adaptation at that frame rate). One of the subsampled frame rates may be considered to be a 'normal' frame rate. For example, some embodiments may use a playback frame rate of 60 fps for normal playback, in which case a stream that has a sampled frame rate of 60 fps may be referred to as a "normal stream." If such embodiments use adaptive bit rate (ABR) encoding, all streams with a sampled frame rate of 60 fps may be referred to as normal streams. One or more normal streams may be generated by, for example, selecting every fourth frame of a 240 fps full-rate stream. In some embodiments, the normal stream may also be derived from a higher framerate stream by standard frame interpolation methods. It should be noted that there is no restriction that the normal stream has to be an integer subsample framerate of the full framerate stream. The frame rate corresponding to normal playback may vary based on client device capabilities, user preference, or other factors. A server may offer streams at multiple different subsampled frame rates, and may allow each user or client device to select which of these subsampled frame rates should be used for normal playback.

[0081] In some embodiments, a video player is informed that different streams are available to accommodate different playback speed factors. For example, with a player using a normal playback frame rate of 60 fps, slow motion playback with a playback speed factor of ½ may be accommodated by a stream having a sampled frame rate of 120 fps. Similarly, slow motion playback with a playback speed factor of ¼ may be accommodated by a stream having a sampled frame rate of 240 fps. In some embodiments, the original DASH or HLS manifest provided to the player via the server includes available stream information via an extension. In some embodiments, when the player requests the normal stream (e.g., the 60 fps stream), the player receives the normal stream and a corresponding MPD (indicating the available ABR variants). In some embodiments, two modes of operation are available. In a first mode, when the channel is requested, an MPD is received that carries the full description (including a "default" or "normal" channel description that is labeled as such. The client device requests playback of the "normal" channel and also presents to the end user the availability of the frame rate zoom features. In a second mode, the MPD contains all options. Based on the MPD, the end user is presented with a set of choices including the "normal" playback rate channel. The end user then chooses the mode, and only then streaming commences. In addition, the manifest could identify (in an extension) the available slow-motion sequences. The manifest may indicate the subsampled frame rates for each stream. Furthermore, the manifest may indicate effective slow motion and/or fast motion playback speed factors which correspond to the streams with different subsampled frame rates. The manifest may indicate multiple playback speed factors for a stream, which may correspond to different values of a "normal playback" framerate which may be used by a playback device. In some embodiments, the player displays on the video playback screen information that identifies the available playback speed factors (e.g. ½ speed and ¼ speed slow motion playback). These playback speed factors may be displayed in response to receiving a request from the user, as will be described below. In some embodiments, the playback speed factor may be displayed as a frame- rate zooming factor: for example, a playback speed factor of ½ may correspond to a frame-rate zooming factor of 2x. In this disclosure, "zooming" refers to the sense of viewing at a higher level of detail (e.g. slow motion effect created by a higher capture rate and a slower playback rate), rather than the sense of moving quickly.

[0082] The video player may be informed of the availability of different streams for different slow-motion or other trick play modes through the use of an HTTP DASH media presentation description (MPD) file or other manifest file. FIG. 5 depicts a flowchart of a DASH messaging flow, in accordance with some embodiments. FIG. 5 depicts an MPD provider 502, a DASH server 504, and a DASH client 506. At 508, the DASH client 506 requests an MPD from the MPD provider 502, and the MPD provider 502 responsively provides the MPD to the DASH client 506 at 510. At 512, 4he DASH client 506 requests segments from the DASH server 504, and at 514, the DASH server responsively provides the media segments to the DASH client 514. This process may repeat at 516 and 518.

[0083] The contents of an exemplary MPD file are as follows:

<?xml version="1.0" encoding="UTF-8"?>

<MPD

<Program Information

<Title>Example of a DASH Media Presentation Description using Spatial Relationship Description to indicate that a video is a zoomed part of another</Title>

</Programlnformation>

<Period>

<!— Video with 4 frame rates ->

<AdaptationSet segmentAlignment- 'true" subsegmentAlignment- 'true" subsegmentStartsWithSAP="l"> <Role schemeldUri="urn:mpeg:dash:role:2011" value="main"/>

<SupplementalProperty schemeldUri="urn:mpeg:dash :rtvideoparam:2014" value="1920,1080,29.97"/> Representation mimeType="video/mp4" codecs="avcl.42c033" width="1920" height="1080" bandwidth="1055223" maxplayoutrate="l" startWithSAP="l">

<BaseURL> panorama_videol.mp4</BaseURL>

<SegmentBase indexRangeExact="true" indexRange="839-990"/>

</Representation>

<Representation mimeType="video/mp4" codecs="avcl.42c033" width

bandwidth="1055223" maxplayoutrate="2" startWithSAP="l">

<BaseURL> panorama_video2.mp4</BaseURL>

<SegmentBase indexRangeExact="true" indexRange="839-990"/>

</Representation>

<Representation mimeType="video/mp4" codecs="avcl.42c033" width

bandwidth="1055223" maxplayoutrate="3" startWithSAP="l">

<BaseURL> panorama_video3.mp4</BaseURL>

<SegmentBase indexRangeExact="true" indexRange="839-990"/>

</Representation>

<Representation mimeType="video/mp4" codecs="avcl.42c033" width

bandwidth="1055223" maxplayoutrate="4" startWithSAP="l">

<BaseURL> panorama_video4.mp4</BaseURL>

<SegmentBase indexRangeExact="true" indexRange="839-990"/>

</Representation>

<Representation mimeType="video/mp4" codecs="avcl.42c033" width

bandwidth="1055223" maxplayoutrate="0.5" startWithSAP="l">

<BaseU RL> panorama_video5.mp4</BaseU RL>

<SegmentBase indexRangeExact="true" indexRange="839-990"/>

</Representation>

<Representation mimeType="video/mp4" codecs="avcl.42c033" width

bandwidth="1055223" maxplayoutrate="0.25" startWithSAP="l">

<BaseU RL> panorama_video6.mp4</BaseU RL>

<SegmentBase indexRangeExact="true" indexRange="839-990"/>

</Representation>

</AdaptationSet>

[0084] In the exemplary MPD, the SupplementalProperty tag is used to identify the frame rate to be used for normal-speed playback. For example, in the example above, the value "1920, 1080,29.97" indicates that the relevant video has the width of 1920 pixels, a height of 1080 pixels, and that the sampled frame rate to be used for normal-speed playback is 29.97 fps (often referred to as 30 fps). The value of the "maxplayoutrate" parameter of each representation identifies the slow motion factor that results from playing that representation at the normal-speed playback rate. For example, the representation "panorama_video2.mp4" has a maxplayoutrate of 2, indicating that playing panorama_video2.mp4 at 29.97 fps will result in a playback speed factor of ½. The representation "panorama_video3.mp4" has a maxplayoutrate of 3, indicating that playing panorama_video3.mp4 at 29.97 fps will result in a playback speed factor of 1/3. In some embodiments, a manifest file may provide information regarding a plurality of different representations, where each representation may have a different normal-speed playback frame rate.

[0085] In some embodiments, a player capable of playing video at a high playback frame rate such as 120 fps or 240 fps may request streams with sampled frame rates of 120 fps or 240 fps and play back those streams at their respective native frame rates. [0086] In some embodiments, a server provides information identifying, for each available playback speed factor, which stream should be used and which playback frame rate should be used for that stream. This information may be provided in a manifest file. As an example, a particular stream "panorama_videol20.mp4" may have a sampled frame rate of 120 fps. A manifest file may indicate that, to obtain a playback speed factor of ½, the stream "panorama_videol20.mp4" should be played back at a playback frame rate of 60 fps. Where 60 fps is the default frame rate, it may not be necessary to specify the playback frame rate. As another example, a particular stream

"panorama_video240.mp4" may have a sampled frame rate of 240 fps. A manifest file may indicate that, to obtain a playback speed factor of ½, the stream "panorama_video240.mp4" should be played back at a playback frame rate of 120 fps. The manifest file may also provide information indicating that, to obtain a playback speed factor of ¼, the stream "panorama_video240.mp4" should be played back at a playback frame rate of 60 fps. In other embodiments, the server does not expressly identify which streams should be used to obtain particular playback speed factors.

In such embodiments, a manifest file may simply identify the sampled frame rates of the respective available streams, and the video player may determine which stream to fetch and which playback frame rate should be used for each playback speed factor.

[0087] FIGs. 6A-6C depict various representations (also referred to herein as profiles), including recommended playback rates and various bitrates BitRate-A, BitRate-B, BitRate-C), in accordance with some embodiments. Each profile depicted in FIGs. 6A-6C may correspond to a different normal playback framerate, and each profile may identify and describe various options for trick mode playback (e.g. slow motion at different playback rates) that are compatible with the normal playback framerate of the profile.

[0088] FIG. 6A illustrates the contents of an exemplary manifest file. The manifest file of FIG.

6A includes a 'PROFILE A' which has a normal playback framerate of 30 fps. A client device that uses or selects a normal playback framerate of 30 fps may retrieve any of the ABR streams with sampled framerate of 30 fps, and may play such streams at the normal playback framerate of 30 fps in order to achieve normal playback. 'PROFILE A' also includes information identifying the availability of trick mode options for 2x Slow Motion, 4x Slow Motion, and 8x Slow Motion

(corresponding to the available streams with sampled framerates of 60 fps, 120 fps, and 240 fps, respectively). For example, a client may request any of the ABR streams with sampled framerate of 120 fps and may play back such streams at a playback framerate of 30 fps in order to achieve a

4x Slow Motion effect, as indicated by the '4x Slow' label shown for PROFILE A in association with the streams having a sampled frame rate of 120 fps. As another example, the manifest file of

FIG. 6A includes information for a 'PROFILE C which has a normal playback framerate of 120 fps. A client device that uses or selects a normal playback framerate of 120 fps may retrieve any of the ABR streams with sampled framerate of 120 fps and may play such streams at the normal playback framerate of 120 fps in order to achieve normal playback. 'PROFILE C also provides information on the availability of a trick mode option for 2x Slow Motion for the streams with sampled framerate of 240 fps. For example, the client may request any of the ABR streams with sampled framerate of 240 fps and may play back such streams at a playback framerate of 120 fps in order to achieve a 2x Slow Motion effect, as indicated by the '2x Slow' label shown for PROFILE C in association with the streams having a sampled frame rate of 120 fps. The profile definitions and tags illustrated in FIG. 6A may be used to label the streams at different bit rates and sampled frame rates in order to indicate to a client device (or to a user) which streams should be played at which playback frame rates in order to achieve a particular trick mode effect, given the normal playback rate that a particular client device may be using. The profile definitions and tags illustrated in FIG. 6A may be conveyed to a client device in a manifest file such as a DASH MPD, for example.

[0089] FIG. 6B illustrates the contents of another exemplary manifest file. The manifest file of FIG. 6B includes information on a 'PROFILE A' which has a normal playback framerate of

30 fps. A client device that uses or selects a normal playback framerate of 30 fps may retrieve any of the ABR streams with sampled framerate of 30 fps and may play such streams at the normal playback framerate of 30 fps in order to achieve normal playback. 'PROFILE A' also includes information identifying the availability of trick mode options for 2x Slow Motion, 4x Slow Motion, and 8x Slow Motion (corresponding to the available streams with sampled framerates of 60 fps,

120 fps, and 240 fps, respectively). For example, a client may request any of the ABR streams with sampled framerate of 120 fps and may play back such streams at a playback framerate of

30 fps in order to achieve a 4x Slow Motion effect, as indicated by the '4x Slow' label shown for

PROFILE A in association with the streams having a sampled frame rate of 120 fps. As another example, FIG. 6B includes information on a 'PROFILE C which has a normal playback framerate of 120 fps. A client device that uses or selects a normal playback framerate of 120 fps may retrieve any of the ABR streams with sampled framerate of 120 fps and may play such streams at the normal playback framerate of 120 fps in order to achieve normal playback. 'PROFILE C also includes information identifying the availability of a trick mode option for 4x Fast Motion for the streams with sampled framerate of 30 fps, 2x Fast Motion for the streams with sampled framerate of 60 fps, and 2x Slow Motion for the streams with sampled framerate of 240 fps. For example, the client may request any of the ABR streams with sampled framerate of 240 fps and may play back such streams at a playback framerate of 120 fps in order to achieve a 2x Slow Motion effect, as indicated by the '2x Slow' label shown for PROFILE C in association with the streams having a sampled frame rate of 120 fps. Further, the client may request any of the ABR streams with sampled framerates of 30 fps or 60 fps and may play back such streams at a playback framerate of

120 fps in order to achieve 4x Fast Motion and 2x Fast Motion effects, respectively. The profile definitions and tags illustrated in FIG. 6B may be used to label the streams at different bit rates and sampled frame rates in order to indicate to a client device (or to a user) which streams should be played at which playback frame rates in order to achieve a particular trick mode effect, given the normal playback rate which a particular client device may be using. The profile definitions and tags illustrated in FIG. 6B may be conveyed to a client device in a manifest file such as a DASH

MPD, for example.

[0090] FIG. 6C illustrates the contents of another exemplary manifest file. The manifest file of FIG. 6C includes information on a 'PROFILE A' which has maximum playback framerate of

30 fps. A client device which uses or selects a playback framerate of 30 fps may retrieve any of the ABR streams with sampled framerate of 30 fps, and may play such streams at a normal playback framerate of 30 fps in order to achieve normal playback. 'PROFILE A' also includes information identifying the availability of trick mode options for 1.5x Slow Motion, 4x Slow

Motion, 8x Slow Motion, and 12x Slow Motion. The 1.5x Slow motion may correspond to an available stream with a sampled framerate of 60 fps played back at a playback rate of 22.5 fps. 4x

Slow Motion may correspond to an available stream with a sampled framerate of 120 fps played back at a playback rate of 30 fps. 8x Slow Motion may correspond to an available stream with a sampled framerate of 240 fps played back at a playback rate of 30 fps. 12x Slow Motion may correspond to an available stream with a sampled framerate of 240 fps played back at a playback rate of 20 fps. For example, a client may request any of the ABR streams with sampled framerate of 120 fps and may play back such streams at a playback framerate of 30 fps in order to achieve a

4x Slow Motion effect, as indicated by the '4x Slow' label shown for PROFILE A in association with the streams having a sampled frame rate of 120 fps. As another example, the manifest file of

FIG. 6C includes information identifying the availability of a 'PROFILE C which has a maximum playback framerate of 120 fps. A client device that uses or selects a normal playback framerate of

120 fps may retrieve any of the ABR streams with sampled framerate of 120 fps and may play such streams at a normal playback framerate of 120 fps in order to achieve normal playback.

'PROFILE C also provides information on the availability of a trick mode option for 2x Slow

Motion and 4x Slow Motion for the streams with sampled framerate of 240 fps. For example, the client may request any of the ABR streams with sampled framerate of 240 fps, and may play back such streams at a playback framerate of 120 fps in order to achieve a 2x Slow Motion effect, as indicated by the '2x Slow' label shown for PROFILE C in association with the streams having a sampled frame rate of 240 fps. Further, the client may request any of the ABR streams with sampled framerates of 240 fps, and may play back such streams at a playback framerate of 60 fps in order to achieve 4x Slow Motion. The profile definitions and tags illustrated in FIG. 6C may be used to label the streams at different bit rates and sampled frame rates in order to indicate to a client device (or to a user) which streams should be played at which playback frame rates in order to achieve a particular trick mode effect, given the normal playback rate that a particular client device may be using. The profile definitions and tags illustrated in FIG. 6C may be conveyed to a client device in a manifest file such as a DASH MPD, for example.

[0091] In some embodiments, the full-rate stream is encoded once at the captured frame rate (or other high sampled frame rate), and streams with lower sampled frame rates are generated from this encoded stream on an as-needed basis. Such an embodiment is depicted in FIG. 7A. FIG. 7A depicts the flowchart 700 that includes source video captured at a high frame rate 702, a hierarchical group of pictures (GOP) structure 704, ABR representations 706, an IP network 708, communication paths 710A-C to respective clients 712A-C. In such embodiments, the full-rate stream may be encoded with a hierarchical group of pictures (GOP) structure 704, and streams 706 with lower frame rates may be extracted from the full-rate stream 702 as necessary. Such embodiments may be more efficient than the independent-encoding technique as there are not multiple ABR encodes of the same stream.

[0092] FIG. 7B depicts an example hierarchical GOP. In some embodiments, the original captured stream at the highest frame rate (e.g., 240 fps) may be tagged into a hierarchical GOP frame structure. As another example, the frame sequence could have the following structure:

I-B-Bhier-B-P-B-Bhier-B-P-B-Bhier-B-P-B-Bhier-B-I

[0093] where:

I: Intra Frame (Black frame in FIG. 7B)

B: Normal Bipredictive frame (White Frame in FIG. 7B)

Bhier: Hierarchical B (Diagonal Frame in FIG. 7B)

P: Predictive Frame (Cross-hatch Frame in FIG. 7B)

[0094] In some embodiments, such sequences may be created in arbitrary combinations compliant with the video compression standard specification to suit trick play features.

[0095] In some embodiments, starting with the original full frame, extraction only of the Intra

(I) and Predictive (P) only frames may generate a stream with a captured frame rate of 60 fps, corresponding to the "normal" rate, while the inclusion of Bhier frames creates a trick-play version

(with a capture frame rate of, e.g. 120 fps), and the addition of B frames creates a second trick- play version (with a capture frame rate of, e.g. 240 fps). In some embodiments, the captured frame rates may be selected to suit the application.

[0096] In some embodiments, the full-rate stream is only ABR encoded once (e.g., the version with a sampled frame rate of 240 fps) as using the full complement of I, P, B and Bhier frame types. The encoded full-rate stream may be used to present video with the lowest available playback speed factor. Subsampled streams (for example choosing I and P frames only) may represent the "normal coded" stream (e.g. the 60 fps version), while choosing I, P and Bhier frames could generate a first enhanced version with a higher sampled frame rate (e.g. 120 fps), and choosing I, P, Bhier and B could generate a second enhanced version with a still higher sampled frame rate (e.g. 240 fps). In some embodiments, another level in the hierarchy may be created by choosing I frames only to generate a stream with a lower sampled frame rate (e.g. 30 fps). Selecting I frames only may enable a "fast-forward" mode in some embodiments, for example where a stream with a sampled frame rate of 30 fps is played back at 60 fps. Alternatively, an I frame only stream may be used as the "normal coded" stream, and the versions described above could provide three levels of slow motion. After a full-rate stream is encoded once using the full complement of I, P, B and Bhier frame types, the already encoded frames may be extracted from the stream and repackaged to produce the enhanced streams with different subsampled frame rates, without the need to re-encode the individual frames to produce each subsampled frame rate stream.

[0097] In some embodiments, ABR packaging is performed (either on the fly or ahead of time depending on whether the content is live or a video on-demand (VOD)) for the normal, and enhanced streams independently. In some embodiments, only the I and P frame coded versions (at various ABR bit rates) are packaged in the "normal" ABR stream, while the first enhanced version is packaged in a second set of ABR packages, and the second enhanced version is packaged in a third set of ABR packages.

[0098] In some embodiments, rate control for encoding the various representations in the ABR streams is done at a GOP level with bitrate allocation done to various frames (e.g. I, P, B etc.). In some embodiments, each stream may use a subset of the frames of the full-rate stream. In some embodiments, the stream ABR rates for each representation in the subset cases are determined by the coding rate of the frames in the full-rate stream. When the various representations are generated for the full-rate stream, the representation bitrates for the other subsampled (slow motion and normal) streams are automatically set.

[0099] For example, the 240 fps stream may be encoded into four different ABR representations with bitrates as follows: 15 Mbps, 8 Mbps, 4 Mbps and 2 Mbps (possibly with different corresponding resolution choices). Further, the 60 fps streams at different ABR representations may be derived in some embodiments by selecting and repackaging the Intra (I) and Predictive (P) frames from the encoded 240 fps streams. In some embodiments, the encoder is configured to automatically allocate bitrates over GOPs to the I, P, B and Bhier frames using appropriate algorithms for all the representations generated above. In a first example, four "normal" 60 fps representations are derived from the four 240 fps representations by selecting the I and P frames. Since the encoding algorithm allocated the bits to the I and P frames, the encoder user may not have direct control of the bitrates of the 4 "normal" representations. In some embodiments, the aggregate bitrate may be specified to be allocated to I and P frames (at, for example, 8 Mbps for the first ABR representation, 4 Mbps for second ABR representation, 2 Mbps for the third ABR representation and 1 Mbps for the fourth ABR representation). Similarly, for each ABR representation, the B frame bit rate allocation may be specified. In such embodiments, the server may deliver specific bit rates for streams with subsampled frame rates corresponding to "normal" playback or to any of the other slow motion (or trick mode) playback rates.

[0100] In a second example, if setting the ABR representation bitrates for the normal playback streams (e.g., 60 fps streams as defined and derived above) is desired, then such a constraint can be imposed on an appropriate subset of frames while encoding the full frame-rate stream. In some embodiments, this concept extends to an I-frame only encoding as well. Further, these techniques work independently of the choice of bitrate control algorithms as long as the bitrate constraints are met.

[0101] In a further embodiment, the various streams may be created using standardized scalable video techniques outlined in H.264/H.265, as shown in FIG. 8. FIG. 8 depicts source video captured at high frame-rate 802, scalable video coding (SVC) video techniques 804, ABR representations 806, an IP network 808, and communication paths 810A-C to the respective clients 812A-C. Such an embodiment is similar to the previous embodiment utilizing hierarchical GOP of FIGs. 7A-7B, except that the standardized principles of temporal scalability are adopted, as described in Heiko Schwarz, Detlev Marpe, and Thomas Wiegand's "Overview of the Scalable Video Coding Extension of the H.264/AVC Standard," (IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 17, NO. 9, SEPTEMBER 2007 pp. 1103- 1119). It may be noted that several constraints may be imposed in the stream creation process as a result of the SVC standard extensions in H.264 (and HEVC).

[0102] FIG. 9 depicts a flowchart of a modified delivery mechanism to account for the slow motion streams that may be additionally requested by the client, in accordance with some embodiments. FIG. 9 depicts the flowchart 900 that includes communications between a content source 902, an encoder 904, a transport packager 908, an origin server 910, a web server 914, and a client 916. In some embodiments, subsampled streams are ABR encoded (shown as dashed lines), packaged, and made available at the streaming server to send to the player.

[0103] In some embodiments, the ABR Encoding mechanism and rate allocation thereof can be done differently for the independent-encoding methods versus allocating bitrates to each frame type (I, P, B, Bhier) and constraining the rate control module to adhere to these constraints. In a scalable coding system, the rate controller for the single highest frame encoding will have these additional constraints imposed.

[0104] In an exemplary method, the client 916 may initially receive an MPD which describes the available streams at different bit rates and frame rates, and which may additionally include labeling to indicate which streams (or which frame rates) may be suitable to achieve different trick mode playback rates (e.g. for a given normal playback framerate which may be used by the client).

The client 916 may first request a "normal" stream (e.g., a stream with a sampled frame rate corresponding to the client's preferred normal playback rate, for example a stream with sampled frame rate of 60 fps). Along with the "normal" stream, the client 916 receives a manifest that includes information identifying available enhanced streams with different (e.g. higher) captured frame rates than the "normal" stream requested by the client. In response to the information on available streams, the client/player 916 displays to the user through a user interface information identifying the playback speeds (e.g. slow motion modes) that are enabled by the enhanced streams

(e.g. the available streams at different subsampled bit rates). A user may request one of these playback speeds, and the client 916 may identify and may request an enhanced stream that corresponds to the requested playback speed. In the meantime, the player goes back, for example to a point in time prior to the start of a significant event (either immediately before or some predetermined amount of time or frames corresponding to a significant event), and slows down the current normal playback of content in the local cache (e.g., playing the 60 fps "normal" stream content previously received and cached in the client at a 30 fps reduced playback rate), producing an immediate slow motion effect but at a reduced frame rate. Reducing playback rate provides a low-resolution slow-motion effect (in comparison to frames of the true slow motion effect of the enhanced stream that is being downloaded at the client's request). However, once a sufficient number of packets have been received for the enhanced stream with the higher frame rate, the client/player can switch to decoding and displaying the frames of the enhanced stream at the normal playback rate (e.g. 60 fps). In some embodiments, the player determines when there is a sufficient number of packets corresponding to buffer models to effect the switching to a higher resolution slow-motion picture. Thus, the slow motion effect is initiated with very low latency in response to user selection, even before the player receives the enhanced stream. Once the player receives enough of the enhanced stream to begin rendering and displaying the frames thereof, the slow motion playback quality is improved.

[0105] Note that different techniques (such as independent encoding, GOP, and SVC) described herein for slow motion stream creation will result in different ABR stream packages. In embodiments utilizing independent encoding approaches, the file packages are created by independently encoding the higher frame rate streams. In such embodiments, when a specific playback speed factor is requested by the client player 916, the edge streaming server 912 will send the files corresponding to the requested playback speed factor. In embodiments utilizing hierarchical GOP, the file packages may be created by selecting coded streams corresponding to specific types of frames. In such embodiments, a first set of ABR encoded streams with a first sampled frame rate may contain only I-frames, while a second set of ABR encoded streams with a second sampled frame rate may contain only I, P, and B frames (as an example). Such embodiments should not be considered limiting as other frame combinations are possible. Furthermore, in GOP embodiments, it is possible (and may be advantageous) to package on-the- fly (especially for VOD applications) and to select only sub-streams corresponding to the specific frames that must be sent down. In SVC embodiments, the same aspects as GOP are possible. User Interface.

[0106] A client player may use various different techniques to implement slow-motion features. In some embodiments, a manifest file or other information source alerts the client player to the availability of enhanced streams for slow-motion playback. For example, the manifest file may indicate that, in addition to one or more streams sampled at a normal playback rate for normal viewing (e.g. at 60 fps), streams sampled at other sampled frame rates (e.g. 120 fps, 240 fps, etc.) may be available for slow motion viewing. In response to an indication that enhanced slow motion streams are available, the encoder indicates to the user that the additional rates are available for slow motion feature implementation.

[0107] In some embodiments, slow-motion playback may be initiated automatically (e.g., event driven) or by an end-user controlled "instant replay." A user interface or a TV remote may, for example, have a button or other control linked to an 'instant slow motion replay' feature. In some embodiments, a user may select a slow motion rate among the available choices of slow motion rates that were generated and signaled by to the player. In some embodiments, automatic slow-motion replay may be triggered by detection of a significant event. In some embodiments, the significant events are user defined. There are several cues to determine a "significant event" and how far a client/player should go back to replay the last "significant event". In some embodiments, example cues include: a. Audio level at the venue: Loud audio levels may correspond to the level of importance of a given event (e.g., when the audio level at a sports venue goes beyond a threshold level after a scoring play, slow motion playback may be automatically triggered to replay the scoring play based on the point in time at which the audio level increased or exceeded the threshold).

b. Sentiment tracking of the end user: There are several applications on smartphones that can track user reaction to events. When a detected sentiment level reaches a threshold, an auto replay feature may be triggered.

c. Sentiment tracking at the venue: Similar to sentiment tracking of the end user, based on the sentiment of the crowd at the venue, auto replay actions may be trigged by the client/player such as an action replay. In some embodiments, sentiment tracking at the venue may be enabled by the actions of the crowd on social media, spectral analysis of cheering (boos vs. cheers), analysis of facial expressions of the crowd, as well as various other methods.

d. Based on biometric measurements of a viewer: Several applications may track user biometric data on a continuous basis. In some embodiments, a user's pulse rate may be measured by various means known to those of skill in the art. In some embodiments, if the measured pulse rate of the user exceeds a threshold, the auto slow motion playback could be triggered.

e. Social Media Tracking: Based on feedback from user subscribed blogs/forums/user groups, there could be a replay of an event that is flagged as being significant (based on social media groups and/or keywords in social media posts).

f. Crowd-sourced incentives for player action: In some embodiments, crowd-sourced feedback (e.g., a request from a number of end users) may induce an event to be replayed or played back.

When a user chooses a playback speed factor corresponding to slow motion playback or when the automatic mechanism triggers a slow motion replay, the player may be configured to go back a predetermined amount of time or a predetermined number of frames to start the trick play stream.

[0108] In embodiments where the player is configured to play a trick play stream for a fixed amount of time, the player may recalculate (for each playback speed) how far back to go in real time to play the trick play stream for the fixed amount of time (alternatively, how far back in segments to go to get the trick play stream). For example, when using a stream sampled at 240 fps,

(for playback at a playback speed factor of ¼, for example), 40 seconds of slow-motion replay corresponds to 10 seconds of real time playback, thus the client/player may go back only 10 seconds. Similarly, for a stream sampled at 120 fps (for playback with a playback speed factor of ½), 40 seconds of half-speed slow-motion replay corresponds to 20 seconds of full-speed playback, thus the player may go back only 20 seconds.

[0109] In some embodiments, non-integer submultiple frame rates may be derived from the full frame-rate using interpolation techniques. Thus, some embodiments may provide arbitrarily slower frame-rate delivery (up to a threshold which is limited by the ratio of the full capture frame- rate versus the "normal" frame rate).

[0110] There are various ways for the player to decide the playback frame rate. In some embodiments as mentioned above, the encoder/server recommends playback frame rates for different streams via an out-of-band manifest file (for ABR), since the encoder is aware of the relative frame rates. In such embodiments, the player may follow this recommendation and play back the stream at the recommended playback frame rate. Alternatively, this information may be sent in-band in the private data segment of the transport stream.

[0111] In some embodiments, the player derives the playback frame rates based on the representation specification. The player may calculate the relative rates of the enhanced versus normal streams and adjust playback accordingly.

[0112] During normal playback, the user may want to see action replay in slow motion. In some embodiments, the user clicks an appropriate button, and the player rewinds a programmable amount of time or frames, and slow-motion playback begins from there. To catch back up to the real-time video, the user may simply click another button and the player may responsively switch to playing from the latest downloaded and decodable segment in real time. The user may also have an option to revert to the stream where the slow motion replay was initiated (in which case they are always behind real-time). In such embodiments, the user also retains the option of catchup to real time at any desired time.

[0113] In some embodiments, the player may also have the ability to initiate slow motion playback without jumping backward in the stream. For example, a user who anticipates that an exciting event is about to occur may select slow motion play. In response, the player may (i) slow down the playback frame rate of the currently-played stream and (ii) request a stream with a higher sampled frame rate, with the player switching to the stream with the higher sampled frame rate once enough of that stream has been received. For example, if the user is viewing content with a captured frame rate of 60 fps and a normal playback frame rate of 60 fps, and the user selects (non- replay) slow motion, the player may begin playing the original stream at 30 fps, giving a playback speed factor of ½, and the player may request a corresponding stream captured at 120 fps. Once enough of the 120 fps stream has been received, the player begins playing that stream at a playback frame rate of 60 fps, still giving a playback speed factor of ½ but with an improved quality.

[0114] One issue to consider is the switch over latency. Switch over latency may occur since the streams are segmented (each segment usually lasts a few seconds), and then sent over to the receiver. If a normal switch over is requested, then several such segments would have to be requested from the head end (to fill the decode/play out buffers) before the playback can begin. This may cause undue latency. In some embodiments, to mitigate this latency, the player may slow down the playback of the current normal stream to ½, ¼ or any other frame rates as requested. Then when the buffer fills up (e.g. the buffer receives sufficient content from the enhanced stream suitable for high quality slow motion playback), then the enhanced stream content is decoded and played back at the appropriate frame rate, in place of the reduced rate playback of normal content. In some embodiments, a sequence of events as seen by this approach may include the following:

1. The user selects an icon for playback at a certain rate (e.g. half the speed of the incoming streams). In some embodiments, the selection may be automatic. In some embodiments, the selection is made at a first playback position.

2. The client player requests a corresponding enhanced stream from the server.

3. In the meantime, the client player determines that a slow motion effect is being requested. In some embodiments, the client player immediately plays back the current stream in its buffer and slows down the playback to the slow motion rate (e.g. at half speed). This may give the user the effect of immediate slow motion playback.

4. When enough frames of the enhanced stream have been delivered, then the client/player may switch playback to the enhanced stream. The switch may be a synchronized switch, such that the playback from the original stream ends at the same playback position in the video content where the playback from the enhanced stream begins, such that playback does not appear to jump forward or backward in time when the playback is switched from the original stream to the enhanced stream. In some embodiments, each representation may use a common time base in which the frames are consistently marked with time stamps, and the segments can be of equal duration and time aligned. In such embodiments, if the player is currently playing the normal stream segment, the player can switch to the enhanced stream at the beginning of the next segment. In alternative embodiments, the player may switch during the segment period. In such embodiments, the player may use the time stamps to determine when to switch from one representation to another. In some embodiments, players may be downloading and buffering several segments (e.g. many seconds or tens of seconds) of content ahead of the current playback point. For example, the player may be in a situation where it has many seconds of buffer from the normal stream, and if slow motion is selected without enough time to switch at the next segment, the player may wait for the remaining time left in a whole segment period (worst case), or switch in the middle of the segment using the time stamps.

From a quality perspective, the sequence sampled at 120 fps being used for the half-speed slow motion effect would be expected to have a higher quality than a normal signal (e.g. sampled at

60 fps) stretched to play out at half the frame rate.

In some embodiments, the playback rate for the enhanced stream is the same playback rate for the current stream prior to slowing the playback rate down during buffering (See FIGs. 6A, 6B). In some embodiments, the playback rate for the enhanced stream may be different according the representations received in the MPD (see FIG. 6C).

[0115] In some embodiments, the client may determine when to switch from a current stream played back at a reduced rate to an enhanced stream played back at normal playback rate based on making a determination that a sufficient number of frames of the enhanced stream have been received (or are in the buffer). In some embodiments, the client may determine when a sufficient number of packets of the enhanced stream have been received (or are in the buffer). In some embodiments, the client may determine when a sufficient number of segments of the enhanced stream have been received (or are in the buffer). In some embodiments, the client may determine when a sufficient amount of playback time for the enhanced stream has been received (or is in the buffer).

[0116] FIG. 10 illustrates an example user interface (UI) 1000 when using the multi -frame rate slow motion feature. In this example two replay slow motion rates are available. A simple setup at the player could be "go back to the start of the last significant event", "back N frames" or "back M seconds". The UI depicted in FIG. 10 includes multiple user-clickable icons displayed on the right based on indications from the headend and/or based on available rates derived from the manifest, there are two additional speeds available to go back and play back in slow motion - "BSloMo 2x" and "BSloMo 4x," which are two and four times slower respectively than real time playback. With a single button touch, a user could go back N frames, M seconds, or to the start of the last significant event or sequence. The playback would then commence at half the normal rate for the BSloMo 2x setting and four times slower than the normal rate for the BSloMo 4x setting.

[0117] Similarly, the user could use a feature that allows them to play forward in slow motion using, for example, the "FSloMo 2x" or the "FSloMo 4x" icons. The FSloMo 2x plays forwards at two times slower than real time and the FSloMo 4x plays forwards at four times slower than real time. Similar to the BSloMo feature, the FSloMo features could have its own set of forwards slow motion rates. As shown, the UI in FIG. 10 includes 4 different SloMo settings, however it should be apparent to one of skill in the art that any number of SloMo settings is possible.

[0118] Various techniques may be used to determine how far to go back to start the slow- motion replay. This determination can be made on the basis of a number of possible criteria that was discussed earlier. In any of these cases, the client may request and the server may deliver the higher frame rate streams for rendering in slow motion.

[0119] In some embodiments, the player may be configured to operate in a forward slow motion scenario. When the user selects the forward slow motion button, the player can immediately slow down the action with frames in its existing cache (i.e., frames that have already been downloaded from the server) by the requested slow motion factor. In some embodiments, the player requests from the server files corresponding to the higher frame rate sequences. When a sufficient number of such frames are downloaded, the play forward function will switch from just slowing down the regular frame rate sequence to playing back the higher frame rate sequence at the appropriate slowed-down rate and still providing the full resolution and a superior slow motion experience.