CROSS-REFERENCED TO RELATED APPLICATIONS

The present application claims priority to United States Provisional Patent Application Nos. 61/687,049, filed on April 17, 2012 entitled SERVER SIDE CROSSFADE FOR PROGRESSIVE DOWNLOAD MEDIA, and 61/687,048, filed on April 17, 2012 entitled SYSTEMS AND METHODS FOR IMPLEMENTING EFFICIENT CROSS-FADING BETWEEN COMPRESSED AUDIO STREAMS the entire disclosure of each which is hereby fully incorporated herein by reference.

TECHNICAL FIELD:

The present invention relates to digital media delivery and playback, and in particular to systems and methods for implementing cross-fading, interstitials and other

effects/processing of two or more media elements on a downstream device so as to replicate, to the extent possible, the feel, sound and flow of broadcast programming or "DJ" (disc jockey) enhanced playlists.

BACKGROUND OF THE INVENTION:

Media delivery of both audio and video programming relies heavily on the ability to provide crossfades, blends and other processing or interstitial effects, to both enhance the user experience and to provide high quality delivery to the customer.

In a traditional broadcast model, such as, for example, FM radio, or a national television network, these kinds of effects are typically generated at the broadcast station or headend, typically by an automation system. This approach to effect insertion works well because all users receive the same broadcast stream, and complex effects can be processed, implemented and tightly controlled at a single location. Moreover, the amount of processing required is both constant for a given effect and independent of the number of end users.

With modern multimedia delivery, especially systems that deliver a personalized experience, where each user receives different programming or content that (i) varies with time relative to all other users, and that (ii) is personalized, to a greater or lesser degree, for each user, this traditional model does not work. In a personalized service, all users receive a custom set of audio or video elements usually based on a customized playlist.

If blends, interstitial effects, processing or cross fading are required or desired between successive media elements in such a personalized stream, one approach is to push the burden of processing the effect or crossfade to the client device, including the

responsibility to both manage the download of the various elements needed, with sufficient lead time. This approach is often sub optimal. Not all client devices may be capable of processing complex effects to deliver a high-quality user experience, for a variety of reasons. For example, some devices may not be capable of such processing because of processing power, memory size, power or other limitations of processing effects. Other devices may have been the zenith of their class at one time, but are now legacy devices, and as techniques for on device cross fading advance, their capabilities are no longer sufficient. This can easily lead to a spectrum of unsatisfactory user experiences.

What are thus needed in the art are systems and methods to address these problems of the prior art so as to implement and facilitate cross-fading, interstitials and other effects/processing of two or more media elements from the server side, and have the ultimate result easily effected and implemented on the downstream device.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 illustrates Media Clip Chunking in a progressive download media delivery system; Fig. 2 depicts an exemplary Sub-optimal Server Side Crossfade technique according to exemplary embodiments of the present invention;

Fig. 3 depicts an exemplary Optimal Server Side Crossfade Technique according to exemplary embodiments of the present invention; and

Fig. 4 illustrates Core and Processed Chunks according to exemplary embodiments of the present invention.

SUMMARY OF THE INVENTION

In exemplary embodiments of the present invention systems and methods are provided to implement and facilitate cross-fading, interstitials and other effects/processing of two or more media elements in a personalized media delivery service so that each client or user has a consistent high quality experience. The various effects or crossfade processing can occur on the broadcast, music service, publisher or other server-side, but can still be personalized to a specific user, thus still allowing for a personalized experience for each individual user, in a manner where the processing burden is minimized on the downstream side or client device. This approach enables a consistent user experience, independent of client device capabilities, both static and dynamic. The cross-fade, for example, can be implemented after decoding the relevant chunks of each component clip, processing, recoding and rechunking, or, in a preferred

embodiment, the cross-fade or other effect can be implemented on the relevant chunks to the effect in the compressed domain, thus obviating any loss of quality by re- encoding. A large scale personalized content delivery service can be implemented by limiting the processing to essentially the first and last chunks of any file, since there is no need to process the full clip. In exemplary embodiments of the present invention this type of processing can easily be accommodated in cloud computing technology, where the first and last files may be conveniently extracted and processed within the cloud to meet the required load. Processing may also be done locally, for example, by the broadcaster, with sufficient processing power to manage peak load. DETAILED DESCRIPTION OF THE INVENTION:

In exemplary embodiments of the present invention systems and methods are provided to implement and facilitate cross-fading, interstitials and other effects/processing of two or more media elements in a personalized media delivery service so that each client or user has a consistent high quality experience.

Thus, in exemplary embodiments of the present invention, the effects or crossfade processing can occur on the broadcast, publisher, music service, or other server-side, but may still be personalized to a specific user, thus allowing a personalized experience for each individual user in a manner where the processing burden is minimized on the downstream side or client device. This approach enables a consistent user experience, independent of client device capabilities, both static and dynamic.

A contemporary method of media delivery is called "Progressive Download". In this method, compressed audio or video bitstreams are encoded at the broadcast side and divided into chunks, for example, of mostly equal lengths. This chunking produces many small files which can easily be distributed over a low-cost infrastructure such as, for example, a web server. In addition, streams may be encoded using multiple bit rates, and a client device can switch between the streams, at file boundaries, to optimize playback quality at any given delivery rate. Examples of this kind of technique include, for example, Apple's HLS or Microsoft's Smooth Streaming. Players for this type of decoding and playback are able to seamlessly splice the discrete file chunks, and provide a smooth, uninterrupted playback of audio and video. The chunking feature of Progressive Download can be leveraged to implement cross fading, blends and other interstitial processing/effects on the server side, as next described.

Exemplary Server Side Cross Fade: Sub-optimal Method

Fig. 1 depicts an exemplary media file 1 10 of one minute and 10 seconds duration (1 :10). In this exemplary case the chunk lengths used are equal to 20 seconds. Here a broadcaster may encode the file using a media codec suitable for the purpose, and then divide or chunk the resultant output into, for example, four files: three files, each containing 20 seconds of content, and a fourth file containing only 10 seconds of content - the latter's length being different inasmuch as this last file contains the end of the audio or video. Usually four files such as these 120 would be made available for distribution referenced by an Index or Manifest File 130. Such a Manifest File 130 may generally contains a pointer or URL to the four files in sequence, as well as a duration value for each file.

In the following illustrative description, we assume a crossfade or other effect between the end of a first clip, Clip1 205, and the beginning of a subsequent clip, Clip2 207. Assuming the duration of the desired crossfade or other effect is shorter than the duration of the last chunk of Clip1 , which is, as shown in Fig. 2, Compressed

Clip1 :Chunk 4, then Clip1 :Chunk 4 and Clip2:Chunk 1 would be decoded to baseband , time aligned and processed with the desired effect. Such processing can be done in a crossfading module 210, which has stored, or can be passed, various parameters for implementing the crossfade, including, for example, (i) audio trajectory, (ii) fade or blend type, (iii) number of elements in the effect (e.g., 2, 3 or more, for complex voice-overs and other transitions), (iv) volume/attenuation levels for each component or element during the cross fade, and (v) intro and outro points, etc.. Such parameters can, for example, be stored in crossfading module 210, as noted, or can be modified/updated by inferences from user behavior on the client device and sent via a message protocol to the crossfade module 210 on the server side. In this context reference is made to PCT Patent Application No. PCT/US2012/65943, filed on November 19, 2012, entitled SYSTEMS AND METHODS FOR IMPLEMENTING CROSS-FADING, INTERSITITALS AND OTHER EFFECTS DOWNSTREAM (the "Crossfade Application"), under common assignment herewith, which describes various parameters and types of crossfades and other effects. This commonly owned application is hereby fully incorporated herein by this reference. Effectively, any crossfade or other effect that can be performed, as described therein, can be implemented using techniques according to the present invention, on the server side.

Returning to Fig. 2, the resultant effect or crossfade would then be re-encoded and chunked. This results in a new chunk, namely a "revised" Clip2:Chunk1 231 being generated and a new custom index file 220 would also be generated to reflect the new output clip(s) 230 and modified clip duration. As can be seen, the last chunk of Clip1 has been combined with the first chunk of Clip2 to make a "revised" Clip2:Chunk1 , which contains the cross faded portions of the two clips, for example.

It is noted that in the example of Fig. 2 the cross fade or effect duration is less than 0:10, which was the original duration of Clip1 :Chunk4 as shown in Fig. 1 . In general if the desired effect(s) or crossfade duration is less than that of the last chunk in the file, then only the last chunk of the file needs to be retrieved and processed. On the other hand, if the required Crossfade duration is longer than the last chunk, then both Chunk N and Chunk N-1 must be retrieved for processing.

Additionally, unless the last chunk of Clip1 - here Clip1 :Chunk4 - or the first chunk of Clip2 - here Clip2:Chunk1 , is equal to the crossfade length, the resultant chunk made from the last chunk of the prior clip, the first chunk of the subsequent clip and the overlap between them will often be longer than the chunk length limit. In such cases there may, for example, be two "revised" chunks taking the place of, for example, original Clip1 :Chunk4 and Clip2:Chunk1 . For example, a 3 second duration last chunk with a 2 second crossfade could either (a) produce a new 1 second chunk followed by a 20 second chunk which includes the crossfade, or (b) produce a 21 second chunk including the crossfade. In exemplary embodiments of the present invention, either option is valid.

The particular embodiment shown in Fig. 2, where prior to crossfade processing the relevant chunks are decoded, is considered sub-optimal as the audio or video is subject to quality loss upon re-encoding, if the codec is a lossy type, because of the

requirement to decode and re-encode the content on the server side. If the codec in question is in fact lossless, then the technique of Fig. 2 would be considered optimal. However it is highly unusual to utilize lossless compression for consumer delivery, as significantly more bandwidth is typically required to deliver a given signal. Thus, in the real world, where nearly every codec is lossy, although this method achieves the cross fade result, there is a better way, as next described, which we term the "Optimal Method". Exemplary Server Side Cross Fade: Optimal Method

Fig. 3 depicts an optimal embodiment of server side crossfading with lossy

compression. In this method, the same criteria are used to select Clip1 :Chunk 4 and Clip2:Chunk 1 for the cross fade or other effect, as described above. The same parameters needed for crossfade module 310 may, for example, be passed to it via messaging, or pre-stored in it, as described above, and in the Crossfade Application. However, in this exemplary Optimal Method, the effect or crossfade occurs in the compressed domain - without any requirement to first decode to baseband for processing. This method allows preservation of the full quality of the signal, such that the output of this processing should be equal in quality to that of the input clips.

Crossfade module 310 may thus combine compressed data from each relevant chunk to generate the new "revised" chunk(s), also containing compressed data. The precise mechanism of the compressed crossfade is described in detail in U.S. Provisional Patent Application No. 61/687,048 filed on April 17, 2012, entitled SYSTEMS AND METHODS FOR IMPLEMENTING EFFICIENT CROSS-FADING BETWEEN

COMPRESSED AUDIO STREAMS (the "Compressed Crossfade Provisional"), under common assignment herewith. This application describes various such mechanisms for compressed crossfading and similar effects. This application is hereby incorporated herein by this reference, as noted above.

Core Chunks and Processed Chunks - Optimizing Personalized Service Delivery

Fig. 4 illustrates a number of compressed chunks for a given exemplary file. As can be readily appreciated from the discussion of the exemplary server side cross-fade methods shown in Figs. 2 and 3, in exemplary embodiments of the present invention one can take advantage of the fact that typically only the first and last compressed chunk, of an audio (or other content type) clip or stream needs to be processed. This significantly reduces the amount of processing that is required to support, for example, custom playlists and streaming.

Thus, Fig. 4 illustrates division of an audio clip into an initial compressed chunk 410 that is relevant to a cross fade or interstitial effect, multiple "core files" 420 that will not change and are not needed to implement the crossfade or effect, and a final relevant compressed chunk needed for any crossfade or blend with a subsequent clip. Fig. 4 clearly indicates that most files in the clip are generally considered core files 420 (colored in blue) and therefore may be made available to all users without any modification. Each user will have at least one custom chunk 410 at the beginning of the file and one custom chunk 430 at the end of the file, where the cross fading or other effect will occur. Each user will also have, for example, a custom index file which accounts for the naming and duration of these custom files.

As noted, if the desired effect(s) or crossfade duration is less than that of the last chunk in the file, then only the last chunk needs to be retrieved and processed. As also noted, if the required crossfade duration is longer than the last chunk, then chunk N and chunk N-1 must be retrieved for processing. Similar considerations apply to the first chunk in the subsequent file.

As noted, if the crossfade duration is longer than the 20 seconds as considered in the examples provided above, then both Chunk 1 and Chunk 2 of Clip2 must be retrieved for processing.

As can thus readily be seen, by limiting the processing to essentially the first and last chunks of the file, the disclosed technique can be successfully scaled to many hundreds of thousands, or even millions, of simultaneous users, since it does not require processing the full clip - as to any of the many clips in a given library. In exemplary embodiments of the present invention this type of processing can easily be

accommodated in cloud computing technology, where the first and last files may be conveniently extracted and processed within the cloud to meet the required load.

Alternatively, processing may also be done locally, for example, by the broadcaster, or music service provider, with sufficient processing power to manage peak load.

Encryption of Content Clips

For security reasons, in exemplary embodiments of the present invention the

compressed audio chunks may be encrypted. If this is the case, decryption will be required for the first and last chunk (or, for example, additional chunks if the effect's duration exceeds a chunk length, as noted above) prior to processing. An alternate approach would be to have the first and last chunks (or additional chunks, as noted) unencrypted, which would alleviate the requirement for decryption, but not sacrifice security in any material way.

Exemplary Cross-Fade

To illustrate a simple cross fade, involving two sequential songs, the following exampl is helpful.

An exemplary crossfade on an Android device was accomplished using two

MediaPlayers and setVolume(). Crossfade works by lowering the volume on one MediaPlayer and raising the volume on the other.

The exemplary version used 5 threads, but because 5 threads may use significant power, this may be reduced to a 1 or 3 threaded version.

Threads

1 . Timing thread - this thread wakes up every 10 milliseconds and checks the current playback positions against crossfade parameters to initiate a crossfade

2. Track 1 fade down thread;

3. Track 1 fade up thread;

4. Track 2 fade up thread; and

5. Track 2 fade down thread.

All the track fade threads work pretty much the same way. A fade up thread for a 5 second fade up would thus work like this:

1 . The timing thread sees a crossfade is to start, it starts a fade up thread 2. In Android based devices, for example, the setVolume() function has 100 steps of volume level. The thread thus does 100 iterations to fade up the volume using sleep to pause to match the fade duration. The sleep duration is set using:

Crossfade in milliseconds / Number of volume steps = sleep duration

So for a 5 second fade up, for example, on a device with 100 volume level steps, as here,

(5 ^* 1000) / 100 = sleep duration for (float i = 0; i < 10; i=(float) (i+ .1 ))

{

Sleep(50);

setVolume(i);

}

Similarly, for example, for a fade down the algorithm can, for example, be similar but the iterations go down: for (float i = 10; i > 0; i= (float) (i - .1 ))

{

Sleep(50);

setVolume(i);

}

An exemplary two song cross fade using exemplary song durations and cross fade lengths can thus be as follows:

Song #1

Length - 4:30

Crossfade begins 4:23

Fade up Position ^{■ ■} 1 -1 No fade up Fade up Duration - -1 -1 No fade up Duration Fade Down Position - 4:25

Fade Down Duration - 0.04

Song #2

Length - 3:00

Crossfade - 2:52

Fade - 0.03

Fade up Position - 0.02

Fade up Duration - 0.04

Fade Down Position - 4:25

Fade Down Duration - 0.04

1 . Song #1 starts playing with no fade up, Timer thread starts;

2. Timer thread waits for song #1 "Crossfade - 4:23";

3. At 4:23 timer thread starts fade down thread with these parameters:

crossfade in milliseconds / Number of volume steps = sleep duration Fade Down Duration of 0.04 / 100 = sleep duration of 2.5 milliseconds 40 milliseconds /100 = .4 milliseconds sleep duration;

4. Also at 4:23 Song #2 starts playing Timer thread sees Fade up at 0:02 and starts fade up thread;

5. crossfade in milliseconds / Number of volume steps = sleep duration Fade up Duration of 0.06 / 100 = sleep duration of 1 .7 milliseconds

60 milliseconds /100 = .6 milliseconds sleep duration Exemplary Implementations

Any suitable programming language can be used to implement the routines of particular exemplary embodiments including, but not limited to, the following: C, C++, Java, JavaScript, Python, Ruby, CoffeeScript, assembly language, etc. Different

programming techniques can be employed such as procedural or object oriented. The routines can execute on a single processing device or multiple processors. Although the steps, operations, or computations may be presented in a specific order, this order may be changed in different particular embodiments. In some particular embodiments, multiple steps shown as sequential in this specification can be performed at the same time

Particular embodiments may be implemented in a computer-readable storage device or non-transitory computer readable medium for use by or in connection with the

instruction execution system, apparatus, system, or device. Particular embodiments can be implemented in the form of control logic in software or hardware or a combination of both. The control logic, when executed by one or more processors, may be operable to perform that which is described in particular embodiments.

Particular embodiments may be implemented by using a programmed general purpose digital computer, by using application specific integrated circuits, programmable logic devices, field programmable gate arrays, optical, chemical, biological, quantum or nanoengineered systems, components and mechanisms may be used. In general, the functions of particular embodiments can be achieved by any means as is known in the art. Distributed, networked systems, components, and/or circuits can be used.

Communication, or transfer, of data may be wired, wireless, or by any other means.

It will also be appreciated that one or more of the elements depicted in the

drawings/figures can also be implemented in a more separated or integrated manner, or even removed or rendered as inoperable in certain cases, as is useful in accordance with a particular application. It is also within the spirit and scope to implement a program or code that can be stored in a machine-readable medium, such as a storage device, to permit a computer to perform any of the methods described above. As used in the description herein and throughout the claims that follow, "a", "an", and "the" includes plural references unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of "in" includes "in" and "on" unless the context clearly dictates otherwise.

While there have been described methods for implementing efficient cross-fading between compressed audio streams, it is to be understood that many changes may be made therein without departing from the spirit and scope of the invention. Insubstantial changes from the claimed subject matter as viewed by a person with ordinary skill in the art, no known or later devised, are expressly contemplated as being equivalently within the scope of the claims. Therefore, obvious substitutions now or later known to one with ordinary skill in the art are defined to be within the scope of the defined elements. The described embodiments of the invention are presented for the purpose of illustration and not of limitation

The above-presented description and accompanying figures are intended by way of example only and are not intended to limit the present invention in any way except as set forth in the following claims. It is particularly noted that persons skilled in the art can readily combine the various technical aspects of the various exemplary embodiments described into a wide variety of techniques, systems and methods, all being

encompassed within the present invention. For the sake of clarity, it is noted that the term "crossfade" includes any transition, blend or interstitial effect implemented on or near a boundary between two successive content clips or files provided in a content delivery service or method.