SYSTEM AND METHOD OF DETERMINING PERSONAL MUSIC PREFERENCES

This invention pertains to a music playback system and method of playing back music, and more particularly, to a method of determining personal musical preferences of a user, and an audio player that determines personal musical preferences of a user.

With the proliferation of digital music file formats, such as mp3, wav, wma, etc., and the success of digital audio players such as MICROSOFT® WINDOWS® Audio player and APPLE® IPOD®, consumers are gaining increasing flexibility and options for storing, organizing, and playing digital music files. Capacity is ever increasing for consumers to store and organize more and more digital music files. It is not uncommon for a digital audio player to have access to thousands of digital music files. Digital audio players are able to organize literally thousands of songs and play them back to a user in a wide variety of different playback modes. In addition to personal-computer-based digital audio players, there are an increasing number of portable digital audio players available, such as the APPLE® IPOD®. Digital audio players are now being rapidly integrated into other portable devices, including personal digital assistants, portable e-mail devices, and mobile telephones. Such devices are rapidly becoming ubiquitous. Meanwhile, as these devices proliferate, there is a growing demand to add more and more features.

Accordingly, it would be desirable to provide a feature whereby a user's musical preferences can be characterized based on the music to which he/she listens. It would also be desirable to provide an audio player that includes such a feature. It would be further desirable to provide a method which recommends songs to a user based on their characterized musical preferences. The present invention is directed to addressing one or more of the preceding concerns.

In one aspect of the invention, a method of characterizing an individual's musical preferences comprises: receiving audio samples for each portion of music that is played to a user through an audio playback system; extracting from the audio samples a plurality of low level features of each portion of music that is played to the user through the audio playback system; computing, from the extracted low level features, a value for each of a plurality of low level audio descriptors for each portion of music that has been played to

the user through the audio playback system; averaging each of the plurality of computed low level audio descriptor values for all portions of music that have been played to the user through the audio playback system over a time period; and storing in memory the averaged low level audio descriptor values for all portions of music that have been played to the user through the audio playback system over the time period.

In another aspect of the invention, a system for playing music comprises: memory; an audio playback system; a user interface adapted to receive a request from a user to play music through the audio playback system; and a processor configured to execute a method comprising the steps: receiving audio samples for all music that is played to a user through the audio playback system; extracting from the audio samples a plurality of low level features for all music that is played to the user through the audio playback system; computing, from the extracted low level features, a value for each of a plurality of low level audio descriptors for all music that is played to the user through the audio playback system over a time period; and storing in the memory the computed low level audio descriptor values for all music that has been played to the user through the audio playback system over the time period.

In yet another aspect of the invention, a method of characterizing an individual's musical preferences comprises: receiving audio samples for all music that is played to a user through an audio playback system; extracting from the audio samples a plurality of low level features for all music that is played to the user through the audio playback system; computing, from the extracted low level features, a value for each of a plurality of low level audio descriptors for all music that is played to the user through the audio playback system over a time period; and storing in memory the computed low level audio descriptor values for all music that has been played to the user through the audio playback system over the time period.

Further and other aspects will become evident from the description to follow.

FIG. 1 is a functional block diagram of an audio player; FIG. 2 is a flowchart illustrating on embodiment of a method of characterizing musical preferences of a user of an audio playback system.

The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which preferred embodiments of the invention are shown. This invention may, however, be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided as teaching examples of the invention.

FIG. 1 is a functional block diagram of an audio player 100. As will be appreciated by those skilled in the art, the various "parts" shown in FIG. 1 may be physically implemented using a software-controlled microprocessor, hard- wired logic circuits, or a combination thereof. Also, while the parts are functionally segregated in FIG. 1 for explanation purposes, they may be combined in any physical implementation.

Audio player 100 includes a processor 120, memory 140, one or more input-output ports 160, audio subsystem 170, and user interface 190. In the embodiment of FIG.1, user interface 190 includes a graphical user interface 195. In that case, audio player 100 also includes video subsystem 180. It should be understood that when audio player 100 does not include graphical user interface 195, then video subsystem 180 may be omitted.

Meanwhile, in the case where audio player 100 includes video subsystem 180, in some embodiments audio player 100 may play back both audio files and audio-video files, such as music videos.

Audio player 100 may be integrated into a portable device that includes one or more other functions, such as a portable computing device, a personal digital assistant (PDA), a mobile telephone, a digital camera, a portable e-mail terminal, etc. In that case one or all of the processor 120, memory 140, and user interface 190 may be used to perform functions of audio player 100, and also to perform one or more additional functions such as a portable computing, personal information management, mobile telephony, a digital photography, e-mail transmission and reception, etc.

It should be understood that in an alternative embodiment to what is shown in FIG. 1 , the audio player may be provided as a software program executed by a processor of a general purpose personal computer (PC), such that there are no dedicated hardware components to the audio player. In that case, the functions of processor 120 described below may be performed by the microprocessor of the PC, the functions of memory 140 may be provided by the memory of the PC, the functions provided by audio subsystem 170 may be provided by the audio systems of the PC, and the functions of user interface 190 may be provided by a graphical user interface 195 of the PC, etc.

Audio player 100 plays music. Music may include digital music files, which may be stored in memory 140. Such music files may include digital audio files, and may also include the audio component of digital audio-video files, such as music videos. Music files played by audio player 100 may have any convenient file format, such as mp3, wma, wav, MPEG, etc. Furthermore, audio player 100 may play music that is not stored locally, and instead is streamed to audio player 100 from an external source (e.g., a file on a local area network; an Internet site; etc.) via input/output port 160. Optionally, audio player 100 may also play some music that originates in analog format. Such music may be received via input/output port 160 (e.g., an RCA jack), or another device with which audio player 100 is integrated, such as an FM radio.

Turning again now to FIG. 1, processor 120 is configured to execute one or more software algorithms in conjunction with memory 140 and user interface 190 to provide the functionality of audio player 100. Beneficially, processor 120 includes its own memory (e.g., nonvolatile memory) for storing executable software code that allows it to perform the various functions of audio player 100. Alternatively, the executable code may be stored in designated memory locations within memory 140.

In general, audio player 100 may include a variety of memory components to perform a variety of functions. For example, it may include: processor cache memory; non- volatile memory (e.g., ROM or Flash memory) for storing executable code of software algorithms executed by processor 120; volatile memory (e.g., SRAM/DRAM, etc.) necessary to execute the software algorithms; and additional non- volatile memory (e.g., Flash memory, hard disk drive, or optical or magnetic disc, etc.) for storing the digital music files and, beneficially, associated metadata. Memory 140 shown in FIG. 1 is intended to represent memory performing all of the functions described above. Memory 140 stores data in response to the processor 120. Such data may include user preferences for operation of audio player 120 by one or more users. Such data may also include data for organizing music files, such a playlists, directories, and other mechanisms for grouping, separating, and otherwise generally organizing music files stored in audio player 100. Beneficially, memory 140 also stores music files for playback by audio player 100. Alternatively, music files may be stored in a separate memory other than memory 140.

Input/output port(s) 160 facilitates the transfer of music into and out of audio player 100. Input/output port(s) 160 may include one or more USB ports, Firewire ports,

Bluetooth ports, wireless Ethernet ports, RCA jacks, etc.

Audio subsystem 170 includes audio circuitry 172, audio speaker 176, and audio output jack(s) 178 (e.g., a headphone jack). In some cases, some or all of audio circuitry 172 may be integrated with processor 120. Also, either audio speaker 176 or audio output jack(s) 178 may be omitted.

Video subsystem 180 includes audio circuitry 182 and video display device 184. In some cases, some or all of video circuitry 182 may be integrated with processor 120.

User interface 190 includes objects that allow a user to control the operation of audio player 100. In the embodiment of FIG. 1, user interface 190 includes a graphical user interface 195. However in some embodiments graphical user interface 195 may be omitted. The objects can include one or more manual buttons 192, knobs 194, click- wheels 196, etc. mounted on a case that houses audio player 100. When audio player includes graphical user interface 195, objects may also include buttons, check boxes, data entry (e.g., text) boxes, pull down menus, etc, that are displayed via display screen 184. Such objects may be selected via click-wheel 196, or another scrolling device, such as a mouse, a trackball, a track-pointer, or by arrow buttons, etc. Alternatively, display screen 184 may be a touchscreen displaying buttons, check boxes, data entry (e.g., text) boxes, pull down menus, etc which a user selects on display screen 184 with a pointer.

Hereafter, a feature of audio player 100 that characterizes one or more users' musical preferences will be described. In order to not obscure the explanation of this feature, the description of other audio playback operations and features of audio player 100 will be omitted.

Audio player 100 plays back music in response to one or more user instructions received user interface 190. Music files may be retrieved from memory (e.g., memory 140), or may be provided from an external source, such a streaming music file from an external source (e.g., a file accessible through a local area network; the Internet; an external audio system; etc.) via input/output port 160. Music files may be played back in a variety of different playback modes. Such modes include playing a preset playlist, playing an individual music file, random playback, etc. When a music file is being played back, during the audio processing of the file a series of audio samples are produced that are ultimately converted by audio circuitry 172 to an analog audio signal to be provided to audio speaker 176, or audio output jack 178.

As noted above, optionally, audio player 100 may also play some music that originates in analog format. However, in general processor 120 cannot process analog audio signals. In that case, preferably audio subsystem 170 includes an analog-to-digital converter (AJO) that digitizes the analog music to produce the series of audio samples that are to be used to characterize a user's musical preferences, as described in further detail below.

In one embodiment, processor 120 executes an algorithm to extract a plurality of low level features from the audio samples of the music that is played to the user through the audio player 100. Processor 120 then determines, from the extracted low level features, a value for each of a plurality of low level audio descriptors for the music that is played to the user through the audio player 100. As long as the values for the set of low level audio descriptors are determined based on a significant portion of the music that is actually played by the user, this set of descriptor values provides a useful characterization of the user's musical preferences. As used herein, a feature is a distinctive characteristic of data that has some meaning. A descriptor is a representation of the feature. For example, if "color" is a feature, then one descriptor for color might be a frequency spectrum plot. Another descriptor might be a color space diagram. A descriptor value is an instantiation of the descriptor for a given data set. Low-level audio features are those related to the signal itself and have little or no meaning to the end-user. In other words, and thinking in terms of the audio domain, these features cannot be heard. On the other hand, high-level audio features are meaningful and might be related to semantic or syntactic features of the sound. They can be used to classify sound objects into the class they belong. The MPEG-7 Audio Standard defines seventeen temporal and spectral low level audio descriptors that can be divided into following groups:

• Basic: (1) Instantaneous waveform; and (2) power values;

• Basic spectral: (3) Log-frequency power spectrum; (4) audio spectrum centroid; (5) audio spectrum spread; (6) audio spectrum flatness;

• Signal parameters: (7) fundamental frequency; and (8) harmonicity of signals;

• Temporal Timbral: (9) Log attack time and (10) temporal centroid;

• Spectral Timbral: (11) spectral centroid; (12); harmonic spectral centroid; (13)

harmonic spectral deviation; (14) harmonic spectral spread; (15) harmonic spectral variation;

• Spectral basis: (16) audio spectrum basis; and (17) audio spectrum projection.

An LLD can be instantiated as a single value for an audio segment or as series. In

MPEG-7 Audio, two different low level descriptor (LLD) Types exist. AudioLLDScalarType is useful for scalar values, such as power or fundamental frequency. AndioLLDVectorType can be used for vector types, such as spectra. Any descriptor that is inherited from one of the two types can be instantiated. More information of MPEG-7 Audio descriptors can be found in ISO document ISO/IEC JTC 1/SC 29/WG 11 N7708 (Oct. 2005), which is incorporated herein by reference.

Although the MPEG-7 low level audio descriptors are becoming widely adopted and are therefore particularly useful for characterizing a user's musical preferences, the audio player of FIG. 1 can use any conveniently defined set of low level audio descriptors to characterize a user's musical preferences.

Turning back again to the audio player of FIG. 1, beneficially audio player 100 characterizes the user's musical preferences based only on the actual music that is played for the user by audio player 100. That is, instead of averaging descriptor values from songs on a playlist or in a song database, audio player 100 uses the music to which the user actually listens to determine the low level audio descriptor values that are employed to characterize a user's musical preferences. So, for example, if a user only listens to part of a song or other musical selection, only that part of the musical selection that was actually played goes into the determination of the low level audio descriptor values used to characterize the user's musical preferences. Furthermore, if a user listens to a favorite portion of a musical selection repeatedly, then that portion will be weighed accordingly in determining the low level audio descriptor values used to characterize the user's musical preferences.

Beneficially, processor 120 determines the low level audio descriptor values over a particular time period. That is, the low level audio descriptor values are updated when the user listens to additional music. Furthermore, either automatically or under user control, optionally processor 120 may periodically reset the low level audio descriptor values to insure that the values are not unnecessarily influenced by old, stale, data that no longer reflects the user's current musical preferences.

Additionally, one or more time intervals may be excluded from the time period used to determine the low level audio descriptor values. For example, a user may provide an instruction through user interface 190 to turn-on or turn-off the musical preference characterization feature for one or more time intervals. This may be the case when the user loans audio player 100 to another person and therefore turns off the musical preference characterization feature. Also, audio player 100 may automatically exclude one or more time intervals from the time period used to determine the low level audio descriptor values. This may be the case when an embodiment of audio player 100 is equipped to receive and play analog music, but is not equipped to convert the analog music to a digital format to produce the audio samples which are needed for extraction of the low level audio features.

Audio player 100 can determine the low level audio descriptor values from the audio samples in a variety of ways.

In a first embodiment, processor 120 may compute low level audio descriptor values for each portion of music (e.g., an entire song, or only the portion of a song that is actually played) that audio player 100 plays for the user. Then, processor 120 may average each of the plurality of computed low level audio descriptor values for all portions of music that have been played to the user through audio player 100 over the particular time period.

Alternatively, processor 120 may maintain a running computation of each of the plurality of low level audio descriptors over all music that is played to the user through audio player 100 over the particular time period.

In a second embodiment, instead of computing low level descriptor values "on-the- fly" from the audio samples as each portion of music is being played to the user, processor 120 may compute the low level descriptor values "off-line" by accessing a database which contains low level audio descriptor values for each portion of music that is played to a user. For example, in this embodiment, it is assumed that low level audio descriptor values have already been computed for each portion of music that may be played to a user, and these low level audio descriptor values are stored in a database accessible by processor 120. In that case, processor 120 may compute "off-line" the low level audio descriptor values for all portions of music that are played to a user by accessing the database. For example, processor may make a "playback record" listing each portion of music that is played to a user. Then, periodically, processor 120 may use this playback record to access the corresponding low level audio descriptor values from the database, and update its

computation of the low level descriptor values for all music played to the user. Alternatively, the data may be retrieved automatically from the database and the low level audio descriptor values may be updated automatically at the conclusion of each portion of music being played to the user. Beneficially, a weighting factor may be applied to the computation of the low level audio descriptors based on one or more external factors. For example, when the audio output level is low (e.g., near 0) for a portion of music, then the audio descriptor values for that portion of the music may be given a small or reduced weighting factor, compared to low level audio descriptor values that apply when the audio output level is very high. Other external factors may also be used to applying weighting to the audio low level descriptors.

Beneficially, processor 120 stores the low level audio descriptor values in memory 140.

The low level audio descriptor values stored in memory 140 can then be used to identify additional music that matches the user's musical preferences, and recommend such music to the user. Such recommendations may be provided via user interface 190. Alternatively, the recommendation may be made by simply playing the musical selection to the user and letting the user decide whether to continue listening, or to stop playing the musical selection. If a user likes the recommended musical selection, then audio player 100 may provide the user with an option to purchase and permanently download the corresponding music file to memory in audio player 100.

For example, audio player 100 may have access to an external or remote song database (e.g., via a local area network; via the Internet; etc.). In that case, processor 120 may compare values for the plurality of low level audio descriptors for a song in the song database, with the low level audio descriptor values for music that has been played to the user through the audio playback system over the time period and stored in memory 140, and recommend the song to the user when the comparison result indicates that the song is a good match. Any desired threshold(s) may be used to determine when the comparison results indicate that the song is a "good match." Meanwhile, the plurality of low level audio descriptor values for the song may have been predetermined and already stored in the song database or, alternatively, processor 120 may compute the plurality of low level audio descriptor values for the song.

In another embodiment, audio player 100 may be configured to periodically play advertisements to a user. In particular, audio player streams musical files form a remote server (e.g., the Internet), then as part of that streaming music service, the remote server may periodically provide an advertisement to be played to the user. For example, the low level audio descriptor values stored in memory 140 can be used to identify advertisements that are targeted to user's having particular musical preferences. In one embodiment, the advertisements are assigned low level audio descriptor values that describe musical preferences of a targeted listener. In that case, processor 120 may compare values for the plurality of low level audio descriptors assigned to an advertisement, with the low level audio descriptor values for music that has been played to the user through the audio playback system over the time period and stored in memory 140, and deliver the advertisement to the user when the comparison result indicates that the advertisement is a good match.

Audio player 100 may be configured to allow a plurality of users to log-in and identify themselves so that, for example, pre-stored individual playback preferences (e.g. equalizer setting; playback mode; volume) may be employed for each user. In that case, audio player may individually characterize the musical preferences for each user by separately determining the low level audio descriptor values for the music that audio player 100 plays for each user. Beneficially, processor 120 stores separately in memory 140 the plurality of low level audio descriptor values for each user, together with an indication of the user (e.g., a user ID) to which the low level audio descriptor values correspond.

Additionally, audio player 100 may be configured to import, export, merge, or otherwise share low level audio descriptor values for a user with another device. For example, if a user purchases a new audio player 100, the low level audio descriptor values for the user from the user's old audio player 100 may be imported into the new audio player 100 so that the accumulated history of the user's music preferences will not be lost. In another example, if a user has two audio players 100, low level audio descriptor data may be synchronized or merged between the two devices. Such updating and synchronization could be done by directly connecting audio players 100, by docking audio players 100 to a personal computer and using a computer program executed on the personal computer, etc.

FIG. 2 shows a flowchart for a method of characterizing the musical preferences of one or more users which may be executed by an audio playback system such as audio player 100.

In an optional step 205, a user may log-in or otherwise identify herself/himself to the audio playback system. In that case, the audio playback system may retrieve a set of user preferences corresponding to that particular user, including a set of low level audio descriptor values corresponding to the particular user.

In a step 210, the audio playback system plays music to a user. Beneficially, this is in response to one or more instructions received from the user (e.g., selecting a playlist or a particular music file for playback.)

In a step 212, a processor determines whether the current time interval should be excluded from the characterization of the user's musical preferences. If so, then the process returns to step 210 and continues playing the music. Otherwise, the process proceeds to step 215. In a step 215, audio samples for each portion of music that is played to a user through the audio playback system are received by a processor.

In a step 220, the processor extracts from the audio samples a plurality of low level features of each portion of music that is played to the user through the audio playback system. Then, in a step 225, the processor computes, from the extracted low level features, a value for each of a plurality of low level audio descriptors for the current portion of music that has been played to the user through the audio playback system.

Next, in a step 230, the processor averages each of the plurality of computed low level audio descriptor values for all portions of music that have been played to the user through the audio playback system over a particular time period.

In step 235, the processor stores in memory the low level audio descriptor values that were determined in the preceding step.

In a step 240, the processor compares values for the plurality of low level audio descriptors for a song in the song database, with the low level audio descriptor values for music that has been played to the user through the audio playback system over the time period and stored in memory 140.

In a step 245, the comparison result is compared to one or more thresholds to determine whether the match is good.

In a step 250, when the match is good, then the song is recommended to the user. Then the process may return to step 210.

Alternatively, when the audio playback system delivers advertisements to a user, then in step 255 the processor compares values for the plurality of low level audio descriptors for an advertisement, with the low level audio descriptor values for music that has been played to the user through the audio playback system over the time period and stored in memory 140.

In a step 260, the comparison result is compared to one or more thresholds to determine whether the match is good. In a step 265, when the match is good, then the advertisement is delivered to the user. Then the process may return to step 210.

In an alternative embodiment, as illustrated in FIG. 2, steps 225 and 230 may be replaced by a single step 227 wherein the processor maintains a running computation of each of the plurality of low level audio descriptor values over all music that has been played to the user through the audio playback system over the particular time period.

While preferred embodiments are disclosed herein, many variations are possible which remain within the concept and scope of the invention. Such variations would become clear to one of ordinary skill in the art after inspection of the specification, drawings and claims herein. The invention therefore is not to be restricted except within the spirit and scope of the appended claims.