CHUA, Gim, Guan (Institute for Infocomm Research1 Fusionopolis Way,#21-01 Connexis,South Tower, Singapore 2, 13863, SG)
RAHARDJA, Susanto (Institute for Infocomm Research1 Fusionopolis Way,#21-01 Connexis,South Tower, Singapore 2, 13863, SG)
TANG, Ka Yin, Christina (Institute for Infocomm Research1 Fusionopolis Way,#21-01 Connexis,South Tower, Singapore 2, 13863, SG)
FARBIZ, Farzam (Institute for Infocomm Research1 Fusionopolis Way,#21-01 Connexis,South Tower, Singapore 2, 13863, SG)
CHUA, Gim, Guan (Institute for Infocomm Research1 Fusionopolis Way,#21-01 Connexis,South Tower, Singapore 2, 13863, SG)
RAHARDJA, Susanto (Institute for Infocomm Research1 Fusionopolis Way,#21-01 Connexis,South Tower, Singapore 2, 13863, SG)
TANG, Ka Yin, Christina (Institute for Infocomm Research1 Fusionopolis Way,#21-01 Connexis,South Tower, Singapore 2, 13863, SG)
| CLAIMS 1. A method of generating an interactive output, the method comprising the steps of: calculating and assigning a probability score to each of a plurality of motion clips based on a set of factors; selecting one or more motion clips from the plurality of motion clips based on their probability scores; and blending the one or more selected motion clips to form a continuous sequence of motion clips as the interactive output. 2. The method as claimed in claim , wherein the plurality of motion clips comprise motion clips of a virtual character. 3. The method as claimed in claim 2, wherein the interactive output comprises a continuous sequence of motion clips of the virtual character. 4. The method as claimed in any of the preceding claims, wherein the set of factors comprises the virtual character's status, a user's motions, and input music features. 5. The method as claimed in claim 4, wherein the virtual character's status comprises the virtual character's position and orientation of its joints. 6. The method as claimed in claim 4, wherein the user's motions are analyzed through kinetic energy estimation, motion entropy estimation, motion timing estimation, and dance action classification. 7. The method as claimed in any of the preceding claims, wherein the plurality of motion clips are stored in a motion clip database. 8. The method as claimed in any one of claims 4 to 7, wherein the set of factors further comprises features of a music that is being played during the animation of the virtual character. 9. The method as claimed in any of the preceding claims, wherein the step of selecting one or more motion clips from the plurality of motion clips based on their probability scores comprises the step of: weighting the plurality of motion clips based on their probability score, such that a motion clip with a higher score has a higher chance of being selected. 10. The method as claimed in claim 9, wherein the step of weighting the plurality of motion clips based on their probability score comprises the steps of: sorting the plurality of motion clips in a cumulative distribution function such that each of the plurality of motion clips is assigned a portion on the cumulative distribution function based on its probability score; and generating a random number within a range of the cumulative distribution function; wherein a motion clip is selected if the random number falls within the portion associated with the motion clip. 11. A system for generating an interactive output, comprising: a probability score generator configured for calculating and assigning a probability score to each of a plurality of motion clips based on a set of factors; a processor means configured for selecting one or more motion clips from the plurality of motion clips based on their probability scores; and the processor means also configured for blending the one or more selected motion clips to form a continuous sequence of motion clips as the interactive output. 12. The system as claimed in claim 11 , wherein the plurality of motion clips comprise motion clips of a virtual character. 13. The system as claimed in claim 12, wherein the interactive output comprises a continuous sequence of motion clips of the virtual character. 14. The system as claimed in any one of claims 1 1 to 13, wherein the set of factors comprises the virtual character's status, a user's motions, and input music features. 15. The system as claimed in claim 14, wherein the system further comprises a monitoring module configured for monitoring the virtual character's status, the virtual character's status comprising the virtual character's position and orientation of its joints. 16. The system as claimed in claim 14, wherein the system further comprises a motion analysis module configured for analyzing the user's motions through kinetic energy estimation, motion entropy estimation, motion timing estimation, and dance action classification. 17. The system as claimed in any of claims 1 to 16, wherein the plurality of motion clips of the virtual character are stored in a motion clip database. 18. The system as claimed in any of claims 14 to 17, wherein the set of factors further comprises features of a music that is being played during the animation of the virtual character. 19. The system as claimed in any claims 1 1 to 18, wherein the processor means is further configured for weighting the plurality of motion clips based on their probability score, such that a motion clip with a higher score has a higher chance of being selected. 20. The system as claimed in claim 19, wherein the processor means is further configured for: sorting the plurality of motion clips in a cumulative distribution function such that each of the plurality of motion clips is assigned a portion on the cumulative distribution function based on its probability score; generating a random number within a range of the cumulative distribution function; and selecting a motion clip if the random number falls within the portion associated with the motion clip. 21. A data storage medium having stored thereon computer program code means for instructing a computer system to execute the method of generating an interactive output, as claimed in any one of claims 1 to 10. |
FIELD OF INVENTION The invention relates to system and method of generating an interactive output.
BACKGROUND In an immersive and natural interaction between a human subject and a virtual character, for example dancing with the virtual character in a virtual dance floor or dance club environment, besides having the dance sequences and motions synchronised with the music, the motion and animation of the virtual character's dance sequences should not appear to be predictable or repetitive.
Currently, when a virtual dancing character accompanies the playing of a music file, complete pre-recorded dance motion data is associated with the entire music file. That is, in e.g. current dance game machines, there is a full-length prerecorded dance motion file for every music file. As such, every time a user selects a music file, the exact same dance motion sequence is performed.
A need therefore exists to provide a system and method of generating an interactive output that seeks to address the abovementioned shortcoming. SUMMARY
According to the first aspect of the present invention, there is provided a method of generating an interactive output, the method comprising the steps of: calculating and assigning a probability score to each of a plurality of motion clips based on a set of factors; selecting one or more motion clips from the plurality of motion clips based on their probability scores; and blending the one or more selected motion clips to form a continuous sequence of motion clips as the interactive output.
The plurality of motion clips may comprise motion dips of a virtual character. The interactive output may comprise a continuous sequence of motion clips of the virtual character.
The set of factors may comprise the virtual character's status, a user s motions, and input music features.
The virtual character's status may comprise the virtual character's position and orientation of its joints. The user's motions may be analyzed through kinetic energy estimation, motion entropy estimation, motion timing estimation, and dance action classification.
The plurality of motion clips may be stored in a motion clip database. The set of factors may further comprise features of a music that is being played during the animation of the virtual character.
The step of selecting one or more motion clips from the plurality of motion clips based on their probability scores may comprise the step of weighting the plurality of motion clips based on their probability score, such that a motion clip with a higher score has a higher chance of being selected.
The step of weighting the plurality of motion clips based on their probability score may comprise the steps of: sorting the plurality of motion clips in a cumulative distribution function such that each of the plurality of motion clips is assigned a portion on the cumulative distribution function based on its probability score; and generating a random number within a range of the cumulative distribution function; wherein a motion clip is selected if the random number falls within the portion associated with the motion clip.
According to the second aspect of the present invention, there is provided system for generating an interactive output, comprising: a probability score generator configured for calculating and assigning a probability score to each of a plurality of motion clips based on a set of factors; a processor means configured for selecting one or more motion clips from the plurality of motion clips based on their probability scores; and the processor means also configured for blending the one or more selected motion clips to form a continuous sequence of motion clips as the interactive output.
The plurality of motion dips may comprise motion clips of a virtual character.
The interactive output may comprise a continuous sequence of motion clips of the virtual character.
The set of factors may comprise the virtual character's status, a user's motions, and input music features.
The system may further comprise a monitoring module configured for monitoring the virtual character's status, the virtual character's status comprising the virtual character's position and orientation of its joints.
The system may further comprise a motion analysis module configured for analyzing the user's motions through kinetic energy estimation, motion entropy estimation, motion timing estimation, and dance action classification. The plurality of motion clips of the virtual character may be stored in a motion clip database.
The set of factors may further comprise features of a music that is being played during the animation of the virtual character.
The processor means may be further configured for weighting the plurality of motion clips based on their probability score, such that a motion clip with a higher score has a higher chance of being selected. The processor means may be further configured for: sorting the plurality of motion clips in a cumulative distribution function such that each of the plurality of motion clips is assigned a portion on the cumulative distribution function based on its probability score; generating a random number within a range of the cumulative distribution function; and selecting a motion clip if the random number falls within the portion associated with the motion dip. According to the third aspect of the present invention, there is provided a data storage medium having stored thereon computer program code means for instructing a computer system to execute the method of generating an interactive output, as described herein.
BRIEF DESCRIPTION OF THE DRAWINGS
Example embodiments of the invention will be better understood and readily apparent to one of ordinary skill in the art from the following written description, by way of example only, and in conjunction with the drawings, in which:
Figure 1a is a flow chart illustrating a method of generating an interactive output, according to an example embodiment of the present invention.
Figure 1b is a cumulative distribution function of probability scores, according to an embodiment of the present invention.
' Figure 2 is a schematic illustrating a system for generating an interactive output, according to an example embodiment of the present invention.
Figure 3 is a schematic of a computer system for implementing the system and method of animating a virtual character in example embodiments. Figure 4 is a flow chart illustrating a method of animating a virtual character, according to an example embodiment of the present invention.
DETAILED DESCRIPTION
Embodiments of the present invention provide a system and method of animating a virtual character wherein a probability score for motion clips is computed based on a set of input requirements (such as input music and a user's motion input), and subsequent motion clips to be played back are selected based on the probability score. In particular, the selection of subsequent motion clips is biased towards motion clips with higher probability values. Accordingly, a set of unique or at least pseudo-random dance sequences comprising motion clips can be generated when a dance routine is initiated between a human user and the virtual character.
In an example embodiment, a dance motion is split into smaller clips using the bar timing information of the music. Each of the smaller clips is individually tagged; and a database is built, comprising of the smaller dance motion clips. A non- repetitive dance motion sequence can be generated for the same music, each time it is played, based on the input music, tag information of each smaller clip, as well as a user's dance actions recorded and analyzed by a camera/sensor device in real-time.
Some portions of the description which foiiows are explicitly or implicitly presented in terms of algorithms and functional or symbolic representations of operations on data within a computer memory. These algorithmic descriptions and functional or symbolic representations are the means used by those skilled in the data processing arts to convey most effectively the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self- consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities, such as electrical, magnetic or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated.
Unless specifically stated otherwise, and as apparent from the following, it will be appreciated that throughout the present specification, discussions utilizing terms such as "scanning", "calculating", "determining", "replacing", "generating", "initializing", "outputting", or the like, refer to the action and processes of a computer system, or similar electronic device, that manipulates and transforms data represented as physical quantities within the computer system into other data similarly represented as physical quantities within the computer system or other information storage, transmission or display devices.
The present specification also discloses apparatus for performing the operations of the methods. Such apparatus may be specially constructed for the required purposes, or may comprise a general purpose computer or other device selectively activated or reconfigured by a computer program stored in the computer. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose machines may be used with programs in accordance with the teachings herein. Alternatively, the construction of more specialized apparatus to perform the required method steps may be appropriate. The structure of a conventional general purpose computer will appear from the description below. In addition, the present specification also implicitly discloses a computer program, in that it would be apparent to the person skilled in the art that the individual steps of the method described herein may be put into effect by computer code. The computer program is not intended to be limited to any particular programming language and implementation thereof. It will be appreciated that a variety of programming languages and coding thereof may be used to implement the teachings of the disclosure contained herein. Moreover, the computer program is not intended to be limited to any particular control flow. There are many other variants of the computer program, which can use different control flows without departing from the spirit or scope of the invention.
Furthermore, one or more of the steps of the computer program may be performed in parallel rather than sequentially. Such a computer program may be stored on any computer readable medium. The computer readable medium may include storage devices such as magnetic or optical disks, memory chips, or other storage devices suitable for interfacing with a general purpose computer. The computer readable medium may also include a hard-wired medium such as exemplified in the Internet system, or wireless medium such as exemplified in the GSM mobile telephone system. The computer program when loaded and executed on such a general-purpose computer effectively results in an apparatus that implements the steps of the preferred method.
The invention may also be implemented as hardware modules. More particular, in the hardware sense, a module is a functional hardware unit designed for use with other components or modules. For, example, a - module may be implemented using discrete electronic components, or it can form a portion of an entire electronic circuit such as an Application Specific Integrated Circuit (ASIC). Numerous other possibilities exist. Those skilled in the art will appreciate that the system can also be implemented as a combination of hardware and software modules.
Figure 1a is a flow chart, designated generally as reference numeral 100, illustrating a method of generating an interactive output, here in the form of an animated a virtual character, according to an example embodiment of the present invention.
At step 1 10, a probability generator generates a selection probability score which is associated with each motion clip that is stored in a database (described below). The selection probability score indicates how close the motion dips are able to meet the input requirements. This score is based on (i) the virtual character's status (such as position and orientation of all joints), (ii) a user's motion analysis results, and (iii) music features (e.g. music energy and drum groove pattern similarity). The probability score is calculated as described below:
Let M ={m 1 ;m 2 ;m 3 ; . ..;m N } be the motion database 102 with N motion clips, and R = {r,; r 2 ; r 3 ; ...; rj be the L requirements for the next motion. Assuming independence of the requirement factors in R in an example embodiment, if the current motion clip that is playing is m„ the probability of selecting the motion clip m, is:
p(m J \ m l , R)= p{i rri j | m, ,r, ,r 2 ....,rj p(m y I m, , r k ) can be calculated based on a similarity function f rk (.):
The similarity function f(.) may be a bell shape function with the highest value of 1 when rri j and m, are similar under requirement r k . An example of f(.) for the requirement of "r x = position along the x-axis of the virtual character" is:
-< „,,.„, -X n Y
f, (m r m, ) = e ΰ> - (3) where X 0 is the required position for the virtual character along the x-axis, X m mi is the final position of the virtual character along the x-axis if motion clip m, is played directly after the completion of motion clip m,, and a x is the variance parameter that shows the acceptable deviation range from the desired position X 0 .
The same similarity function can be used for other input requirements such as required energy level. However, for rotation, another similarity function is used in an example embodiment: l + cos(p + φ 0 )
fA ' m m,) = - (4)
where φ 0 is the required orientation angle, (p my,m/ is the final orientation of the virtual character if the motion clip m, is applied right after the completion of motion ciip m„ and σ φ is the function parameter that shows the acceptable deviation range from the desired input φ 0 .
After a selection probability score is determined for each motion clip, the motion clips are selected as described below:
The plurality of motion clips are weighted based on their probability score, such that a motion clip with a higher score has a higher chance of being selected. In an example embodiment, the motion clips are sorted in a cumulative distribution function (from 0 to 1 ) of their respective probability scores.
In an example embodiment, assume clip / has a score of 0.6, clip j is has a score of 0.2, clip k has a score of 0.15 and clip / has a score of 0.05. The clips can be sorted shown in Figure 1a (designated generally as reference numeral 150).
As such, with reference to Figure 1 a, a clip with a higher probability score (such as ciip /) is assigned a larger portion 152 on the cumulative distribution function, as compared to a clip with a relatively lower probability score (such as clip J), which is assigned a relatively smaller portion 154.
A random number from 0 to 1 (i.e.: within a range of the cumulative distribution function) is generated such that if the random number falls within the portion associated with a clip, the clip is selected. For example, if the generated random number is from 0 to 0.6, clip / ' is selected. If the random number is from 0.6 to 0.8, clip j is selected, and so on. As described above, at step 112, a unique or at least pseudo-random dance sequence is advantageously produced by considering the probabilities assigned to each motion clip. A motion clip selection module selects the next motion ciip based on the probabilities obtained from the probability generator at step 110. A ciip with a higher probability value has a higher chance of being selected. The selected motion clip is then linearly blended with the existing motion clip that is currently being played, which would fade out gradually, resulting in a smooth transition for the dance motions.
A typical animation library aliows for multiple animation tracks to animate a single character. The positions and orientations of the resultant poses are then linearly interpolated together based on weights for each track. The weights range from 0 (no effect) to 1 (full effect).
In an example embodiment, the selected motion clip constitutes an animation track and the existing motion ciip constitutes another animation track. During transition from the existing motion clip to the newly selected motion clip, the weight for the existing clip is linearly decreased from 1 to 0, and the weight for the new clip is linearly increased from 0 to 1 , over the transition period. The transition period is the amount of time taken for the existing clip to be fully faded out and the new clip to be fully animating.
At step 114, after the selection of the motion clip, a processor module can adjust the playback rate for the selected motion clip to match the input music's bar duration. In particular, the frame rate of animated paces at each moment is preferably modified in such a way that each motion pace is completed at each music beat. Accordingly, the animated dance motion of the virtual character and the input music are advantageously synchronized together.
In various other embodiments, motion clips are three-dimensional (3D) motion clips of a virtual character that may be obtained via motion capture technology or by manual key-framing. Figure 2 is a schematic, designated generally as reference numeral 200, illustrating a system for generating an interactive output, here in the form of a bidirectional interactive dance with an animated virtual character, according to an example embodiment of the present invention. An image capturing device 202 (e.g. a camera) captures a user's motion and sends it to a user's motion analysis module J 204. The user's motion analysis module 204 is configured to analyze the user's motion data through kinetic energy estimation, motion entropy estimation, motion timing estimation and dance action classification (wherein the user's motion is classified- as one of basic dance moves (e.g. moving forward, backward, spinning, rising hands, etc.).
The user's motion analysis module 204 advantageously provides analyzed data such as required dance energy level, required dance actions, required position and orientation of a virtual character so that the virtual character can appear in front of a user and faces the user during the virtual character's dance performance.
Music features from a music generator 2 0 are fed to a score generator 2 2. The score generator 212 provides a game score to the user based on how his/her motions are synchronous with the input music.
A processor means, here in the form of a dance synthesis module 216, animates a virtual character based on the results obtained from the motion analysis module 204, the dance motion clips obtained from a motion database 214, and the music from the music generator 210. in addition, the dance synthesis module 216 is configured to vary the virtual character's eye gaze and facial expressions. The virtual character is animated based on the selection of one or more motion clips, where each of the selected motion clips are first assigned a probability score by a probability score generator 217. Thereafter, a motion clip selection module 218 selects the one or more motion clips using the method according to an example embodiment, as described above.
The dance motion clip database 214 holds a plurality of different dance routines. Each dance routine includes several animated dance motion clips. Each motion clip is tagged with information pertaining to the particular clip. This information includes the total change in position and orientation of a biped character within the clip; the variance of orientation and movements in the X, Y, and Z directions; the energy level of the character's movements; and the types of motions performed within the clip. All this information is stored in the motion database 214.
A display module 226, displays the synthesized dance in 3D. Embodiments of the present invention advantageously enable realistic pseudo-random dance animation synthesis that can provide a set of unique or at least pseudo-random dance sequences, with or without a user's explicit input.
The method and system of the example embodiment can be implemented on a computer system 300, schematically shown in Figure 3. It may be implemented as software, such as a computer program being executed within the computer system 300, and instructing the computer system 300 to conduct the method of the example embodiment.
The computer system 300 comprises a computer module 302, input modules such as a keyboard 304 and mouse 306 and a plurality of output devices such as a display 308, and printer 310.
The computer module 302 is connected to a computer network 312 via a suitable transceiver device 314, to enable access to e.g. the Internet or other network systems such as Local Area Network (LAN) or Wide Area Network (WAN).
The computer module 302 in the example includes a processor 318, a Random Access Memory (RAM) 320 and a Read Only Memory (ROM) 322. The computer module 302 also includes a number of Input/Output (I/O) interfaces, for example I/O interface 324 to the display 308, and I/O interface 326 to the keyboard 304.
The components of the computer module 302 typically communicate via an interconnected bus 328 and in a manner known to the person skilled in the relevant art.
The application program is typically supplied to the user of the computer system 300 encoded on a data storage medium such as a CD-ROM or flash memory carrier and read utilising a corresponding data storage medium drive of a data storage device 330. The application program is read and controlled in its execution by the processor 318. Intermediate storage of program data maybe accomplished using RAM 320. The method and system according to the various embodiments described are applicable to a wide range of entertainment systems, such as multiplayer online games, single player games, and virtual world simulations. The method and system according to the various embodiments are also applicable to edutainment and training systems.
Figure 4 is a flow chart, designated generally as reference numeral 400, illustrating a method of generating an interactive output, according to an example embodiment of the present invention. At step 402, a probability score is calculated and assigned to each of a plurality of motion clips based on a set of factors. At step 404, one or more motion dips are selected from the plurality of motion dips based on their probability scores. At step 406, the one or more selected motion clips are blended to form a continuous sequence of motion clips as the interactive output. It will be appreciated by a person skilled in the art that numerous variations and/or modifications may be made to the present invention as shown in the embodiments without departing from a spirit or scope of the invention as broadly described. The embodiments are, therefore, to be considered in all respects to be illustrative and not restrictive.