CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No.

61/288,674, filed December 21, 2009, which is incorporated by reference in its entirety.

BACKGROUND

[0002] Smart toys such as robots or interactive dolls are increasingly being made to give the impression of intelligence and behavioral interactivity with children. These toys fall short of the ideal objective because of several limitations.

[0003] First, cost is a major concern. Sophisticated toy behavior can only be currently implemented using a combination of software and hardware that dramatically increases the cost of toy. Second, there is also the limited space inside a toy puppet or doll to include all the necessary electronics, sensors, and mechanisms. Therefore, most current 'smart toys' are not really smart at all due to the limited hardware and software capabilities built into them. This limits also the behavioral repertoire that can be programmed into them.

[0004] The trade-off between sophistication and hardware costs, typically results into the limited interactivity built into such toys. Interactivity involves communication and

responsiveness to specific situations arid contexts. However, most puppets and dolls lack the necessary abilities to recognize speech or gestures and respond verbally and nonverbally, e.g., facial and head gestures. Such communication abilities are necessary for a believably responsive toy character. There are cases, however, that voice capabilities are included in the toy. The novelty introduced by this feature quickly drops as the child realizes the toy simply repeats the same phrases if the toy's recorded voice storage is limited and has no means of being updated. [0005] This confluence of factors between cost, limited communicative and behavioral repertoire is the major reason there is lack of toys that are smart and emotionally responsive.

It is interesting to note that animal pets such as dogs and parrots can be more intelligent and emotionally responsive compared to the most expensi ve toy robots currently available.

[0006] The solution presented in this invention takes inspiration from the successful children's entertainment industry that relies on puppetry! TV shows for children like Sesame Street, the Muppet Show and Barney use. puppetry to create the characters that have made these shows popular with children. To a child, these characters look real because the characters have names, personalities, and individual stories. Also, each episode develops the characters' personality while being used for teaching simple concepts to children; The key elements that make a puppet character believable and likable are the following:

• Natural human voice and speech. The voice is unique to the character and is consistently used throughout the lifetime' of the character.

• Synchronized head and mouth movements. Timing with the voice and story is important. Nonverbal communication is conveyed through these channels and is critical to giving the appearance of believable human-like interaction.

• TTie use of funny stories or episodes. Each episode uses a consistent set of themes that makes a mini-universe for the characters that inhabit it.

• Each character has a consistent and unique personality. For example, a puppet character has certain key phrases or expressions that he regularly uses.

• Each character has a perceived age and intelligence that is slightly older than the child.

Most characters have non-human forms. Typically. their bodies are animal-based or dinosaur-like or even monster-based. The use of cartoonish appearances avoids the well-documented uncanny valley effect that people experience when the artificial similitude to the human form or behavior is very near but not perfectly human.

• Typically has fur like a bear or a monster or feathers like a bird. This adds

additional touch modalities associated with comfort.

[0007] Successful children's TV shows have all these elements but children experience it through a limiting medium that is flat (two-dimensional video). Also, interaction with the characters is impossible. Despite these limitations, children can relate with the show and the story which can sometimes includes real humans or even children. Outside the show, in real life, the children are motivated to buy the furry doll to try to recreate the TV experience by interacting with it using their imagination and the remembered stories or personalities from the show. The experience is not full because the toy doll does not behave like the one in the show and relies solely on the creative imagination of the child.

[0008] In TV scenes where a puppet interacts with a human child another key insight emerges: social-behavioral fidelity is paramount over fidelity to human-like form. The child believes and accepts the puppet as a real social being capable of displaying emotions through voice qualities and human-like correctness of nonverbal cues such as head nods and facial expressions (such as blinks or eye/eye-brow movements when such abilities are built into the puppet). Social behavioral fidelity also means responsiveness and sensitivity to the child's emotional state in addition to the context of the scene. This life-like puppet behavior is possible in traditional puppetry because the puppet is being controlled by a human, the puppeteer, who can detect the emotional state of the child and is sensitive to the ongoing interaction between the child and puppet.

[0009] The motivation for developing this platform is to make these toys come alive with the child by providing the technologies to make puppets behave like the ones in the TV show. The platform brings the puppeteer's intelligence beyond the stage or recording studio and into every playroom and living room. With the platform, an ongoing toy-character universe can be Unfolded and adapted to the child's age and personal history with the puppet. Hence, the puppet can become a friend, teacher, companion and guardian to the child.

[00Ϊ0] The present invention and its various embodiments create an ecosystem beneficial for children's TV show producers, toy companies, game developers, parents, researchers, and children. Producers of current children's shows using puppets can use the platform to make their stories more interactive by creating puppet behavior databases for toy puppet versions of their characters. Toy companies can use the platform to make more socially interactive puppets without significantly raising the cost of the toy. Updating puppet behaviors by continually creating more sophisticated puppet behavior databases provide ew revenue streams for toy companies, TV show producers, or independent developers. Game developers for the console market can now target the younger age group by creating a new type of game environment where one of the characters is present with the player as a robotic puppet and is also the interface to the game. Parents benefit by using online services that can keep track of the child's behavior and long-term development while playing with the puppets. Researchers in human-robot interaction will have a stable platform to conduct studies and also a robotic platform for testing advanced intelligence routines that can recognize emotions and react naturally to the child. Such research can be eventually be used by developers, toy companies, and TV show producers. Finally, the children benefit by having a toy companion while learning from a TV show or while playing a computer game with the puppet.

[0011] Other advantages of the present invention become more apparent by a more detailed description based on the following embodiments of the present invention and the accompanying drawings. SUMMARY

[0012] A platform and system for making robotic toys or puppets that can behave as if it is part of an ongoing children's TV show or computer game while simultaneously being responsive to a child user. A puppet can take on a variety of appearances. In general, a puppet resembles a character or even an object in a story and is able to exhibit behaviors that create a more immersive experience for the user. The platform consists of a puppeteer base or puppet base station that is in communication with one of or more puppets within its range. The story meta-data for various media formats, such as television shows and video games, and corresponding puppet behavior database are downloaded from the internet using online puppet services. Real time sensor data from puppets in combination with decoded story timing and meta-data is analyzed at the puppeteer base to select the appropriate puppet behavior consisting of joint movements and voice data which is played back by the puppet. The puppet base station transmits information to the puppet indicating when a certain puppet behavior or action should be performed. Puppet behaviors are developed and tested using puppetry development tools and to make stories or games compatible with the platform. In another embodiment of the present invention, a telepuppetry-based business method is provided.

Brief Description of the Drawings

[0013] FIG. 1 is a diagram showing the elements of the platform in its preferred embodiment;

[0014] FIG. 2 is a block diagram of a puppet showing its various functional elements;

[0015] FIG. 3 is a block diagram showing the functional components of the puppeteer base in one embodiment;

[0016] FIG. 4 is an alternative embodiment of the puppeteer baseband

[0017] FIG. 5 is ah embodiment of the puppetry development tool. [0018] The figures depict various embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternati ve embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.

DETAILED DESCRIPTION

[0019] Referring to Figure 1, the primary goal of this entire system 1 is to provide a live- puppetry experience for a child while watching a compatible story 22 on a video display device 14 such as a TV. It is designed such that the puppet is physically in the presence of the child and behaves in a way that is responsive to the child as well as the context provided by the story 22. One scenario, by way of example, is that the puppet sings along with the other characters in the story 22 or even asks the child to sing along. Another scenario is that the puppet can ask the child to participate by asking questions that can help one of the other characters in the story 22. This technique of using puppets 12 to ask the child watching the TV is used often in children's educational TV shows. However, unlike the traditional TV shows, this time the puppet is not on the TV screen but right beside the child. In this manner, the educational and entertainment benefits of children's TV shows is enhanced by the live interactive presence that traditional puppetry provides.

[0020] To realize this live puppetry experience, the telepuppetry platform is a system comprised of one or more robotic puppets 12 that are in wireless communication 24 with a puppeteer base 10. The puppeteer base 10 controls each puppet's motion and speech. The puppeteer base 10 also receives all the sensor data from each of the puppets 12 for interpretation of the child responses to the ongoing story 22. The story 22 can take the form of a live children's show video e isode or one downloaded beforehand from the Internet 20 using a set of online puppet services 18 and stored on the puppeteer base 10. In another embodiment, a story 22 may also take the form of 2D or 3D game environment with the puppet serving both as a character and the game interface to effect changes in the story line.

[0021] The system also includes a set of puppetry development tools 16 that provide the means to make stories, in the form of videos or games, compatible with the platform. The set of puppetry development tools 16 also provides the means for creating and testing puppet behaviors and different combinations of puppets 12. The set of puppetry development tools 16, made available via the Internet 20, can be used by both professionals and hobbyists to make new puppet behaviors for old stories, or new stories for old puppets 12. Stories and puppet behaviors developed using the tools can be published for both free and commercial use using online puppet services 18. Online puppet services 18 also provide the user community access to an online library of stories and compatible puppet behaviors that can be downloaded individually or, preferably, automatically by subscription to educational services supporting the platform. Online puppet services 18 extend the possible uses of the platform beyond puppetry. For example, parent-centered services can take advantage of the platform by monitoring their children's behavior and learning progress through the collected puppet sensor data that may be analyzed professionally by educators or child behavior specialists. The insight gained from the analysis of child behavioral data recorded using the puppeteer base 10 can be used for customizing puppet behaviors that are specific to the child's stage of emotional, intellectual, and social development.

[0022] Referring now to Figure 2, a puppet includes a main controller 42, memory storage 44, power supply 36, wireless communication 24, joint control 38, sound speakers 30, and various sensors. The main controller 42 runs a software program that manages the different functional elements of the puppet. The main -controller 42 may be implemented using microcontrollers or similar microprocessors with adequate computing capability as can be determined by those familiar in the art of embedded systems. Memory storage 44 is included for storing the puppet software and for storing of parameters related to the operation of the puppet. Memory storage 44 may take the form of solid-state memory devices such as flash memory which have low power consumption and hold data in a non- volatile manner. A puppet also has a power supply 36 unit that provides electric power for all electronic components and joint actuators 40. Typically, removable batteries that preferably are rechargeable are used for supplying the necessary power.

[0023] The wireless communication 24 element provides the means to communicate with the puppeteer base 10 in an untethered manner to allow flexibility in the use of the puppet while increasing safety for the intended child users of the puppet. The specific mechanism for wireless communication 24 should provide sufficient bandwidth to allow real-time transfer of sensor data from the puppet to the puppeteer base 10 and the receipt of real-time behavioral commands to the puppet. For example, any of the WiFi standards such as 802.1 la/b/g/n or Bluetooth communication can be used for the wireless communication 24. The use of common wireless standards can help reduce costs and are constantly improving in important features such as power consumption and communication bandwidth.

[0024] The joint control 38 element of a puppet provides a means to control puppet joint motions by controlling one or more joint actuators 40. It uses the joint sensors 41 for feedback to implement an algorithm for controlling one or more variables such as position, velocity, and torque. This algorithm for controlling joint variables may be implemented in hardware as dedicated circuitry or as software routines running in the main controller 42. Desired puppet movements are executed by the main controller 42 by sending the appropriate signals to joint control 38 which then executes the control algorithm for controlling each joint actuator.

[0025] Each puppet includes sensors for sound, proprioception, touch, and vision. A sound system 32 captures signals from the microphone 28 and also plays back audio via the speakers 30 it drives. One or more touch; sensors 34 placed at strategic locations on the puppet's skin are used for detecting tactile gestures received by the puppet. The joint sensors 41 of the puppet provide the proprioceptive feedback: In combination with touch,

proprioceptive feedback can be used for detection of gestures such as a child shaking the puppet's hand, a child tapping the puppet's face or head, or a child hugging the puppet. One or more cameras provide visual sensory data for the puppet. All sensory inputs are sent to the puppeteer base 10 for real-time analysis, recording and interpretation. Although some minimal processing of the sensor data may be done at the puppet, it is preferred that the computationally demanding processing be performed at the puppeteer base 10 which has more significant computing resources. In effect, without the puppeteer base 10 a puppet cannot demonstrate the sensitivity to the Ongoing story 22 and responses of the child. It requires a puppeteer base 10 that has the computational intelligence to interpret the puppet's sensors and coordinate the puppet's movement and speech in relation to the ongoing interaction between child, story 22, and puppet.

[0026] The elements of the puppet as shown in Figure 2 are shown separately, but in practice, as is known to those familiar to 15 the art of building electronic systems, these may be highly integrated into just a few integrated circuits using a system-on-a-chip design, for example. The variety of sensors that can be used is numerous and those shown in the Figure 2 are by way of example only, to illustrate the functionality of the puppet. For example, accelerometers can be included to detect various puppet movements such as when the child shakes the puppet's body, hand, or head. Another possible sensor is a gyroscope and electronic compass that can be used for determining the orientation and movement of the puppet in relation to the video display device 14, the floor, and the child.

[0027] Now referring to Figure 3, we discuss one embodiment of the puppeteer base 10. Each puppet can receive and execute movement commands and playback audio received wirelessly from a puppeteer base 10. Each puppet also communicates wirelessly to the puppeteer base 10 the sensor data it receives from puppet's microphone 28, camera 26, touch sensors 34, joint sensors 41. Puppet joint commands 70 and voice data 72 are sent from a puppeteer base 10 to one or more puppets 12 within its range. A collection of puppet joint commands 70 and voice data 72 make up a puppet behavior database 66 that are specific for a TV episode for a children's show (here called a story 22) and are downloaded to the puppeteer base 10 via the Internet 20. The video for a story 22 is watermarked with timing data 54 and story meta-data 52 that is detected by the puppeteer base 10.

[0028] The operation of the puppeteer base 10 is explained by showing the relationships between the internal functional blocks, one or more puppets 12 and video display device; 14 Watermarked video 50 from live TV or a recorded TV show containing embedded time and story meta-data 52 is obtained using a video capture 48 device. The captured video is processed by the meta-data decoder 46 to extract said timing data 54 and story meta-data 52. At the same time, sensor data from each puppet is continuously sent to the puppeteer base 10 for processing, interpretation and recording. Interpretation of puppet sensor data is done by the sensory interpretation module 56 containing one or more of the following submodules: speech recognition 58, gesture recognition 60, tactile gesture recognition 62, facial expression recognition 76, face recognition 78, emotion recognition 80. To assist in recognition and interpretation of the sensor data, the context of the story 22 is input to these modules in the form of the extracted story meta-data 52 and timing information. The outputs of the recognition modules and the story meta-data 52 and timing data 54 are used by the behavior selection logic 64 to choose and appropriate behavior for the puppets 12. The puppet behavior selection logic 64 uses a puppet behavior database 66 appropriate for the story 22 being played. The selected puppet behavior 68 is translated into joint commands 70 and voice data 72. These joint commands 70 arid voice data 72 are sent to the puppets 12 using the wireless communication 24 means between the puppeteer base 10 and the puppets 12.

[0029] In each of the puppets 12, the received joint commands 70 are executed by the joint controller to move the actuators 40 of the puppet to effect the desired motion. Joint commands 70 may take the form of joint position, velocity, or torque values. Voice data 72 is played back on the puppet's sound system 32 in synchrony with the puppet's mouth, head, and body movements. In this particular embodiment illustrated by Figure 3, the puppets 12 can be made to appear to be participating with the story 22 being shown on the TV. Also, since the puppet behavior 68 is dependent on the interpreted puppet sensor data, the puppets 12 can be made to appear naturally responsive to the child's verbal and nonverbal responses and the child's emotional state. The puppeteer base 10 can be implemented using a computer such as a desktop PC, laptop, or entertainment PC equipped with a video capture 48 device, internet 20 connectivity, and a compatible wireless communication 24 device. Alternatively, it can also be implemented on a gaming console with internet 20 connectivity, a compatible wireless communication 24 device, and an external video capture 48 device attached to the console by standard ports such as the universal serial bus (USB).

[0030] Referring to Figure 4, another embodiment of the platform is illustrated. In this embodiment, the puppeteer base 10 implemented either on a PC or game console, such as the XBOX 360 or SONY PlayStation 3, and the video and audio is generated by the said PC or game console. The story 22 is implemented as a computer game using standard 3D or 2D game technology for the said PC or game console platforms. The story meta-data 52 and timing data 54 is extracted from the state of the game engine 74 implementing the story 22. Similar to the live TV embodiment, the story meta-data 52, timing data 54 and the interpreted puppet sensor data are used by the behavior selection logic 64 to determine the appropriate puppet movements and voice response of each of the participating puppets 12. Such joint movements and voice data 72 are retrieved from a puppet behavior database 66. The selected joint movements are translated as joint commands 70 and together with the voice data 72 are sent to each of the puppets 12 via the wireless communication 24 link. What is different in this embodiment is that the selected puppet behaviors are used as inputs to the game engine 74. The progression of the story 22 is thus affected by the puppet behavior 68, which in turn is influenced by the detected verbal and nonverbal responses of the child. Thus, in this embodiment, the use of a game engine 74 allows for extensive story 22 branching - the path through which is determined by the child by interacting with the puppet. The puppet or puppets 12 can be considered a novel gaming interface where speech and gestures can be used to communicate with each puppet. For example, at one point in the story 22, a choice between two doors must be made and at that point the puppet can face the child and ask "which door do you think we should take?" The child can respond by pointing or say "let's take the left one". The gesture recognition 60 module uses the video coming from the camera 26 of the puppet and the story 22 context (that there are 2 doors, one on the left, one on the right) to infer the child's answer. If the response was verbal, the speech recognition 58 module would interpret the child's response. In the event that the child's response cannot be inferred, the behavior selection logic 64 could make the puppet ask the question again or ask if the child wishes to do something else (if the emotion recognition 80 system has detected boredom, for example). The possibilities are only limited by the creator of the story 22 and the associated puppet behaviors compatible with the story 22.

[0031] Referring now to Figure 5, an embodiment of the puppetry development tool for creating and testing puppet behaviors for use with videos of children's shows is discussed in detail. This software running on a PC or workstation imports videos and provides the means to encode story meta-data 52 into the video frames using standard digital watermarking techniques. It also interfaces with one or more puppets 12 to capture the joint movements which can be initiated by clicking the puppet record button 98. Recorded puppet movements and recorded puppet voice data 72 are saved together with the story meta-data 52 in a puppet

; behavior database 66 The use of the software is similar to video editing software packages and is discussed in the succeeding paragraphs.

[0032] There is a story timeline 82 that shows the timing relationship of the tracks that make up a story 22. The first track is the video track 88, containing at least one video clip 90 of the story 22. The editing and encoding process typically begins by dragging one of the video clips from the Video clips collection 102 onto the ideo track 88. Then, there is at least one audio track 104 containing the original sound of the video clip 90 is loaded. A puppet is selected which creates at least one puppet voice track 92 and at least one puppet movement track 94 for each puppet supporting the story 22. Then there is the meta-data track 96 that contains the meta-data segments corresponding to a particular scene in the video. Playing back the story 22 can be done using either the timeline controls 110 or the video playback controls 112. The video scene corresponding to the current time is displayed in the video preview 84 window. If puppet movements are available for that scene snapshot, the puppet preview 86 displays the current pose of the puppet.

[0033] Puppet behaviors can be recorded using a puppet connected to the PC or workstation running the puppet development tool. This puppet is operated in passive joint mode or puppeteering mode when recording the joint movements. The puppet can also be controlled to demonstrate playback of the movements during the editing process. The speech of the puppet is recorded onto the puppet voice track 92 using an actor's voice. Puppet playback can be controlled using the puppet playback controls 100. A 3D graphical representation of the puppet is also shown in the puppet preview 86 window. This 3D puppet is animated during recording and playback of puppet joint movements. Puppet behaviors can also be synthesized using the puppet development tool or selected from various pre-define puppet behaviors. Puppet behaviors are then encoded as meta-data associated with a story such as a TV show or video game.

[0034] Story meta-data 52 can be edited on a per scene basis. The meta- data segment 108 highlighted on the timeline is shown in detail in the meta-data editor window 114. The validity of the meta-data syntax is checked against an xml schema describing how such xml documents are formed. As an example, RRL, a Rich Representation Language can be used as the xml standard for describing scenes, characters, semantic content, and the interaction between characters. To support multiple standards and conversion between them, an .

ontology-based description language is more suitable. The meta-data is encoded in the video by pressing the encode story button 106. Alternatively, only timing information can be encoded in each video frame to reduce the amount of data being encoded using digital watermarking techniques. The timing data 54 can be used at the puppeteer base 10 to retrieve the appropriate story meta-data 52 that was downloaded and stored in the puppeteer base 10 in advance.

[0035] The creation of puppet behaviors and story 22 editing takes a different process when the puppetry development tools 16 are incorporated as part of a game development engine. Each puppet is a character in the game or story 22 but also serves as the input device to substitute for joystick or game pads. The puppet can be simulated in 3D graphics with its physical behavior modeled to represent the real puppet. However, not all the sensing abilities of the puppet can be simulated during game development. For example, the tactile sensors which can be located in the various parts of the puppet are difficult to simulate using mouse and keyboard. It would be simpler to use a real puppet during game development. That way, during testing, touch responses are simply done by physically touching the puppet. Another example is detecting visual gesture responses. Since the camera 26 is presumably mounted on the puppet in the head, gestures can only be recognized if the child or user is within the field of view of the puppet's camera 26. The behavior selection logic 64 in this case should start the "face the child" behavior which could result in head movement before asking the question. Such details are best anticipated when real puppets 12 are used during game or story 22 development.

[0036] The foregoing description of the embodiments of the invention has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above teachings.

[0037] In addition, the terms used to describe various quantities, data values, and computations are understood to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as "processing" or "computing" or "calculating" or

"determining" or the like, refer to the action and processes of a computer system or similar electronic computing device, which manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission, or display devices.

[0038] Embodiments of the invention may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

[0039] Embodiments of the invention may also relate to a computer data signal embodied in a carrier wave, where the computer data signal includes any embodiment of a computer program product or other data combination described herein. The computer data signal is a product that is presented in a tangible medium and modulated or otherwise encoded in a carrier wave transmitted according to any suitable transmission method.

[0040] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description above. In addition, embodiments of the invention are not described with reference to any particular programming language. It is appreciated that a variety of programming languages may be used to implement various embodiments of the invention as described herein, and any references to specific languages are provided for disclosure of enablement and best mode of embodiments of the invention.

[0041] Finally, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, the disclosure of the embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.