Field

The present disclosure relates generally to interactive devices and, in particular, to control of interactive devices. Background

Interactive devices that are intended to interact with users, such as interactive toys, often include components to produce motions and/or sounds that are directed to a user. In some cases, an interactive device may also include sensors, and the interactive device reacts to the sensors being activated, by push buttons being pushed for example, to produce the motions/sounds.

However, such interactive devices often require some sort of action by the user, e.g., pushing the push buttons in the example above. As such, there is a need to improve interactive devices and associated control techniques.

Summary

An aspect of the present disclosure relates to an interactive device capable of interaction with a user of a media stream. The interactive device comprises: an interface to receive interaction control information related to the media stream; and an interaction response generator, coupled to the interface, to generate an interaction response by the interactive device based on the interaction control information. The interaction response is separate from playback of the media stream.

A server to convey interaction control information for animating an interactive device is also disclosed. The server comprises a control information source to provide the interaction control information for animating the interactive device. The interaction control information is related to a media stream and the animating is separate from playback of the media stream. The server also comprises: a content source, coupled to the communication interface, to provide the media stream; and a communication interface, coupled to the control information source and to the content source, to convey the interaction control information and the media stream.

A control device according to another aspect comprises: a control information generator to generate interaction control information for animating an interactive device. The interaction control information, as noted above, is related to a media stream, and the animating is separate from playback of the media stream. The control device also comprises: a synchronizer, coupled to the interaction control information generator, to synchronize the interaction control information with the media stream; and a communication interface, coupled to the control information generator, to convey the interaction control information synchronized with the media stream.

A further aspect relates to a non-transitory processor-readable medium storing: a media stream playable by a media player for presentation to a user; and interaction control information that is configured to be unplayable by the media player and directed to an interactive device capable of interaction with a user. The interaction control information causes an interaction response by the interactive device. The interaction response is related to the media stream and separate from playback of the media stream by the media player. Methods are also disclosed. A method according to one aspect comprises: receiving, at an interactive device that is capable of interaction with a user of a media stream, interaction control information related to the media stream; and generating an interaction response by the interactive device based on the interaction control information, the interaction response being separate from playback of the media stream.

Another aspect relates to a method to convey interaction control information for animating an interactive device. The method comprises: providing the interaction control information for animating the interactive device, the interaction control information being related to a media stream and the animating being separate from playback of the media stream; providing the media stream; and conveying the interaction control information and the media stream.

A method according to a further aspect comprises: generating interaction control information for animating an interactive device, the interaction control information being related to a media stream and the animating being separate from playback of the media stream; synchronizing the interaction control information with the media stream; and conveying the interaction control information synchronized with the media stream. Another method comprises: storing, on a non-transitory processor- readable medium, a media stream playable by a media player for presentation to a user; and storing, on the non-transitory processor-readable medium, interaction control information that is configured to be unplayable by the media player and directed to an interactive device capable of interaction with a user, the interaction control information causing an interaction response by the interactive device, the interaction response being related to the media stream and separate from playback of the media stream by the media player.

Other aspects and features of embodiments of the present disclosure will become apparent to those ordinarily skilled in the art upon review of the following description.

Brief Description of the Drawings

Examples of embodiments of the invention will now be described in greater detail with reference to the accompanying drawings.

Figure 1 is a block diagram of a system in which embodiments of the disclosure may be implemented. Figure 2 is a block diagram illustrating a system according to another embodiment of the present disclosure.

Figure 3 is a block diagram of a system according to another embodiment.

Figure 4 is a block diagram of a system according to a further embodiment.

Figure 5 is a block diagram of yet another example of implementation.

Figure 6 is a block diagram illustrating an example intelligent device.

Figure 7 is a block diagram illustrating an example server. Figure 8 is a block diagram illustrating an example control device.

Figure 9 is a block diagram illustrating another example system implementation, with additional internal details of an example intelligent device.

Figure 10 is a data flow diagram illustrating operation of a system according to another embodiment of the present disclosure. Figures 11-14 are flow diagrams illustrating example methods according to further embodiments.

Detailed Description

The present disclosure relates in part to an interactive, multi-feature, intelligent device such as a toy, which in some embodiments is highlighted by an innovative control protocol between a server side (e.g., Cloud) and a client side, which could include both a mobile device and the interactive device, for example. The interactive device could have multiple modes of operations, and thus could be used as either a stand-alone multi-functional device, or an intelligent device that interacts with users through any of a variety of input/output signals, including but not limited to lights, sound, vibration, and/or motion.

The development of this intelligent toy/device involves innovative and scalable design in both software and hardware. It can be further made into devices for other applications, such as Smart Home, Health Care, Personal Assistant, Education and Entertainment, etc.

Figure 1 is a block diagram of a system in which embodiments of the invention may be implemented.

The example system 100 in Figure 1 includes an intelligent device 110, a media player 114, a server 116 and a controller 118. Although a system could include one or more intelligent devices 110, media players 114, controllers 118, and servers 116, only one example of each these components has been shown in Figure 1 to avoid congestion in the drawing. It should therefore be appreciated that the system of Figure 1 , as well as the contents of the other drawings, are intended solely for illustrative purposes, and that the present invention is in no way limited to the particular example embodiments explicitly shown in the drawings and described herein.

In an embodiment, the intelligent device 110 is an interactive device that is capable of interacting with a user of a media stream. In particular, the intelligent device 110 includes one or more interfaces to receive input signals from the controller 118 and/or from the user, and also includes one or more components that are controllable to react to such input signals by emitting a sound, by moving, by vibrating, by displaying an image or a video, and the like. The intelligent device 110 could be an interactive intelligent toy, for example. More detailed examples of intelligent devices, as well as interfaces, input devices, and other components that could be provided in such devices, are provided below. The media player 114 is a device that is capable of playing a media stream. In particular, the media player 114 could include one or more input devices or interfaces to receive inputs from the user to start, pause, stop, and/or otherwise control playback of the media stream. The media player 114 may also be able to interact with other devices by any of various means, for example by a wired connection, Bluetooth, Wi-Fi, NFC, infrared signals, radio signals, and so on. A display device such as a television is an example of the media player 114. A media player 114 could be implemented in multiple parts, such as a television, monitor, or other display device connected to a separate receiver for example, in some embodiments.

The particular implementation of the media player 114 could be dependent upon the type(s) of media stream(s) to be supported. Some types of media player could be capable of playing images, video, and audio, whereas others might only play audio, for example. Media player type could also or instead be dependent on how media streams are to be received. A media stream could be streaming media that is streamed to the media player 114 from the server 116, or from another content source in other embodiments. In other embodiments disclosed herein, a media stream could be played from a storage device or medium such as a DVD or Blu-ray disk, in which case the media player need not necessarily be capable of communications with the server 116.

The server 116 communicates with the media player 114 and the controller 118 in the example system 100, and is cloud-based in some embodiments. Communications with the server 116 could be through any of various types of connections or protocols, and in some embodiments server communications are through an internet connection.

Illustrative example embodiments of a server 116 are also provided below. The controller 118 controls the intelligent device 110 in the system 100, and could be implemented in any of various ways. For example, in some embodiments the controller 118 is implemented in another type of intelligent device such as a computer, a smartphone, or a tablet. The controller 118 includes components to enable communications with the intelligent device 110, and could support wired connections and/or wireless connections (e.g., Bluetooth, Wi-Fi, NFC, infrared signals, radio signals). The controller 118 also includes components to enable communications with the server 116, using an internet connection for example. An internet connection could be established through Wi-Fi, LTE, 4G and/or 5G networks in some embodiments. Different components, or some common components, could be used in supporting communications by the controller 118 with the server 116 and the intelligent device 110.

In some embodiments, the controller 118 is part of the intelligent device 110, and communicates with the media player 114 using wireless connections and with the server 116 using an internet connection. In other embodiments, the controller 118 is part of the media player 114, and communicates with the intelligent device 110 using wireless connections (e.g., Bluetooth, Wi-Fi, NFC, infrared signals, radio signals) and with the server 116 using an internet connection. In yet other embodiments, the controller 118 is part of the server 116 and communicates with the intelligent device 110 and the media player 114 using an internet connection.

In an embodiment, the system 100 enables video streaming from the server 116 to the media player 114, and the media player 114 periodically (each second, for example) reports its time sequence to the server 116. That time sequence, in a time sequence file for example, is then downloaded from the server 116 by the controller 118, and the controller 118 synchronizes to the video playtime from the server 116 by adjusting (if needed) or maintaining its local time serial sequence for controlling the intelligent device 110. The controller 118 communicates control information to the intelligent device 110 according to the local time serial sequence. The controller 118 / intelligent device 110 communications could be via Bluetooth Low Energy (BLE), for example.

The controller 118 may thus communicate with the intelligent device 110 by sending interaction control information to the intelligent device 110, and that interaction control information triggers an interaction between the intelligent device 110 and a user. The controller 118 may also communicate with the media player 114 to obtain information regarding the media stream that is being read / played by the media player 114. For example, a code, a signal, a digital key, or the like, may be provided by the media player 114 to the controller 118 to characterize the media stream that is being played by the media player 114. The controller 118 may further communicate with the server 116 to obtain and refresh instructions regarding the signals that are to be sent to the intelligent device 110.

The foregoing represents one illustrative example of how the system could operate in one embodiment. Other embodiments are also contemplated. For example, Figure 2 is a block diagram illustrating a system according to another embodiment of the present disclosure.

The example system 200 is substantially similar to the example system 100 in Figure 1 , but in the example system 200 the controller 218 also includes an interface or input device for receiving inputs from the media player 214, illustrated by the connection 202. The controller 218 could be implemented in much the same way as the controller 118 in Figure 1 , but could also include one or more scanners to scan an input in the form of a code, image, video, and/or audio from the media player 214, for example, and be configured to control the intelligent device 110 based on the scanned input. In such an embodiment, the media player 214 could be implemented in the same manner as the media player 114. In other embodiments, the media player 214 could also include one or more additional components to enable other types of communication with the controller 218, or components that are used to communicate with another device such as the server 116 could also be used to communicate with the controller 218. The server 116 and the intelligent device 110 could be implemented in the system 200 in the same manner as discussed above with reference to Figure 1.

Figure 3 is a block diagram of a system according to another embodiment. As previously mentioned, in some embodiments, a controller may be part of a media player, and Figure 3 illustrates one such embodiment.

In the example system 300, the media player 310 includes a memory 312, one or more media output devices 314, a media playback module 316 coupled to the memory 312 and to the media output device(s) 314, and a controller 318 coupled to the media playback module 316. The memory 312 stores a media stream 322 with embedded interaction control information 324 in this embodiment, but in other embodiments the media stream could be streaming from another device such as a server, and need not be locally stored at the media player 310. The media player 310 is one example of a media player that could be implemented in a system 100, 200 in Figures 1 , 2.

The memory 312 could include one or more memory devices. Memory devices that could be used to implement the memory 312 include solid-state memory devices and memory devices with movable or even removable storage media.

The media output device(s) 314 could be implemented using a display screen for images/video, speakers for audio, and/or possibly other types of output devices to provide media playback outputs to a user.

Either or both of the media playback module 316 and the controller 318 could be implemented using hardware, firmware, components which execute software, or some combination thereof. Electronic devices that might be suitable for implementing any or all of these components include, among others, microprocessors, microcontrollers, Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), and other types of“intelligent” integrated circuits.

In operation, according to one embodiment the media playback module 316 obtains the media stream 322 from the memory 312. In other embodiments, the media stream could be obtained from another source such as a server, through a communication interface. In the example shown in Figure 3, the media stream 322 includes the interaction control information 324. The interaction control information 324 is embedded into the media stream 322. Media playback outputs (e.g., video and/or audio) are provided to the media output device(s) 314, and at least the interaction control information 324 is conveyed to the intelligent device 330.

The intelligent device 330 receives and processes the interaction control information 324, and operation of the intelligent device 330 is controlled based on the interaction control information 324. In one embodiment, the interaction control information 324 is parsed or otherwise extracted from the media stream 322 by the controller 318, and the interaction control information 324 is transmitted to the intelligent device 330. In other embodiments, the media stream 322 with embedded interaction control information 324 is transmitted to the intelligent device 330, and the intelligent device 330 detects the media stream 322 and the interaction control information 324, and processes the media stream 322 to extract the interaction control information 324. In either of these scenarios, the interaction control information 324 triggers an interaction response by the intelligent device 330. For example, if the media stream 322 reflects a sad moment, the interaction control information 324 could trigger a sadness reaction (e.g., motion and/or sound) of the intelligent device 330. Similarly, if the media stream 322 reflects a joyful moment, the interaction control information 324 could trigger a joyful reaction of the intelligent device 330.

Figure 4 is a block diagram of a system according to a further embodiment. The example system 400 in Figure 4 is substantially similar to the system 300 in Figure 3, but in the example system in Figure 4, the media stream 422 and the interaction control information 424 are distinct or separate from one another, i.e. , they may be provided in different files. For instance, this may allow the media playback module 416 to read the media stream 422 and provide media playback outputs to the media output device(s) 314 without simultaneously reading the interaction control information 424. In some embodiments, this may also provide to a user with the option to buy or otherwise acquire the media stream 422 first, and then subsequently acquire the interaction control information 424 corresponding to the media stream 422, or alternatively to acquire the interaction control information 424 first, and then subsequently acquire the media stream 422 to which the interaction control information 424 corresponds.

In some embodiments, the system 400 may facilitate the option of providing the interaction control information 424 to the controller 418. For example, while the media playback module 316 in Figure 3 obtains the media stream 322 that includes the interaction control information 324 and provides at least a part of the media stream 322 to the media output device(s) 314 and the interaction control information 324 to the controller 318, the media playback module 416 could handle just the media stream 422, and the separate interaction control information 424 could be obtained by the controller 418 from the memory 412, or another control information source in other embodiments. The controller 418 could therefore be coupled to the memory 412 as shown in Figure 4, instead of the media playback module 416 as shown in Figure 3.

Figure 5 is a block diagram of yet another example of implementation. The example system 500 is substantially similar to the example system

100 in Figure 1 , but in the example system 500, the controller is part of the server 516, such that the server 516 provides the interaction control information to the intelligent device 510, as illustrated at 524. The server 516 and the intelligent device 510 could be implemented in the system 500 in the same manner as discussed above with reference to Figure 1. In some embodiments, while the media player 514 plays the media stream 522 and may provide media-based inputs (illustrated at 526) such as signals from the media stream 522 to the intelligent device 510, the intelligent device 510 may communicate with the server 516 to obtain the interaction control information at 524. For instance, in some embodiments, at the beginning of a movie, the media stream 522 may enable the intelligent device 510 to recognize the media stream 522, and to search or otherwise contact the server 516 for interaction control information corresponding to the media stream 522. The server 516 may then transmit the interaction control information to the intelligent device 510. In this embodiment, the intelligent device 510 may no longer need to communicate with the server 516 for the whole duration of the media stream 522. In other embodiments, the intelligent device 510 may continuously or periodically stream a portion of the media stream 522 to the server 516 and the server 516 may continuously or periodically stream the interaction control information to the intelligent device 510 to synchronize interactions of the intelligent device 510 directed to a user with the media stream 522.

Figure 6 is a block diagram illustrating an example intelligent device. The example intelligent device 600 is an interactive device that is capable of interaction with a user of a media stream, and includes one or more interface(s) 602, a command generator 604 coupled to the interface 602, one or more interaction response generator(s) 606 coupled to the command generator 604, and one or more input device(s) 608 coupled to the command generator 604.

The interface(s) 602 in Figure 6 include an interface to receive interaction control information related to a media stream. An interface 602 that is to receive the interaction control information could also be configured to receive other information, including the media stream for example. In some embodiments, multiple interfaces are provided. Each interface at 602 includes a physical port, connector, antenna, and/or other physical interface to a communication medium, as well as at least a receive path or chain to enable communications via the interface. The particular form of each interface 602 is implementation-dependent, and may vary depending on the type(s) of communications to be supported. A short-range wireless communications interface such as a BLE interface is provided in some embodiments. In general, an interface 602 could be implemented using hardware, firmware, components which execute software, or some combination thereof.

The interaction response generator(s) 606 is coupled to the interface(s) 602, indirectly through the command generator 604 in the embodiment shown, and represents one or more components to generate an interaction response by the interactive device based on the interaction control information. The interaction response is separate from playback of the media stream. For example, the interaction response could include motion directed to the user, and the interaction response generator(s) 606 could include one or more motors and/or other components to generate such motion. The motion includes vibration in an embodiment. The interaction response could also or instead include sound directed to the user, and therefore the interaction response generator(s) 606 could include one or more speakers. The interaction response generator(s) 606 could also or instead include a display screen to provide an interaction response in the form of video and/or images directed to the user. More generally, an interaction response generator 606 could be implemented using hardware, firmware, components which execute software, or some combination thereof.

The interaction control information could include a command. The command could be generated by a device other than the interactive device 600, received through an interface 602, and provided to an interaction response generator

606. The other device could include a display to display the media stream, for example. In some embodiments, the other device is another type of intelligent device, such as a smartphone, a computer, or a tablet. The example intelligent device 600 includes an integrated command generator 604, coupled to the interface(s) 602, to generate a command. The command generator 604 could be implemented using hardware, firmware, components which execute software, or some combination thereof. The command generator 604 could be configured to generate the command based on the media stream. In some embodiments, the command generator 604 is configured to generate the command based on control information that is embedded into the media stream. Command generation by the command generator 604 could be based on one or both of: audio analysis of the media stream and video analysis of the media stream, for example.

The input device(s) block 608 in Fig. 6 represents one or more components to receive input information associated with the media stream. In the embodiment shown, the command generator 604 is coupled to the input device(s) 608, and could be configured to generate a command based on the input information. Other input devices, for control and/or other purposes, could also or instead be provided in other embodiments.

One example of an input device 608 is a scanner to scan a display code from a player of the media stream. The input device(s) 608 could also or instead include a scanner to scan a code from a storage medium on which the media stream is stored. In another embodiment, the input device(s) 608 could include a scanner to scan sounds from a player of the media stream. Such scanners could be used to recognize the media stream.

A processor could also or instead be provided, in an input device 608 or separately, and be at least coupled to a sound scanner and configured to recognize voices in sounds that are scanned from a media player. Such a processor could be further configured to identify context based conversations. Yet another example of an input device 608 is a scanner to scan images from a player of the media stream. A processor could be at least coupled to the scanner, or even provided as part of the scanner, and configured to recognize gestures in the images and/or to recognize faces in the images. A voice input device such as a microphone to receive natural language processing (NLP) commands could also or instead be provided as an input device 608.

In some embodiments, interaction control information or a command is embedded into the media stream, and could be parsed or otherwise extracted from the media stream by an interface 602, the command generator 604, or by an external device.

The example intelligent device 600 could be used in conjunction with any of various types of media stream, including but not limited to any one or more of: movies, tv shows, and music. Such a device could also be implemented in any of various ways, including but not limited to a toy.

Figure 7 is a block diagram illustrating an example server. The server 700 is an example of a server to convey interaction control information for animating an interactive device, and includes a content source 702, a control information source 704, a synchronizer 706 coupled to the control information source, and one or more communication interface(s) 708 coupled to the content source and to the synchronizer.

The content source 702 is coupled to the communication interface(s) 708, to provide a media stream. The content source could include, for example, a memory to store the media stream and/or an interface to receive the media stream from another device or system. The control information source 704 is configured to provide the interaction control information for animating the interactive device, and could similarly include a memory to store the interaction control information and/or an interface to receive the interaction control information from another device or system. The interaction control information and the media stream are provided together in some embodiments, and the content source 702 and the control information source 704 could be implemented together as a single source in such embodiments. In other embodiments, the interaction control information and the media stream are provided separately, by separate sources as shown. The server 700 could also include a control information generator, coupled to the control information source 704, to generate the interaction control information for the control information source. Such a control information generator could be implemented using hardware, firmware, components which execute software, or some combination thereof. The interaction control information that is provided by the control information source 704 is related to the media stream that is provided by the content source 702, and as noted above is for animating an interactive device. The animating is separate from playback of the media stream, and could include one or more of: controlling the interactive device to move, controlling the interactive device to vibrate, controlling the interactive device to emit a sound, controlling the interactive device to display an image, and controlling the interactive device to display a video.

The communication interface(s) 708 in Figure 7 include an interface to convey the interaction control information and the media stream. In some embodiments, multiple interfaces are provided. Each communication interface at 708 includes a physical port, connector, antenna, and/or other physical interface to a communication medium, as well as at least a transmit path or chain to enable communications via the interface. The particular form of each communication interface 708 is implementation-dependent, and may vary depending on the type(s) of communications to be supported. In general, a communication interface 708 could be implemented using hardware, firmware, components which execute software, or some combination thereof.

A communication interface 708 could be configured to convey the interaction control information to a media player by which the media stream is playable. In some embodiments, a communication interface 708 is configured to convey the interaction control information to an intelligent device such as a smartphone, a computer, or a tablet that is configured to relay the interaction control information to the interactive device. A communication interface 708 could instead be configured to convey the interaction control information to the interactive device rather than through an“intermediate” intelligent device.

The example server 700 also includes the synchronizer 706 to synchronize the interaction control information to the media stream. The synchronizer 706 is another component that could be implemented using hardware, firmware, components which execute software, or some combination thereof.

In an embodiment, the synchronizer 706 is configured to synchronize the interaction control information to the media stream by conveying a time sequence based on a playback of the media stream. The synchronizer 706 could also compare the time sequence based on a playback of the media stream to another time sequence that is based on the interaction control information, and synchronize the interaction control information to the media stream based on the comparison.

A server such as the example server 700 could be used in conjunction with any of various types of media streams, examples of which are disclosed elsewhere herein, and any of various types of interactive devices, such as interactive toys.

Figure 8 is a block diagram illustrating an example control device. The control device 800 includes a control information generator 802, a synchronizer 804 coupled to the control information generator, and one or more communication interface(s) 806 coupled to the synchronizer.

The control information generator 802 could be implemented using hardware, firmware, components which execute software, or some combination thereof, and configured to generate interaction control information for animating an interactive device. The interaction control information is related to a media stream, and the animating is separate from playback of the media stream. The animating could involve any one or more of: controlling the interactive device to move, controlling the interactive device to vibrate, controlling the interactive device to emit a sound, controlling the interactive device to display an image, and controlling the interactive device to display a video.

The example control device 800, and in particular the control information generator 802 in some embodiments, could be configured to provide the interaction control information separately from the media stream. The control information generator 802 could be configured to derive the interaction control information from the media stream, and in real time as the media stream is played in some embodiments.

The synchronizer 804 could be implemented using hardware, firmware, components which execute software, or some combination thereof, and configured to synchronize the interaction control information with the media stream. Synchronization could be based on one or more time sequences as disclosed elsewhere herein, for example.

The communication interface(s) 806 in Figure 8 include an interface to convey the interaction control information synchronized with the media stream. In some embodiments, multiple interfaces are provided. Each communication interface at 806 includes a physical port, connector, antenna, and/or other physical interface to a communication medium, as well as at least a transmit path or chain to enable communications via the interface. The particular form of each communication interface 806 is implementation-dependent, and may vary depending on type(s) of communications to be supported. In general, a communication interface 806 could be implemented using hardware, firmware, components which execute software, or some combination thereof.

A communication interface 806 could be configured to convey the interaction control information to the interactive device, possibly through at least one other device such as media player by which the media stream is playable. In some embodiments, a communication interface 806 is configured to embed the interaction control information into the media stream, and to convey the media stream with the embedded interaction control information. Embedding of the interaction control information into the media stream could be handled by another component that is separate from the communication interface(s) 806 in some embodiments.

A control device such as the example control device 800 could be implemented, for example, in a media player by which the media stream is playable. In another embodiment, a control device is implemented in a server with which at least one of the interactive device and a media player by which the media stream is playable is configured to communicate. A control device could also or instead be implemented in an intelligent device such as a smartphone, a computer, or a tablet. Like other components such as the example server 700, a control device such as 800 could be used in conjunction with any of various types of media streams, examples of which are disclosed elsewhere herein, and any of various types of interactive devices, such as interactive toys.

Various embodiments are disclosed above, and are not intended to limit the present invention. Other embodiments may include different and/or additional features. For example, at least some of the intelligent device 110, the media player 114, the server 116 and the controller 118, may be implemented using software and one or more processors to execute such software, to support any of various communication protocols and control schemes. In some embodiments, one or more of the intelligent device 110, the media player 114, the server 116 and the controller 118 may access and connect to internet through Wi-Fi, and also or alternatively through a broadband cellular network (e.g., 3G, 4G, 5G, LTE). One or more of the intelligent device 110, the media player 114, the server 116 and the controller 118 may also be capable of accessing and connecting to each other and/or to other components or devices such as mobile applications (mobile apps) using wireless technology such as Bluetooth Low Energy (BLE), NFC, and the like.

In accordance with some embodiments, control of one or more of an intelligent device, a media player, a server and a controller may be synchronized by Cloud-hosted multimedia content (e.g., images, video, audio). In some embodiments, mobile apps may control one or more of an intelligent device, a media player, a server and a controller, even when the intelligent device, for example, is operating in offline mode, i.e. without being connected to a server. Offline mode control could involve, for example, using finger taps, gestures, camera/motion integration, games, stories, audio/video playing, sound, and so on. In some embodiments, one or more of an intelligent device, a media player, a server and a controller may be controlled by using system-integrated (i.e., onboard) sensors, such as push buttons, motion sensors, light sensors, NFC sensors, and so on.

Control of one or more of an intelligent device, a media player, a server and a controller may also or instead involve printed materials that include barcodes, QR codes, or alphanumeric codes, which can be found on objects such as books, music packages, movie packages, tv-show packages, clothes, and so on. Images or codes could be scanned by one or more scanners in an intelligent device and used in controlling the device, for example. In some embodiments, one or more of an intelligent device, a media player, a server and a controller may be controlled using other types of accessories, to provide such features as: intelligent interaction using voice recognition, gesture recognition, face detection, natural language processing (NLP) or context based conversation; Cloud control using content stored on one or more Cloud-based devices or systems and downloadable to the user, using one or more control files created, coded, and/or stored on one or more Cloud-based devices or systems and then distributed to one or more of an intelligent device, a media player, a server and a controller, and/or using control units and drivers; and/or control using one or more high-performance artificial intelligence (Al) processors, including but not limited to Graphics Processing Unit (GPU), Field-Programmable Gate Array (FPGA), Application-Specific Integrated Circuit (ASIC), and Tensor Processing Unit (TPU).

Any of these features, and/or others disclosed herein, could be implemented in some embodiments using hardware that is configured, by executing software for example, to perform various functions. Figure 9 is a block diagram illustrating another example system implementation, with additional internal details of an example intelligent device. The example system 900 includes a server 916, a controller 918, and an intelligent device 910, coupled together as shown.

In an embodiment, the server 916 is Cloud-based and the controller 918 is a mobile device. The intelligent device 910 could then include hardware having BLE, NFC and/or Wi-Fi capacity to communicate with both Cloud and mobile devices, to support communications with the server 916 and the controller 918. The server 916 and the controller 918 could similarly include communications hardware to support communications through any of various types of connections as disclosed herein.

The intelligent device 910 includes one or more input/output devices 933, a processing unit 936, and one or more peripheral unit(s) 938. The input/output device(s) 933 could be implemented as one or more port(s), for example, and may detect manual inputs from push buttons and/or other user input devices, stimuli of the environment of the intelligent device 910, such as light signals, audio signals, etc. For example, in some embodiments, the input/output device(s) 933 may include one or more general-purpose digital input ports (e.g., push buttons, serial digital input channels, parallel digital input channels, serializer and de-serializer (SERDES) ports); application-specific analog input devices or pins (e.g., digitizers, light sensors (photoresistors, photodiodes, phototransistors, etc.)); microelectromechanical systems (MEMs) input sensors (e.g., motion sensors, accelerometers, gyroscopes, NFC sensors); and so on. Other types of input/output devices that could be provided in an intelligent device are also disclosed herein.

In response to received or detected input signal(s) or stimuli, the input/output device(s) 933 may transmit one or more signals to the processing unit 936. The processing unit 936 could include one or more processors such as a central processing unit or microcontroller to provide computing power, and could also include memory. Such memory could be implemented using one or more memory devices, and could include, for example, random access memory (e.g., one or more micro-SD memory cards) to store processor-readable data and/or instructions, read- only memory to store more permanent but still (possibly) reprogrammable data processor-readable data and/or instructions. Other types of memory devices are also disclosed herein.

The processing unit 936 could also or instead include one or more graphical processing units to provide computing power for voluminous data such as images and video, and/or one or more tensor processing units to facilitate artificial intelligence (Al) functions such as machine learning. Other components could also or instead be provided, such as DC/AC and DC/DC converters to convert current, level shifters to translate signals from one logic level to another, modulators to modulate amplitude, frequency and/or phase of signals, and/or one or more antennas in accordance with IEEE Standard 802.11 to complement Wi-Fi, BLE, and/or other protocols. The processing unit 936 processes inputs from the input/output device(s) 933 signal and delivers control signals to the peripheral unit(s) 938 through the input/output device(s) 933. To deliver output signals to the peripheral unit(s) 938, the input/output device(s) 933 may include, for example, general-purpose digital output ports to provide digital control signals and pulse-width modulated (PWM) signals (used as digital output), and/or application-specific analog output pins (e.g., programmable power supplies, arbitrary waveform generators (AWGs)). The peripheral unit(s) 938 may include lights (e.g., LEDs), speakers, motors and/or vibrators, displays, and/or other audio/video devices, and may create an interaction between the intelligent device 910 and a user. In some cases, one or more peripheral units, and similarly one or more input/output devices, may be external to but coupled to the intelligent device 910.

This is illustrative of offline control in which the intelligent device 910 is controlled based on inputs from local input/output device(s). In addition to this the intelligent device 910 may communicate with controller 918 and/or directly or indirectly with the server 916, and signals from the server 916 and/or from the controller 918 may also be processed by the processing unit 936 and used to control one or more of the peripheral unit(s) 938. Control of the intelligent device 910 could therefore be based on one or more of: local inputs to the device, signals from the server 916, and signals from the controller 918. Signals from the server 916 and/or the controller 918 could include, for example, one or more of interaction control information that is associated with a media stream and interaction control information from mobile apps. Therefore, an intelligent device such as 910 is not limited only to interaction related to a media stream. Figure 10 shows a data flow in accordance with some embodiments, and illustrates a user 1002, in addition to an intelligent device 1010, a media player 1014, a server 1016 and a controller 1018, which can be implemented as disclosed elsewhere herein. At 1020, the user 1002 selects, using a remote control, a mobile device, a push button, or a touch button for example, a video for the media player 1014 to play. At 1022, the controller 1018 scans a code (e.g., a QR code) displayed by the media player 1014. At 1024, the controller 1018 connects to the server 1016, through an internet connection for example, and the controller 1018 or the server 1016 searches into a database of the server 1016 or otherwise obtains interaction control information, in the form of a control script in the example shown in Figure 10, corresponding to the video selected at 1020. At 1026, the controller 1018 downloads the control script from the server 1016. At 1028, the user 1002 sends a request to the media player 1014 to play the selected video, again using a remote control, mobile device, push button, or touch button, for example. Video playback could be automatic in some embodiments, in response to selection of the video at 1020. At 1030, the media player 1014 notifies the server 1016 that a request to play the selected video has been received and that the selected video may be playing. At 1032, the server 1016 synchronizes, with the controller 1018, the time and status of the video being played by the media player 1014. At 1034, the controller 1018 controls the intelligent device 1010, for example by connecting to the intelligent device via BLE and by sending interaction control information. During this time, the user 1002 may watch the video being played by the media player 1014 at 1036, and the intelligent device 1010 may provide feedback to the user 1002 during play, at 1038.

When the user 1002 wants to change time of the video being played, by skipping forward or backward for example, the user may send a request to do so to the media player 1014, at 1040. Then, at 1042, the media player 1014 notifies the server 1016 that a request to change time has been received and that the selected video playback may have changed time. At 1044, the server 1016 synchronizes the playback time with the controller 1018. At 1046, the controller 1018 adjusts the control of the intelligent device 1010 to synchronize to the new playback time. When the user 1002 wants to change the status of the video being played, e.g., to stop or pause the video, the user may send a request to do so to the media player 1014, at 1048. Then, at 1050, the media player 1014 notifies the server 1016 that a request to change status has been received and that the selected video may have changed status, i.e. , stopped playing. At 1052, the server 1016 again synchronizes the playback time with the controller 1018. At 1054, the controller 1018 adjusts the control of the intelligent device 1010 to synchronize the status, and interaction by the intelligent device with the user is, like video playback, also stopped or paused. If the user 1002 then wants to restart the video, the process may return to 1028.

Embodiments are described above primarily in terms of systems and devices, and how those systems and devices are configured to operate. Other embodiments are also possible.

For example, a non-transitory processor-readable medium could be used to store a media stream that is playable by a media player for presentation to a user, and interaction control information that is configured to be unplayable by the media player. The interaction control information is directed to an interactive device that is capable of interaction with a user, and causes an interaction response by the interactive device. The interaction response is related to the media stream and separate from playback of the media stream by the media player.

The media stream and the interaction control information are stored separately in some embodiments. One of the media stream and the interaction control information is embedded within the other in other embodiments.

The media stream and the interaction control information could be generated separately, or generated together. In some embodiments, the interaction control information is derived from the media stream.

Other embodiments relate to methods. Figure 11 is a flow diagram illustrating an example method 1100, which could be performed by an intelligent device in the form of an interactive device that is capable of interaction with a user of a media stream. The example method 1100 includes receiving, at 1102, interaction control information related to a media stream. The receiving at 1102 could involve receiving the interaction control information via short-range wireless communications such as BLE.

The example method 1100 also includes, at 1104, generating an interaction response by the interactive device based on the interaction control information. The interaction response is separate from playback of the media stream, and could include, for example, one or more of the following, directed to the user: motion such as vibration, sound, video, and/or images.

The interaction control information received at 1102 could include a command. The command could be generated by the interactive device itself, or by a device other than the interactive device. The other device could include a display to display the media stream, for example. In some embodiments, the other device is another type of intelligent device, such as a smartphone, a computer, or a tablet.

A command could be generated by the command based on the media stream. In some embodiments, the command is generated based on control information that is embedded into the media stream. Command generation could be based on one or both of: audio analysis of the media stream and video analysis of the media stream, for example.

A method could also involve receiving input information associated with the media stream, and generating a command based on the input information. Receiving input information could involve receiving input information from a scanner to scan a display code from a player of the media stream. Input information could also or instead be received from a scanner to scan a code from a storage medium on which the media stream is stored. In another embodiment, the input information is received from a scanner to scan sounds from a player of the media stream. A method could then also include recognizing voices in the sounds and possibly identifying context based conversations.

Input information could also or instead be received from a scanner to scan images from a player of the media stream. A method could then also include recognizing gestures in the images and/or recognizing faces in the images.

In some embodiments, input information is received from a voice input device such as a microphone to receive natural language processing (NLP) commands.

Interaction control information or a command could be embedded into the media stream, and a method could then involve parsing or otherwise extracting or separating the interaction control information from the media stream.

The example intelligent method 1100 could be used in conjunction with any of various types of media stream, including but not limited to any one or more of: movies, tv shows, and music. Such a method could also be used with any of various types of intelligent devices, including but not limited to toys.

Figure 12 is a flow diagram illustrating another example method 1200, which could be performed at a server, for example. The example method 1200 is an example of a method to convey interaction control information for animating an interactive device. The example method 1200 involves providing interaction control information and a media stream, at 1202. This could involve reading from a memory and/or receiving from another device or system through an interface, for example. Although shown in Figure 12 as a single operation, the interaction control information and the media stream need not necessarily be provided together. In other embodiments, the interaction control information and the media stream are provided separately, by separate sources for example. Providing the interaction control information at 1202 could involve generating the interaction control information.

The interaction control information that is provided at 1202 is related to the media stream, and is for animating an interactive device. The animating is separate from playback of the media stream, and could include one or more of: controlling the interactive device to move, controlling the interactive device to vibrate, controlling the interactive device to emit a sound, controlling the interactive device to display an image, and controlling the interactive device to display a video.

The example method 1200 also includes conveying the interaction control information and the media stream at 1204. The conveying at 1204 could involve conveying the interaction control information to a media player by which the media stream is playable. In some embodiments, the conveying at 1204 involves conveying the interaction control information to an intelligent device such as a smartphone, a computer, or a tablet that is configured to relay the interaction control information to the interactive device. The conveying at 1204 could instead involve conveying the interaction control information to the interactive device rather than through an“intermediate” intelligent device.

A method could also involve synchronizing the interaction control information to the media stream. The synchronizing could include synchronizing the interaction control information to the media stream by conveying a time sequence based on a playback of the media stream. The synchronizing could also involve comparing the time sequence based on a playback of the media stream to another time sequence that is based on the interaction control information, and synchronizing the interaction control information to the media stream based on the comparison. The example method 1200 could be used in conjunction with any of various types of media streams, examples of which are disclosed elsewhere herein, and any of various types of interactive devices, such as interactive toys. Figure 13 is a flow diagram illustrating another example method 1300, which includes generating interaction control information at 1302. The interaction control information generated at 1302 is for animating an interactive device, and is related to a media stream. The animating, as in other embodiments, is separate from playback of the media stream. The animating could involve any one or more of: controlling the interactive device to move, controlling the interactive device to vibrate, controlling the interactive device to emit a sound, controlling the interactive device to display an image, and controlling the interactive device to display a video.

The interaction control information could be generated at 1302 separately from the media stream. In another embodiment, the generating at 1302 involves deriving the interaction control information from the media stream, possibly in real time as the media stream is played.

The example method 1300 also includes, at 1304, synchronizing the interaction control information with the media stream. Synchronization could be based on one or more time sequences as disclosed elsewhere herein, for example.

As shown at 1306, the example method 1300 also includes conveying the interaction control information, synchronized with the media stream. In some embodiments, multiple interfaces are provided. The interaction control information is conveyed to the interactive device, possibly through at least one other device such as a media player by which the media stream is playable.

In some embodiments, a method also involves embedding the interaction control information into the media stream. The conveying at 1306 then involves conveying the media stream with the embedded interaction control information.

A method as shown in Figure 13 could be performed, for example, by a media player by which the media stream is playable. In another embodiment, a method is performed by a server with which at least one of the interactive device and a media player by which the media stream is playable is configured to communicate. Such a method could also or instead be performed by an intelligent device such as a smartphone, a computer, or a tablet.

Like other embodiments, the example method 1300 could be used in conjunction with any of various types of media streams, examples of which are disclosed elsewhere herein, and any of various types of interactive devices, such as interactive toys.

Figure 14 is a flow chart illustrating an example method according to yet another embodiment. The example method 1400 includes, at 1402, storing a media stream that is playable by a media player for presentation to a user. The media stream is stored at 1402 on a non-transitory processor-readable medium.

On the same non-transitory processor-readable medium, interaction control information is also stored at 1404. The interaction control information is configured to be unplayable by the media player and directed to an interactive device capable of interaction with a user, and causes an interaction response by the interactive device. The interaction response is related to the media stream and is separate from playback of the media stream by the media player.

The media stream and the interaction control information are stored separately in some embodiments. One of the media stream and the interaction control information is embedded within the other in other embodiments.

The media stream and the interaction control information could be stored separately, or stored together with one of the media stream and the interaction control information embedded within the other.

The media stream and the interaction control information could be generated separately or together. In some embodiments, the interaction control information is derived from the media stream. What has been described is merely illustrative of the application of principles of embodiments of the present disclosure. Other arrangements and methods can be implemented by those skilled in the art.

For example, embodiments could be applied to any of various user scenarios.

In some embodiments, the interactive device is a film themed or tv- show themed accessory. For example, the interactive device may be a figure moving during movies in cinema and motions of the figure may reflect the movie scenes. As another option, the interactive device may be a themed toy synchronized with television content to interact with users watching the television: in this example, the media player is the television. The interactive device in these applications may emphasize emotions of the film or tv-show being played to entertain the user watching the film or tv-show and/or to emotionally educate the user.

In some embodiments, the interactive device may be advertisement toys distributed in public facilities such as supermarkets, restaurants or shopping malls, at promotional events, or during festival celebrations. As the user and/or the advertisement toy are exposed to certain ads content or along with specific events, such as shows, fireworks, light-shows, and so on, the advertisement toy can react to these to promote products, events and/or services. In some embodiments, the intelligent device is driven by Al. In such embodiments, software that is executed by hardware of the intelligent device may include an Al layer. The Al layer may allow the intelligent device to be programmable by the user, i.e. , the user can add, modify or remove intelligent device routines (i.e. , functions and reactions). Routines may be stored in one or more files and shared with other users on an exchange platform for example, to allow the other users to similarly add, modify or remove routines on their own interactive devices. In some embodiments, the intelligent device may be a game doll or a figure from a videogame, a figure from a movie or a television show, or a figure of a celebrity for example, and the Al layer may allow the interactive device to interact with videogames, movies, television shows, media contents, etc., in a manner similar to other embodiments disclosed herein. Furthermore, the Al layer may include a random-generator algorithm influencing the interactions of the interactive device, such that the interactive device may be less predictable, and hence more realistic. For example, in embodiments where the interactive device is a toy reacting to a movie, when a sequence of a movie that usually provokes an interaction from the toy is watched multiple times, the toy may not react identically every time, and sometimes the toy may not react at all. In some embodiments, the interactive device may cooperate synchronously with other interactive devices, i.e. , the interactive devices may interact with multiple different users at the same time and the interactive devices may interact with other interactive devices. The interactive devices may react similarly and simultaneously to some external events, and may also or instead react differently to some external events. This may allow the interactive devices to emphasize the same response in some scenarios, while providing different responses in other scenarios (for example when the interactive devices depict different characters of a movie). In embodiments in which the intelligent devices are toys, this may allow users to play together to exchange toys, to share routines between toys, and/or to provide for alternate usage of toys, for example by using them as walkie-talkies.

Media streams could be streamed to a media player, or could instead originate from another source such as a local memory in an intelligent device or a storage device such as a DVD or Blu-ray disk. Another possible option for providing media streams and control information to one or more intelligent devices would be through a USB or other connection to a computer or a media player, so that the device, or multiple devices, could be controlled to react to music, games, videos, and/or apps on the computer or media player.