Cues of users

FIELD OF THE INVENTION

The present invention relates to a method and a system for changing states of a computer application based on the captured emotions and physical reaction of the user. The computer application could be any generic application, a computer game, a mobile application, or an online event. The states of the computer application could be various modes of the application, or in a computer game it could represent a changing animation state of a character of the game.

BACKGROUND

Emotion has been studied from centuries as an important human behavior for conveying mental and emotional information. The most expressive tool for human is face, which also being a strong communication tool ever. The language of face is more fundamental than any other emotion or gesture, it is a universal form of conveying your sentiments and messages to others.

Currently at least half a billion people around the world are routinely socializing through avatar over the Internet, for example in online games. Others spend many hours each week controlling avatars inside their own homes, using game platforms like the WII and Xbox.The use of avatars in online community is increasing at a rapid rate such as in social media and in videoconferencing system. In order to further advance the technology, the present invention provides a method for extending the concept of avatars by associating the emotional footprints of the user to specific states of a given application. The application of the present invention captures the user behavior, gestures, emotions and other sensory input that include facial and gesture recognition while interacting with any computer application which could be an online game, or a social media interaction, a mobile application, or a generic computer program being used by an individual. The captured physical reaction data of the individual user is processed and used to change the state of the given application.

In one embodiment of the invention this could mean changing the state of an animated character based on the non verbal cues of the individual user.

SUMMARY OF THE INVENTION

Interactivity driven by human inputs on a mobile screen increases engagement of users. The present invention provides a family of Applications (Apps) that are based on the core Non- Verbal Cues (NVC) capture and analysis technology. These Apps are software programs that run on any computing device with appropriate computing power and peripherals such as displays, sensors like webcam, audio capture, accelerometer, GPS etc. The Apps concentrate on creating engagement with the initial user, and in the process collecting some vital data about the interaction that would benefit the overall implementation. The Apps use the data collected in this phase to better correlate contextual reactions of the users in giving informed analytics.

In accordance with the above observation an Application module that drives the states of the running application into other mode based on real time Non- Verbal Cues of the users collected during the interaction is provided. The Non- Verbal Cues may consist of but not limited to any combination of Webcam input, gesture recognition, accelerometer inputs, GPS, simple facial feature tracking like

Head tracking, eye tracking, nose, lips, smile tracking etc. The states of the running application may be a few known animations sets, a selected set of Avatars, commands in the application, suggestion or recommendation, versions, display etc. Furthermore, the running application may include any rich media (audio, video, web based content) input, or even real time interactivity like text messaging, audio/video conferencing, or on-line gaming.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is an illustration of a user' s non-verbal cues driven animation state on the screen of a mobile device in accordance with an embodiment of the present invention.

Figure 2 illustrates an exemplary embodiment of the invention illustrating the change in states of the application showing a string of video.

Figure 3 illustrates an exemplary embodiment of the invention illustrating the non-verbal cues driving state of the mobile game application.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides an application module that collects the non verbal cues or emotional reaction data of the user and analyzes the non verbal cues to drive the state of running application on the client device. The application module is installed on a client device having computational powers and peripherals such as displays, sensors like webcam, audio capture, accelerometer, GPS etc.

In an embodiment of the present invention the system for capturing emotional reaction data in form of non verbal cues comprises of a client sub-system as well as a server sub-system that enable detection of user' s non verbal cues as a reaction of viewing a shared content, interaction in an online event, an online gaming, a web-conferencing, a mobile application, a generic computer application etc. The client device includes a module to capture the non verbal cues in form of input data, and a module to process the input data to do feature extraction, and an application module to drive the states of running application based on the input data. The module to capture the input data consists of a sensor to capture the user's input. The Sensor operates on different auditory, visual, text, location based, or other kinds of sensory inputs. The module to process the input data consists of a processor that processes the input received from the sensor to analyze the emotional reaction of the user. The client device can be an iphone, laptop, a computer, a mobile phone, a Smartphone, a laptop, a camera with Wi-Fi connectivity, a desktop, a tablets (iPAD or iPAD like devices).

In an embodiment of the present invention the sensor captures the input from a user in the form of auditory, visual, text, location based, or any other kind of sensory signal.

The client device has the ability to capture various kinds of auditory, visual, location based, text based, and other kinds of sensory inputs that are used to detect the instantaneous emotional reaction of a user. The client device then processes the sensory inputs and derives an instantaneous Emotional reaction of the user interacting with the application running on the client device. The processing module of the client device represents the emotional reaction of the user into a few set of emotional states in a real time manner. The application module analyzes the change in emotional states of the user which is then used by the application module to drive the state of running application by changing the mode of the running application from one state to another. The client device further has a mechanism to communicate the change in state of the running application within the network of a plurality of client devices connected through a server.

The user' s input data captured as non verbal cues consists of a combination of the emotion, behavior, response, attention span, gestures, hand and head movement, or other reactions or stimuli of the user collected through the sensors available in the client devices.

In an embodiment of the present invention the set of emotional states include Sad, happy, Angry,

Neutral, Fearful, Disgusted, Surprised and other known human behaviors.

In an embodiment of the present invention the user access the application using a client device. The application is provided with enhanced security feature such as with login details and password enabled login.

The user while interacting with the application generates different emotional reactions to the online content, event, conference or online gaming, depending on the type of the event. The client device captures the non verbal cues of the user in real time manner by the sensors that operates on the different auditory, visual, location based, or other kinds of sensory inputs. The captured non verbal cues of the users are then processed to analyze the reaction of the user in real time manner. For each time period, the user's reaction or non verbal cues are derived into a plurality of emotional states such as happy, sad, angry, disgusted, neutral, surprised, fearful or other known human behavior.

Based on the emotional state of the user in a particular time interval, the application module changes the states of the running application. The captured non verbal cues are processed and can be used to drive the animation states of a mobile display.

In an embodiment of the present invention, the application module that is used to drive or change the states of the running application can be present in the client device or on the server.

In an embodiment of the present invention the application could be a mobile Application or a game where the user is provided with an option to select from a set of animated characters, for instance 'avatars' . The avatar is displayed on the mobile device and reflects one of the animated states of the avatar. The non-verbal cues of the user are captured by continuously tracking the user' s eye using a webcam, while the user is interacting with the application running on the mobile device. The user's eye movement are then processed and analyzed by the processing module and depending on the changes in the position, shape and movement of the eye of the user; the animation state of the avatar is changed by the application module.

In alternate cases where the camera is not used, the pitch or intensity of the voice is being tracked, and the processing module processes the voice of the user and analyzes the captured voice in real time manner. The real time change in the frequency and pitch of the voice of the user is plotted against the time of interaction. The change in the frequency and pitch of the voice is then used by the application module to change the state of the application running on a client device.

Fig. 1 is an illustration of a user' s non-verbal cues driven animation state on the screen of a mobile device in accordance with an embodiments of the present invention. The user's non verbal cues are captured by a camera and are then feed in to the processing module in form of images 102(a), 102(b), and 102(c) at different time interval. The application module monitors the eye movement of the user and processes the images 102(a), 102(b), and 102(c). Depending on the changes in eye movement of user in 102(a), 102(b), and 102(c), the corresponding animation character also changes their eye movement as depicted in 104(a), 104(b), and 104 (c).

In an embodiment of the present invention, the state transition of the application running on a client device and the captured non verbal cues are tagged granularly to the running application and is then transferred to a server for storage and communication. Alternatively if the client device is not connected to a server, then the data may be temporarily cached locally and next time connectivity happens during the Application functioning, the data is then transferred to the server. This data collection may be the collection of feature statistics, and tagging with the context. In other embodiments the application's states may be driven by the past collected data about users as well. In other embodiment the data may be shared in the user's network through social networking platform, for instance Facebook, Linkedln, or Twitter.

In further embodiments of the present invention, the change in states of Application may be reflected by showing a sequence of videos instead of or in addition to the animation states. One example is a string of YouTube videos that may be trailers of upcoming movies. The Application will keep track of which video was shown and the user reaction will be captured; towards end of each video segment, the user will be prompted to give the non-verbal cue to go to next video; the video that will be shown depends on the non-verbal cue captured and is based on a table of data referencing the videos, for example videos tagged "sad" may be linked to a particular non-verbal cue and displayed upon input of that cue. The whole user interaction of what video was shown, in what sequence, as well as the user' s reaction is captured, tagged, and sent to the server in a similar fashion as described above. Animation states may also be replaced by or augmented with any rich media (audio, video, web based content) input, or even real time interactivity like text messaging, audio/video conferencing, or on-line gaming. Different methods of monetization may be attached to various versions of the Application linked to both the analytics gathered about the user's non-verbal cue states and the analytics about the animation States (that may be include video, audio and other rich media), as well as standard click based advertising (e.g. AdMob, iAD etc.).

Figure 2 illustrates an exemplary embodiment of the invention illustrating the change in states of the application showing a string of video. The interface 202 shows a string of you-tube videos, when the user selects a video to be played, the non-verbal cues of users are captured through the sensor as shown in 204. The non verbal cues are processed and based on the emotional state of the user, the display gets changed and shows the recommended videos 206.

Figure 3 illustrates an exemplary embodiment of the invention illustrating the non-verbal cues driving state of the mobile game application. Figure 3 shows a mobile gaming application wherein the user selects the animated character 302. The gaming application asks the user to react in one of the known human emotions. The user's non-verbal cues are captured by the sensor present in the mobile device and are then processed to analyze the emotional reaction 304. Based on the non verbal cues of the user, the display of the state machine changes. The state transition along with the non verbal cue is tagged granularly to the gaming application and is shared in the server as shown in 306. In a further embodiment of the present invention, the Application can be used to add feelings to a picture using non verbal cues of the user. The user can select a picture form the gallery and the application then asks the user to add a feeling to the picture. The user then provides his input through non verbal cues. Based on the non verbal cues of the user, the application tagged the picture with the corresponding cue. The tagged picture can then be shared in the user' s network.