BACKGROUND OF THE INVENTION Field of the Invention The present invention is directed to a system which allows at least two subjects to display to each other their response to certain stimuli, such as engagement data, during a communication between the at least two subjects.

Discussion of the Background The inventors of the present application have been involved in developing systems which monitor how a person reacts to certain stimuli, such as an audio presentation, a video presentation, etc. The different ways of monitoring how a person reacts to such stimuli may be by taking an EKG of the patient, taking an EEG of the patient, taking heart rate data of the patient, etc. As one concrete example, it is often desirable to take EEG measurements of a person to monitor the brain activity of that person.

Specifically, it may be desirable to monitor a person's brain activity to measure that person's reaction or interest to certain stimuli, i. e. to monitor that person's engagement level.

At this point, the inventors of the present application have focused on monitoring different test subjects response to stimuli and recording those responses in a manner easily correlated with the stimuli. At this time there are no known systems which allow a person's response to stimuli to be displayed and utilized in an interactive environment in which different persons are communicating with each other.

SUMMARY OF THE INVENTION The inventors of the present invention have recognized that it may be desirable during an interactive communication between plural persons to provide a stimuli response measurement, such as EEG data, of each person to the other persons in the communication.

For example, in a communication between first and second persons, the stimuli response of the first person, for example EEG data indicating a level of engagement of the first person, is monitored and provided to the second person. Simultaneously, the stimuli response of the second person, for example EEG data indicating a level of engagement of the second person, is also monitored and is provided to the first person. In such a system the first person and the second person can communicate with each other and can see the other person's engagement level during the communication.

With such a system in the present invention, both the first and second persons will have a better sense of how engaged the other person is during the communication. Further, both the first and second persons will also have a better sense as to the stimuli response of the other person generated by certain stimuli.

In one embodiment, the present invention is directed to a situation in which the first and second persons are in the same room and are simply talking to each other, showing documents to each other, etc. In that situation each of the first and second persons will know when the other person is most engaged in the conversation.

The communication between the at least first and second persons can, however, take many other forms, such as a simple communication across a conventional telephone line, i. e. a public service telephone network (PSTN), typed communication passed through the world wide web (WWW) such as in a chat room, a local area network (LAN), a wide area network (WAN), etc. Also, in the examples noted above a communication is described between first and second persons. However, the communication which can take place in the present invention is not limited to just two persons, but instead may involve multiple persons.

OBJECTS OF THE INVENTION Therefore, it is an object of the present invention to provide a novel system which allows for the exchange of response stimuli data, e. g. engagement data, between different persons involved in a communication.

It is another object of the present invention to provide a novel system which easily allows for such an exchange of stimuli data between the different persons, and such that the communication between the different persons can take many different forms such as a direct communication when the different persons are in a same room, an indirect communication when the persons are at remote locations and are communicating by a telephone line, by the Internet in a chat room, by a local area network (LAN), by a wide area network (WAN), etc.

BRIEF DESCRIPTION OF THE DRAWINGS A more complete appreciation of the present invention and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein: Figure 1 shows a first embodiment of the present invention in which two subjects are directly communicating; Figure 2 shows a second embodiment of the present invention in which two subjects have an indirect communication ; Figure 3 shows a third embodiment of the present invention in which two subjects also have an indirect communication; and Figure 4 shows the form of a display screen in the embodiment of Figure 3.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Figure 1 illustrates components of a preferred embodiment of the system of the present invention in which two subjects, subject A and subject B, are communicating directly with each other. That is, in the situation shown in Figure I the two subjects A and B may be in the same room and may be simply talking to each other, showing each other various documents, displays, etc.

In the system shown in Figure 1 each subject A, B is monitored by a respective monitor 10,20 which measures the respective subjects response to certain stimuli. In one form that response may be EEG data which provides an indication of the brain activity, i. e. an engagement level, for the respective subject. However, the monitored response stimuli may include instead of EEG data or in addition thereto EKG data, heart rate data, galvonic skin response, blood temperature, body temperature etc. The Applicants of the present invention have also devised a novel way of easily and efficiently monitoring a test subject to detect EEG data, which is disclosed in copending U. S. Patent Application Serial No. 09/187,525, the entire contents of which are hereby incorporated herein by reference. The present invention is not limited to such a system of monitoring a subject to obtain EEG data, and the present invention can be utilized with any system which monitors a subject to obtain such data or similarly desired data.

The outputs from the monitors 10,20 are provided to respective data acquisition analog to digital converters 12,22. The data acquisition analog to digital converters 12,22 take the raw response stimuli data from the monitors 10,20, in the example noted above that would be raw analog EEG data, and convert such raw EEG data to digital data. The data acquisition analog to digital converters 12,22 typically also perform processings of amplification, filtering, etc., as needed on the raw analog EEG data provided from the monitors 10 and 20, and then convert the raw analog EEG data into a digital form that can be used later to provide a graphical representation of the subjects A and B engagement level.

Both of the above-noted functions of the data acquisition analog to digital converters 12,22 can be implemented by discrete circuits, which can be housed in the same"black box". The data acquisition analog to digital converters 12,22 can be structured as stand alone"black box"units, as shown in Figure 1, or could be formed in the same unit as the respective monitors 10,20, and for example if monitors 10,20 are headsets.

The digital EEG data output from the data acquisition analog to digital converters 12 and 22 is then provided to respective stimuli response processors 13 and 23. The stimuli response processors 13,23 take the digital EEG data output from the data acquisition analog to digital converters 12,22 and convert that digital data into a graphical representation which can be displayed on a monitor, such as a cathode ray tube (CRT) monitor, LCD monitor, etc.

Such a graphical representation of the test subjects A and B responses to stimuli can take various forms such as an LED displays, a needle on a meter, an audible indication, etc., but a preferred form may be a graph showing peaks (higher brain activity levels) and valleys (lower brain activity levels) that represent the test subjects A and B interest to the stimuli representing specific brain activity, i. e. which represent the test subjects A and B level of engagement, as a function of time.

Each stimuli response processor 13,23 receives the digital EEG data from the respective data acquisition analog to digital converters 12,22 and converts the digital EEG data into a graphical representation to be sent to respective feedback monitors 11,21.

Outputs of stimuli response generators can also be provided to VGA scan converters (not shown) if the output from the stimuli response processors 13,23 are to be displayed on a monitor which cannot display a VGA signal, or if the outputs of the stimuli response processors 13,23 are to be recorded onto an analog video tape through a video cassette recorder (VCR) or similar device which cannot handle a VGA signal. Such VGA scan converters take an existing VGA signal from one form to another, such as NTSC, PAL, RGB, etc. VGA is only one example of a form that the signals can be converted to, and other forms such as SVGA, 8514/A graphics standard, XGA, etc. are possible. It should also be noted that a VGA scan converter can be formed as part of the stimuli response processors 13,23.

Further, each stimuli response processor 13,23 may simply be a personal computer which performs the processing noted above. That is, each stimuli response processor 13,23, can take several different forms. Each stimuli response processor 13,23 may be its own stand-alone device specifically dedicated to performing the operations discussed above and below. As an alternative, each stimuli response processor 13,23 can be built into some other device, such as being a part of a personal computer with specific software and hardware to perform the functions discussed above and below.

It should also be noted that in the instance that the monitors 10,20 monitor additional data to EEG data, such as EKG data, heart rate data, etc., each stimuli response processor 13, 23 can take that additional data and process it accordingly, by either providing an additional graphical representation for that additional data or synthesizing that additional data with the EEG data to provide one graphical representation signal indicating a level of engagement.

The stimuli response processor 13 shown in Figure 1 receives engagement data of the subject A and provides such to a feedback monitor 21 associated with the subject B.

Similarly, as shown in Figure 1, the stimuli response processor 23 receives engagement data associated with the subject B and provides such to a feedback monitor 11 associated with the subject A. That is, as shown in Figure 1, the output of the stimuli response processor 13, which indicates a level of engagement of the subject A, is provided to the feedback monitor 21 which can be viewed by the subject B. Similarly, the output of the stimuli response processor 23, which indicates a level of engagement of the subject B, is provided to the feedback monitor 11 which can be viewed by the subject A.

With such a system as in the present invention, the subject A has his or her level of engagement monitored, for example by a measurement of EEG data, that level of engagement is converted into a graphical representation, and that graphical representation is provided to a monitor 20 for display to subject B. Similarly, and simultaneously, the subject B has his or her engagement monitored, converted into a graphical representation, and then provided to a monitor 10 for display to subject A.

With such a system as in the present invention, the subject A will know, as he or she communicates with subject B, the level of engagement of subject B, i. e. when the subject B is more engaged and less engaged in their communication. Similarly, the subject B will know, as he or she communicates with subject A, the level of engagement of subject A, i. e. when subject A is more engaged and less engaged in their communication.

It should also be noted that the data acquisition analog to digital converters 12,22 and the stimuli response processors 13 and 23 operate in real-time in the embodiment noted above, so that the feedback monitors 11, 21 display the engagement level of the subjects A and B in real-time. Thus, with the system of the present invention both of the subjects A and B will know the level of engagement of the other subject in real-time.

The operation of the system of the present invention as shown in Figure 1 is specifically directed to an embodiment in which the subjects A and B directly communicate with each other. The example noted above with respect to Figure 1 is that the subjects A and B are in the same room as each other. However, it should be evident that in the situation shown in Figure 1 the subjects A and B need not be in the same room with each other but could be in different rooms and the direct communication could be via an intercom, walkie talkie, phone, etc. Such embodiments could also still involve a direct connection from the stimuli response processor 13 to the feedback monitor 21 for the subject B, and a direct connection between the stimuli response processor 23 and the feedback monitor 11 for the subject A.

However, the present invention is also operable in a system as a further embodiment in which the subjects A and B are remotely located from each other and in which there is no direct hard-wire connection between the stimuli response processor 13 and the feedback monitor 21 and between the stimuli response processor 23 and the feedback monitor 11. As an example, the subjects A and B may be communicating to each other via the telephone, video conference, or again via an intercom system, which are termed"indirect communications"in the terms of the present invention. Such a further embodiment is shown in Figure 2. As shown in Figure 2, instead of having stimuli response processors 313 and 323 connected to the respective feedback monitors 21 and 11, the stimuli response processors 313 and 323 are connected to each other by a communication link 310 and the stimuli response processor 313 is connected to the feedback monitor 11 and the stimuli response processor 323 is connected to the feedback monitor 21.

In such an embodiment of the present invention, the stimuli response processors 313 and 323 may simply be incorporated into personal computers of the respective subjects A and B and the feedback monitors 11 and 21 may simply be the monitors for the respective personal computers.

In Figure 2, the stimuli response processor 313, which again may be essentially a personal computer of the subject A, receives data from the stimuli response processor 323 through the communication link 310, which data is the engagement data, e. g. EEG data, of the subject B generated by the stimuli response processor 323. The stimuli response processor 313 can then take that engagement data provided from the stimuli response processor 323 and processes such, and then provide the processed engagement data of subject B to the feedback monitor 11. In such a way, the subject A can view on the feedback monitor 11 the engagement data of the subject B.

Similarly, and as shown in Figure 2, the stimuli response processor 323 receives the engagement data, e. g. EEG data, output from the stimuli response processor 313 indicating the engagement level of subject A and processes the same, and then provides the processed engagement data of subject A to the feedback monitor 21. In such a way, the subject B can view on the feedback monitor 21 the engagement data of the subject A.

In the system as shown in Figure 2 there must be a communication link 310 established between the two stimuli response processors 313,323. That communication link 310 can take many forms such as a serial cable, a parallel connection, a phone line, cable lines, or networks such as a local area network (LAN), a wide area network (WAN), or the communication link 310 can be the Internet, etc. In the embodiment of Figure 2 the communication between the subject A and the subject B is on a different communication channel than the communication link 310, and can be a telephone line, cellular connection, intercom system, video conference, etc.

However, a further embodiment of the present invention as shown in Figure 3 is a similar system as shown in Figure 2 except that there is no distinction between the indirect communication link between the subject A and the subject B and the connection between the stimuli response processors 313 and 323. That is, in a further embodiment of the present invention all of the communication between the subject A and the subject B is performed via their respective computers which include, respectively, the stimuli response processor 313 and feedback monitor 11 for subject A and the stimuli response processor 323 and the feedback monitor 21 for subject B. The embodiment of Figure 3 is an embodiment which would exist when the subject A and subject B are communicating via the Internet, such as Internet telephone or in a chat room, etc.

In the embodiment of Figure 3, the indirect communication is indicated as a communication link 510 which may typically be the Internet. In the embodiment of Figure 3 the respective stimuli response processors 313 and 323 have the same structure as in the embodiment of Figure 2. However, in the embodiment of Figure 3 the stimuli response processors 313 and 323 must perform an additional synchronization function and provide additional data for display.

Communication over the Internet is typically executed such that data is sent as "packets", i. e. pieces of a message transmitted over a packet-switching network in which each packet contains a destination address in addition to substantive data. With communication over the Internet a communication protocol such as a TCP/IP (transmission control protocol/Internet protocol), which is a standard suite of communications protocol utilized to connect hosts on the Internet, handles such data"packets"and insures that such data "packets"are sent to the appropriate destination. Data can also be sent by"data streaming", which is a technique for transferring data such that the data is processed as a steady and continuous stream. Streaming technologies are becoming more significant with the growth of the Internet because most users do not have a fast enough access to download large multimedia files quickly. With"data streaming"a client browser or plug-in can start displaying data before an entire file has been transmitted. However, for"data streaming"to work, a receiving side must be able to collect data and send it as a steady stream to the application that is processing the data and converting the data to sound or pictures. That means that if a streaming receiving side receives data more quickly than required, it needs to save the excess data in a buffer. If the data doesn't come quickly enough, however, the presentation of the data may not be smooth.

In the applications to the present invention, when data is sent from each stimuli response processor over the Internet, time codes may be utilized with the stimuli and the engagement data to synchronize the stimuli stream with the engagement data stream. Then, appropriate software can read such time codes and periodically synchronize the two data streams so that there is a time base synchronization of the stimuli with the engagement data.

In such a way, a subject receiving the engagement data can also see the stimuli that corresponds to the received engagement data.

More particularly, with the embodiment of Figure 3 the stimuli response processors 313 and 323 can provide the display data, i. e. the graphical representation of the engagement data, to the respective feedback monitors 11 and 21 either in real-time or on the command from the other stimuli response processor, for example based on the click of a mouse in the other stimuli response processor, selecting an"enter"input on a keyboard in the other stimuli response processor, etc. That is, such a communication between the subjects A and B may not be in real-time, and thus in the embodiment of Figure 3 the graphical representation provided on the feedback monitors 11 and 21 preferably also includes the stimuli to which the respective subjects A and B are responding.

As a concrete example, assume that the subjects A and B are communicating in a chat room at a world wide web (WWW) site. Assume that subject A is typing into his computer, which includes the stimuli response processor 313 and the feedback monitor 11, a message to be sent to subject B. In that instance the display on the feedback monitor 11 viewed by subject A can be split into three separate windows, as shown in Figure 4. A first window 60 shows the message that the subject A is typing and is essentially the window provided from the chat room, i. e. window 60 is the display that any user in the chat room would see if there was no further processings of the present invention. A second window 61 is a display indicating the window 60 from the subject B, i. e. window 61 is the screen on subject B's computer. A third window 62 on the feedback monitor 11 indicates the engagement data of the subject B, i. e. the graphical representation of engagement of the subject B as provided from the stimuli response processor 323 of subject B. Such a display is preferable in such an embodiment of the present invention since the communication is not in real-time. In that instance the subject A not only has to receive the engagement data of the subject B, but also has to see to what that engagement data corresponds.

The feedback monitor 21 for the subject B is similarly constructed.

In such ways, in the system of the present invention of Figure 3 the stimuli response processors 313 and 323 perform the additional functions of providing a windowed display as shown in Figure 4 to the respective feedback monitors 11 and 21.

With such systems of the present invention as described above, communication between two different subjects A and B can be enhanced by providing each subject with engagement data of the other subject. The system of the present invention can also operate whether that communication between the two subjects A and B is in real-time or is not in real-time.

Moreover, the system of the present invention as described above has only taken as an example a situation in which two subjects A and B are communicating with each other.

However, the system of the present invention is clearly applicable to systems in which more than two subjects are communicating with each other. In that instance the respective feedback monitors preferably have displays which are divided into windows for each of the different subjects. For example, if three subjects are involved in the communication the feedback monitor for each subject would have its display windowed into two portions, with one portion for each of the other two members of the communication. In such a way, the system of the present invention can be expanded to any number of subjects communicating with one another.

The inventors of the present invention have also disclosed in related U. S. Patent Application Serial Nos. 09/343,524 and 09/357,959 systems in which engagement data can be recorded simultaneously with the stimuli based on which the engagement data is generated. If desired, the systems of the present invention as disclosed in Figures 1-6 above could also be modified to include an additional recording operations by employing VGA converters as noted above, and as disclosed in Applicants'above-noted incorporated patent applications to provide the additional feature of allowing the subjects to make such recordings.

This invention may be conveniently implemented using a conventional general purpose digital computer or microprocessor programmed according to the teachings of the present specification, as will be apparent to those skilled in the computer art. Appropriate software coding, such as in the computer languages C++ or Visual Basic, can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those skilled in the software art. The invention may also be implemented by the preparation of application specific integrated circuits or by interconnecting an appropriate network of conventional component circuits, as will be readily apparent to those skilled in the art.

The present invention includes a computer program product which can be contained on a storage medium including instructions which can be used to program a computer to perform a process of the invention. The storage medium can include, but is not limited to, any type of disk including floppy disks, optical discs, CD-ROMS, and magneto-optical disks, ROMs. RAMs, EPROMS, EEPROMS, magnetic or optical cards, or any type of media suitable for storing electronic instructions.

Obviously, numerous additional modifications and variations of the present invention are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the present invention may be practiced otherwise than as specifically described herein.