Erika Torres

Jayesh Sahasi

Priority Claims/Related Applications

This international application claims priority under 35 USC 120 to U.S. Patent Application Serial No. 15/726,018, filed October 5, 2017 and entitled "ATTENDEE

ENGAGEMENT DETERMINING SYSTEM AND METHOD", the entirety of which is incorporated herein by reference.

Field

The disclosure relates to live web-based content presentations that are intended to promote a product or an idea to a specific audience and in particular to a system and method for determining audience engagement in the live web-based content presentations.

Background

In most live web-based content presentations, a presenter invites possible

customers/leads (collectively, the "attendees") to watch online content. The presenter would like to have more information about the attendees prior to the broadcast so the presenter can devote more attention to attendees that are going to be more engaged with the content. The presenter also can track if the content is going to have a good audience or if he needs to improve the promotion. The problem is how to predict the engagement level of the registrants/attendees before the broadcast.

Engagement prediction for media content is a widely researched topic. However, there are thousands of alleged solutions of this problem. One of the most popular approaches gathers information about the amount of clicks and the time spent in the page of each attendee and uses that information to predict if the user is going to be engaged with a certain topic or topics. However, a lot of the online events offer registration by a third-party which means some attendees will visit the web page only the day of the broadcast. Therefore, this approach is not ideal for the problem in hand.

Another approach is based on social network trends. In this approach, a system monitors social network interactions and topics to predict if certain content is going to attract more attention. This approach however does not work well for sharing online content that requires previous registration since a user that registered several months before the broadcast is unlikely to provide useful input to this approach. Furthermore, a big portion of the web content is private and cannot be shared publicly because it is exclusively of professional interest. Therefore this solution is not applicable for all the online contents in this context.

Thus, there is still a technical problem of being able to determine attendees engagement with content for live web-based content presentations and it is to this end that the disclosure is directed.

Brief Description of the Drawings

Figure 1 illustrates a high level overview of an engagement prediction method;

Figure 2 illustrates a high level implementation of the engagement prediction method;

Figure 3 illustrates an example of an implementation of a live, web-based presentation system that may incorporate an engagement prediction system that may perform the engagement prediction method;

Figure 4 illustrates more details of the front end and backend of the engagement prediction system;

Figure 5 illustrates more details of the feature extraction process;

Figure 6 illustrates more details of the classification process; and

Figure 7 illustrates an example of a user profile of an attendee/registrant that includes the engagement prediction.

Detailed Description of One or More Embodiments

The disclosure is particularly applicable to an engagement determination system and method integrated into a web-based live presentation system and it is in this context that the disclosure will be described. It will be appreciated, however, that the system and method has greater utility since it may be used with or integrated into other content systems in which it is advantageous to be able to determine attendee engagement before the presentation and the system and method may be a stand-alone system that determines the attendee engagement for a plurality of third party content systems.

Figure 1 illustrates a high level overview of an engagement prediction method 10. The method may be carried out by various systems, both hardware or software, such as the system shown in Figure 3. The method may also be carried out by a specialized system, such as web- based live presentation system. The method 10 may include the ordered combination of processes including data extraction 12, data preprocessing 14, feature extraction 16, classification 18 (using decision tree and deep neural networks in one embodiment) and saving of the results 20 in a storage, such as a database. The classification process may also be known as a process for generating a model for the prediction of the engagement of the attendees as described below in more detail. In the method, the information that may be used an input to the engagement prediction may include:

• Engagement level from each or all registrants/attendees in the past. · Engagement source, for example if the attendee watched the online event live or on demand.

• Nature of the online events watched in the past by industry and sector.

• Type of online content watched in the past.

• Registration method of each or all attendees.

· Date and time when each registrant/attendee signed up.

• Broadcast date of the live presentation for which engagement is going to be predicted.

After all the information above may be gathered (data extraction process 12) and processed (process 14) and transformed into features (process 16) to feed the model (the classification process 18). A feature vector is calculated for each attendee. The model may consist of a Decision Tree classifier and Deep Neural Network combined in a voting scheme in one embodiment. After training the classifier, it determines if the attendee is going to have a HIGH or LOW engagement level which may be saved in the storage during the storing process 20.

In this disclosure, engagement refers to a measurement of time the event is viewed, polls answered, downloaded resources and the maximum value possible is 10 and the minimum is 0. For example, if the user did not see all the video, the engagement would be Low - less than 2.0, if he completed all the questions, saw all the video and downloaded all the resources then his score will be High - greater than 4.5.

The engagement prediction may be used before and during the presentation/event. For example, during the presentation/event the engagement prediction may be used to:

• Get different, more detailed communications before and during the event

• Get different (higher) prioritization of their questions

• Have different chat rooms and other mechanisms available to communicate with the presenters, or with other highly engaged users

Figure 2 illustrates a high level implementation of the engagement prediction method 20. The method may be carried out by various systems, both hardware or software, such as the system shown in Figure 3. The method may also be carried out by a specialized system, such as web-based live presentation system. A real implementation of the method may consists of three ordered stages including user registration in the web page 22, a feature extraction via API 24 and a classification 26 resulting in a high or low engagement level determination for each attendee that is determined before the live presentation has occurred.

These methods address a technical problem of being able to assess attendee engagement for a presentation before the presentation has occurred. Thus, existing systems that measure engagement after the presentation has occurred or that measure that engagement via human beings cannot achieve the solution of the disclosed system and method.

Furthermore, the system and method provide a technical solution to the problem using technical features (including the data preprocessing, the feature extraction and the classification in combination) to achieve the high or low engagement prediction for each attendee before the presentation occurs. The disclosed system and method is also not just generic computers, but are specially configured and programmed systems that achieve the technical solution. The disclosed system and method also has a set of models, like a set of rules to perform facial animation, that are executed by a computer system that provide that technical solution.

Figure 3 illustrates an example of an implementation of a live, web-based presentation system 30 that may incorporate an engagement prediction system that may perform the engagement prediction method. The system 30 may have frontend of the system 32, such as one or more computing devices 32A, 32B,..., 32N in the example in Figure 1, that may connect over a communication path 34 to a backend 36 of a web presentation system. The frontend of the system may be used by various users of the system including a

registrant/attendee of a presentation of the system and a presenter/client of the system. Each computing device of the frontend of the system may allow an entity, such as a user, company, etc., to connect to and interact with the backend 36, such as to register for a presentation, to submit a presentation or control a presentation, to submit a request for the predicted engagement of the attendees or to review/view/display the predicted engagement of the attendees for a particular future live presentation. Each computing device 32A, 32B or 32N may be a processor based device with one or more processors, memory, persistent storage, a display, input/output devices such as keyboards or printers, etc. and connectivity circuits that allow the user, using the computing device, to connect to the backend and then interact with the backend 36. For example, each computing device may be a personal computer, a smartphone or mobile device, a terminal device, a laptop computer, a tablet computer, etc. In an embodiment in which HTTPS/HTTP and HTML are used for protocols, each computing device may have a browser application or other application that receives data from the backend and generates a display of that data, such as a webpage and also allows the user to enter data into a form/webpage and send the data to the backend system 36.

The communication path 34 may be a wired path, a wireless path or a combination of the wired and wireless paths. Each piece of the path may be a wired network like Ethernet, a wireless computer network, a wired computer network, a wireless digital data network, a cellular digital data network, a WiFi network and the like. The communication path 34 may use various communication and data transfer protocols. For example, in one embodiment, the communications path may use TCP/IP and HTTP or HTTPS data transfer protocols and the HTML data protocols.

The backend 36 may be implemented using one or more computing resources, such as cloud computing or Amazon Web Services (AWS) resources or server computers. The computing resources used to implement the backend 36 are specifically configured such that, although the computing resources alone are generic computer elements, the combination of the computing resources and the software/hardware described below results in specialized hardware/software that performs the processes of the system as described below.

The backend 36 may include a presentation generator and streamer element 36A and an engagement determiner engine 36B. The presentation generator and streamer element 36 A may be used to help a presenter to generate each presentation, store the presentation, allow the user to control the presentation and delivery the presentation to each attendee and gather data about each attendee for each presentation. The engagement determiner engine 36B is used to gather engagement data, generate a classifier model and then use classification to determine an engagement (high or low in one example) of each attendee of a particular presentation before the presentation has occurred as described below in more detail. In the example in Figure 3, the engagement determiner engine 36B is incorporated into the online, web presentation system 36 as shown. However, the engagement determiner engine 36B may also be a standalone system or a software as a service system that provides its attendee engagement determination to a plurality of third party presentation systems that can provide the requisite data in order to perform the determination of the attendee engagement. Each of the presentation generator and streamer element 36A and the engagement determiner engine 36B may be implemented in hardware or software or a combination of hardware and software. When each of the presentation generator and streamer element 36A and the engagement determiner engine 36B is implemented in hardware, each of the elements may be a specialized hardware device, such as a field programmable gate array,

microcontroller, etc. that may be configured to perform the processes of the presentation generator and streamer element 36A or the engagement determiner engine 36B so that each of these elements is implemented using a specialized piece of hardware. When each of the presentation generator and streamer element 36A and the engagement determiner engine 36B is implemented in software, each of the elements may be a plurality of lines of computer code/instructions that may be executed by a processor of a computing resource of the backend 36 so that the processor is configured to perform the processes of the presentation generator and streamer element 36A or the engagement determiner engine 36B so that each of these elements is implemented using a computing resource with a plurality of lines of computer and it is the plurality of lines of code and the processes implemented that provide the technical solution. When each of the presentation generator and streamer element 36A and the engagement determiner engine 36B is implemented in hardware and software, each element may be a combination of the above described elements that again is a specialized computer system that implements the processes and provides the technical solutions. In the software implementation, the engagement determiner engine 36B may use Python code for the classifier and the database engine for the feature extraction and Java for the services.

Figure 4 illustrates more details of the front end 32 and backend 36 of the engagement prediction system. The front end 32 may further include a registrant element 48 and a client element 49 wherein each of these elements may be implemented using the computing devices as described above that allow a user (the soon to be attendee/registrant for the registrant element and the presenter for a presentation for the client element) to connect to and interact with the backend 36. For example, each of the elements 48, 49 may be a user interface displayed on the computing device that permits the user to interact with the backend 36. For example, the registrant element 48 may be a registration form that is displayed to the registrant and permits the registrant to enter information into a form, such as a web form in one embodiment, to register for a particular presentation. Thus, each registrant for a presentation may be presented with the registrant element user interface 49, an example of which is shown in Figure 4. As shown in Figure 4, a POST request communicates the information filled in by each registrant to the backend 36.

The client element 49 may be a user interface as well that may be displayed to a presenter of a particular presentation. The user interface may be generated for each presentation, such as the event X presentation in the example in Figure 4. The user interface may list each attendee/registrant of a future particular presentation and a predicted

engagement (high or low is the example in Figure 4) for each attendee/registrant that may be generated based on the output of the engagement determiner engine 36B. In one

implementation, the data used to generate the user interface may be retrieved from the backend 36 using a GET request.

The backend 36 may further comprise a feature extraction element 40 and a classification element 44. These two elements cooperate to perform the engagement prediction process and generating an engagement prediction for each attendee of a future presentation. Each of the feature extraction element 40 and a classification element 44 may be implemented in software or hardware as described above and may together be a specialized piece of hardware or software that provides the technical solution of the predicted engagement of each attendee for a future particular presentation.

The feature extraction element 40 may include a database 41 and a feature extractor 42. The feature extraction element 40 may receive the registrant/attendee registration data from each registrant/attendee through the POST request and all of that data may be stored in the database 41. The feature extractor 42 may be an algorithm/process that performs feature extraction based on part on the registrant's data and the feature extractor 42 may loop over every registrant to extract the features. The features extracted during this process may be communicated, in one embodiment, using a GET request, to the classification process 44.

Figure 5 illustrates more details of the feature extraction process 500. In the feature extraction process, the feature extractor 42 may use the registrant/attendee registration data to extract the features for determining the engagement. For example, the type of data for each attendee/registrant as shown in Figure 5 may include:

[USER_ID, CREATE TIMES TAMP, SOURCE, EVENT_DATE, INDUSTRY, SECTOR, EVENT TYPE, ENGAGEMENT SCORE]

Given the above data types, an example of the data from a particular

attendee/registrant may be: [133090061, 11/01/16, "seamlessGET", 02/09/10 , "Medical Devices" , "Life Sciences & Healthcare" , "Live Phone" , 4.6 ] of the examples of the data as shown in Figure 5.

The feature extraction process, using the above features, may extract those features from each attendee/registrant information. In one embodiment, the feature extraction may be done using a set of if/then rules (executed by a database engine) that determine a score, such as a numerical score for each feature. An example of the set of rules for the feature extraction are shown in Figure 5. The result of the feature extraction process is a feature vector for the particular attendee/registrant such as [5, 5, 1, 0, 80, 100] as shown in Figure 5. The feature vector may be used by the classification process to generate the engagement determination for the particular attendee. For example, if a user registered 4.4 months after the event, the registration was done by himself and because of the delayed registration, the probability of High engagement is reduced. The event was intended to be enjoyed live, this user is going to watch it on-demand that reduces the chances of better engagement. In another example, a user that registered less than less than a month before the event and the registration was done by himself, those factors increment the probability of High engagement. The event is type Live Phone that also increments the probability of High Engagement.

Returning to Figure 4, the classification element 44 may further include a classifier process 45, a save process 46 and the database 41. The classifier 45 may generate a model from the various extracted features and then perform a classifier process for each

registrant/attendee of the future presentation to determine an engagement for each attendee/registrant. The save process 46 may save the classification results and the classifier model and the engagement prediction results for each attendee/registrant to the database 41. In one embodiment, the results and data may be saved to the database using a POST request. When the client element 49 of the frontend 32 requests the engagement data for each attendee of a future event/presentation, that data may be retrieved from the database 41 via a GET request as shown in Figure 4.

Figure 6 illustrates more details of the classification process 600. In which a list of registrants/attendees are the particular event (Event X) are processed to generate the feature vector for each registrant/attendee that is communicated to the classifier 45, such as by a GET request in the example shown in Figure 6. The classifier 45 may include a decision tree classifier process 602 and a deep neural network process 604 that receive each feature vector for each registrant/attendee as an input. Each of the processes 602, 604 may be trained using feature vector data that includes the various feature scores and the desired engagement output for a low engagement and a high engagement. For example, the training data may be:

[ 31, 10 , 80 , 80 , 50 , 2 , 0 , HIGH ]

[ 9 , 8 , 10 ,70 , 50 , 1 , 4 , LOW ]

In one embodiment, the system may use a deep learning algorithm, such as the deep learning algorithm that is described more fully at

www.nature.corr nature/journa]/v521/n7553/abs/naturel4539.html?foxtrotcallba ck=true, and may use a decision tree such as the one described in more detail at //scikit- learn.org/stable/modules/tree.html, both of which are incorporated herein by reference.

The deep learning model consists of weights of units that conform several layers, so the final result would depend of the weights, the number of layers and activation function. For the case of the decision tree, the model consists of conditions that splits data in two groups in every branch, these branches can grow until the data cannot be split any more, this is called a leaf. So depending on the different features in the vector, we take a different path in the tree, so there is no factor for HIGH or LOW is a path of branches and leafs which takes to the final result. Both models are capable of discover deeply entangled structures in data, because there is no single feature that classifies the data completely, is the combination of all features which gives us the answer.

For each attendee/registrant, the trained decision tree classifier performs a

classification based on the feature vector and the deep neural network performs a similar process based on the feature vector. An ensemble learning technique called voting may be used. This technique is used to improve the final precision because every one of the models presents a higher error than when both are combined. The decision tree model is better to predict HIGH engagement than LOW engagement, while the deep learning model had a balanced error in both categories. When the two processes are combined, the accuracy for both categories are improved. As an example, the voting algorithm used may be: if at least one classifier predicts HIGH then the result is HIGH, if both classifiers predict LOW then the result is LOW.

The result from each of these processes is a HIGH engagement level or a LOW engagement level for the particular attendee/registrant. The process may then determine

(606), as part of a voting process, if at least one of the scores from the two classifier processes is HIGH. If at least one of the scores from the two classifier processes is HIGH, then the predicted engagement of the particular attendee/registrant is HIGH. If at least one of the scores from the two classifier processes is not HIGH, then the predicted engagement of the particular attendee/registrant is LOW. Although the decision tree classifier and the deep neural network are shown in Figure 6, the classifier process 600 may use other classifiers and /or more classifiers wherein the classifiers are suitable for categorical features and binary objective variable.

Figure 7 illustrates an example of a user profile 700 of an attendee/registrant that includes the engagement prediction wherein the engagement prediction data may be generated by the engagement predictor of the system. This user profile may display the generated engagement prediction and shows that the system and method interacts with users during the process and is thus not just a black box process without any external output. The user profile may include the future event being attended by the user (Event X in one example), the past events attended by the user (Event 1 and Event 2 in this example) that may be used to determine the user engagement for the future event and the predicted engagement for the user for the future presentation (high is this example).

In summary, the disclosed system and method uses the characteristics of the registration such as the method used, how many days prior the online event, the type of online content, among others, instead of time spent in the web page or amount of clicks, to determine an engagement level for each registrant/attendee. The context of this problem is different from other type of web content because we know that the registrant has the intention to consume the online content. We predict if he is going to watch and be present until the end, not if he is going to consume more content about the same topic, which is the usual approach.

The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the disclosure and its practical applications, to thereby enable others skilled in the art to best utilize the disclosure and various embodiments with various modifications as are suited to the particular use contemplated.

The system and method disclosed herein may be implemented via one or more components, systems, servers, appliances, other subcomponents, or distributed between such elements. When implemented as a system, such systems may include an/or involve, inter alia, components such as software modules, general-purpose CPU, RAM, etc. found in general- purpose computers. In implementations where the innovations reside on a server, such a server may include or involve components such as CPU, RAM, etc., such as those found in general-purpose computers.

Additionally, the system and method herein may be achieved via implementations with disparate or entirely different software, hardware and/or firmware components, beyond that set forth above. With regard to such other components (e.g., software, processing components, etc.) and/or computer-readable media associated with or embodying the present inventions, for example, aspects of the innovations herein may be implemented consistent with numerous general purpose or special purpose computing systems or configurations. Various exemplary computing systems, environments, and/or configurations that may be suitable for use with the innovations herein may include, but are not limited to: software or other components within or embodied on personal computers, servers or server computing devices such as routing/connectivity components, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, consumer electronic devices, network PCs, other existing computer platforms, distributed computing environments that include one or more of the above systems or devices, etc.

In some instances, aspects of the system and method may be achieved via or performed by logic and/or logic instructions including program modules, executed in association with such components or circuitry, for example. In general, program modules may include routines, programs, objects, components, data structures, etc. that performs particular tasks or implement particular instructions herein. The inventions may also be practiced in the context of distributed software, computer, or circuit settings where circuitry is connected via communication buses, circuitry or links. In distributed settings, control/instructions may occur from both local and remote computer storage media including memory storage devices.

The software, circuitry and components herein may also include and/or utilize one or more type of computer readable media. Computer readable media can be any available media that is resident on, associable with, or can be accessed by such circuits and/or computing components. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and can accessed by computing component. Communication media may comprise computer readable instructions, data structures, program modules and/or other components. Further, communication media may include wired media such as a wired network or direct- wired connection, however no media of any such type herein includes transitory media.

Combinations of the any of the above are also included within the scope of computer readable media.

In the present description, the terms component, module, device, etc. may refer to any type of logical or functional software elements, circuits, blocks and/or processes that may be implemented in a variety of ways. For example, the functions of various circuits and/or blocks can be combined with one another into any other number of modules. Each module may even be implemented as a software program stored on a tangible memory (e.g., random access memory, read only memory, CD-ROM memory, hard disk drive, etc.) to be read by a central processing unit to implement the functions of the innovations herein. Or, the modules can comprise programming instructions transmitted to a general purpose computer or to processing/graphics hardware via a transmission carrier wave. Also, the modules can be implemented as hardware logic circuitry implementing the functions encompassed by the innovations herein. Finally, the modules can be implemented using special purpose instructions (SIMD instructions), field programmable logic arrays or any mix thereof which provides the desired level performance and cost.

As disclosed herein, features consistent with the disclosure may be implemented via computer-hardware, software and/or firmware. For example, the systems and methods disclosed herein may be embodied in various forms including, for example, a data processor, such as a computer that also includes a database, digital electronic circuitry, firmware, software, or in combinations of them. Further, while some of the disclosed implementations describe specific hardware components, systems and methods consistent with the innovations herein may be implemented with any combination of hardware, software and/or firmware. Moreover, the above-noted features and other aspects and principles of the innovations herein may be implemented in various environments. Such environments and related applications may be specially constructed for performing the various routines, processes and/or operations according to the invention or they may include a general-purpose computer or computing platform selectively activated or reconfigured by code to provide the necessary functionality. The processes disclosed herein are not inherently related to any particular computer, network, architecture, environment, or other apparatus, and may be implemented by a suitable combination of hardware, software, and/or firmware. For example, various general-purpose machines may be used with programs written in accordance with teachings of the invention, or it may be more convenient to construct a specialized apparatus or system to perform the required methods and techniques.

Aspects of the method and system described herein, such as the logic, may also be implemented as functionality programmed into any of a variety of circuitry, including programmable logic devices ("PLDs"), such as field programmable gate arrays ("FPGAs"), programmable array logic ("PAL") devices, electrically programmable logic and memory devices and standard cell-based devices, as well as application specific integrated circuits. Some other possibilities for implementing aspects include: memory devices, microcontrollers with memory (such as EEPROM), embedded microprocessors, firmware, software, etc.

Furthermore, aspects may be embodied in microprocessors having software-based circuit emulation, discrete logic (sequential and combinatorial), custom devices, fuzzy (neural) logic, quantum devices, and hybrids of any of the above device types. The underlying device technologies may be provided in a variety of component types, e.g., metal-oxide

semiconductor field-effect transistor ("MOSFET") technologies like complementary metal- oxide semiconductor ("CMOS"), bipolar technologies like emitter-coupled logic ("ECL"), polymer technologies (e.g., silicon-conjugated polymer and metal-conjugated polymer-metal structures), mixed analog and digital, and so on.

It should also be noted that the various logic and/or functions disclosed herein may be enabled using any number of combinations of hardware, firmware, and/or as data and/or instructions embodied in various machine-readable or computer-readable media, in terms of their behavioral, register transfer, logic component, and/or other characteristics. Computer- readable media in which such formatted data and/or instructions may be embodied include, but are not limited to, non-volatile storage media in various forms (e.g., optical, magnetic or semiconductor storage media) though again does not include transitory media. Unless the context clearly requires otherwise, throughout the description, the words "comprise, " "comprising," and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in a sense of "including, but not limited to." Words using the singular or plural number also include the plural or singular number respectively.

Additionally, the words "herein," "hereunder," "above," "below," and words of similar import refer to this application as a whole and not to any particular portions of this application. When the word "or" is used in reference to a list of two or more items, that word covers all of the following interpretations of the word: any of the items in the list, all of the items in the list and any combination of the items in the list.

Although certain presently preferred implementations of the invention have been specifically described herein, it will be apparent to those skilled in the art to which the invention pertains that variations and modifications of the various implementations shown and described herein may be made without departing from the spirit and scope of the invention. Accordingly, it is intended that the invention be limited only to the extent required by the applicable rules of law.

While the foregoing has been with reference to a particular embodiment of the disclosure, it will be appreciated by those skilled in the art that changes in this embodiment may be made without departing from the principles and spirit of the disclosure, the scope of which is defined by the appended claims.