BENEFIT CLAIM

[0001] This application claims the benefit under 35 U.S.C. 119(e) of provisional application 63/343633, filed May 19, 2022, the entire contents of which are hereby incorporated by reference for all purposes as if fully set forth herein.

COPYRIGHT NOTICE

[0002] A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright or rights whatsoever. © 2021-2022 Grammarly, Inc.

TECHNICAL FIELD

[0003] One technical field of the present disclosure is artificial intelligence and machine learning as applied to natural language processing. Another technical field is computer- implemented grammatical error correction (GEC).

BACKGROUND

[0004] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

[0005] Computer-implemented grammatical error correction (GEC), offered as browser plug-ins or extensions, downloadable applications, and other forms of software have entered wide use. In one mode of use, GEC operates by receiving an electronically digitally stored draft text or source text in human natural language, such as an e-mail message that a user is composing, text entered in a text entry field in a browser that is connected to an online application, or text entered in a word processor document. All or a portion of the text is transmitted to a networked server and evaluated using trained machine learning models, resulting in generating output consisting of corrected text, changes, or suggestions for changes. The suggestions are transmitted back to the user and displayed near the original text, or as markup for the original text. The user can select whether to accept one or more suggestions to produce improved text.

[0006] While this mode is useful, it also has significant drawbacks. Typically the suggestions are generalized based on a large general-purpose training set. The suggestions typically do not reflect information that could be available about the purpose of a document or text, the recipient, or the language fluency of the recipient or a group. The user may have a writing style that is different than the style reflected in the suggestions. For example, some writers use sentence fragments as a stylistic choice, but typical GEC suggestions edit the “voice” out of a user’ s writing. The suggestions typically are generated without details of the context of the communication or a specialized domain of the writing; for example, specialized terms, punctuation, formatting, and repetition of terms in technical writing can lead to GEC suggestions that are inappropriate. Furthermore, newer applications such as SLACK, instant messengers, and chat applications may require a different style of communication than other writing.

[0007] Based on the foregoing, existing GEC systems and models do not produce the best possible text for a particular writer, recipient or audience, or context. There is an acute need in the field for a writing platform that does not merely aid editing but facilitates composition by integrating goals, profiles, structure, and editing into the machine-generated suggestions that are provided.

SUMMARY

[0008] The appended claims may serve as a summary of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the drawings:

[0010] FIG. 1 illustrates a distributed computer system showing the context of use and principal functional elements with which one embodiment could be implemented.

[0011] FIG. 2 illustrates an example process flow or algorithm that can be programmed to implement an embodiment of auto-dismissal of suggestions as an aspect of personalization.

[0012] FIG. 3 illustrates another example process flow or algorithm that can be programmed to implement an embodiment of boosting suggestions as an aspect of personalization. [0013] FIG. 4 illustrates a computer system with which one embodiment could be implemented.

[0014] FIG. 5A illustrates an example of a software architecture that can be used in an embodiment.

[0015] FIG. 5B illustrates an example data flow diagram showing how data values can be programmatically connected.

[0016] FIG. 6 illustrates an example distributed computer system showing the context of use and principal functional elements with which an embodiment could be implemented.

[0017] FIG. 7 A illustrates an example graphical user interface that could be implemented in association with inline boosting.

[0018] FIG. 7B illustrates an example graphical user interface that could be implemented in association with auto-dismiss.

[0019] FIG. 8A illustrates an example graphical user interface that could be implemented in association without and with upstream personalization and shows how the nature of suggestions may be different in upstream personalization.

[0020] FIG. 8B illustrates two views of an interface in which fewer or more suggestions are provided on a personalized basis.

[0021] FIG. 8C illustrates an example in which the aggressiveness of rewrites is tailored to the user or session.

[0022] FIG. 8D illustrates an example of a graphical user interface in which personalized suggestions for a particular user can be displayed.

[0023] FIG. 8E illustrates an example of a graphical user interface in which personalized suggestions for a particular audience can be displayed.

[0024] FIG. 9 is a two-part illustration of graphical user interfaces in which personalization can be used in promoting platform features.

[0025] FIG. 10A illustrates an example graphical user interface having a plurality of visual controls for selecting and invoking a plurality of different writing modes.

[0026] FIG. 10B illustrates the graphical user interface of FIG. 10A in which an Autopilot control has been selected in response to user input.

[0027] FIG. 11 illustrates an example of a suggestion leading to a user request to customize writing style features to cause future suggestions to be different. DETAILED DESCRIPTION

[0028] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.

[0029] The text of this disclosure, in combination with the drawing figures, is intended to state in prose the algorithms that are necessary to program the computer to implement the claimed inventions, at the same level of detail that is used by people of skill in the arts to which this disclosure pertains to communicate with one another concerning functions to be programmed, inputs, transformations, outputs and other aspects of programming. That is, the level of detail set forth in this disclosure is the same level of detail that persons of skill in the art normally use to communicate with one another to express algorithms to be programmed or the structure and function of programs to implement the inventions claimed herein.

[0030] 1. GENERAL OVERVIEW

[0031] In an embodiment, improved computer-implemented text processing techniques seek to achieve personalization of output suggestions by intelligently adapting the suggestions and the environment in which the suggestions are presented, to each user based on signals about the user’s writing style and the context of the communication. In some embodiments, the techniques are implemented in one or more computer programs, methods, functions, routines, or other software elements that are not specific to grammar correction but can be called from and/or used as a foundation framework for applications in multiple different domains relating to writing. With the approach of the disclosure, a variety of document processing or language processing applications can be personalized, and contextually relevant, by default.

[0032] The disclosure encompasses computer-implemented personalization of suggestions using a plurality of different processes. As one example, the text processing platform can be programmed to generate suggestions for a text, then re-rank the suggestions based on a ranking model that has been trained on signals indicating factors of relevance to the user. In practice, the ranking, coupled with programming to surface suggestions only when a suggestion has a rank greater than or equal to a personal threshold value, controls whether to show suggestions to a user based on whether the user is likely to accept, dismiss, or do nothing with the suggestions, as indicated in prior signals received from the user for similar suggestions. The model also can be trained on other features relating to a previous history of the interaction of the user with an application, text-related features based on the application of computational linguistics to an input text, by parsing the text to extract features indicating formality, fluency, or other attributes, user profile data specifying a demographic, job role, or industry, organizational profile, and/or stylometric features that capture a user's style of writing such as the usage of complex words, and/or a reading ease score. As another example, the text processing platform can be programmed to implement in-line boosting in which suggestions are shown that otherwise would not be. For example, in some embodiments, grammar, and spelling suggestions are always shown but suggestions related to clarity, tone, or other issues can be boosted and shown to a particular user based on personalization factors when they otherwise would not be shown on a generalized basis. As with other examples, the text processing platform can be programmed for automatic correction of a text including automatic correction of spelling, bulk acceptance of suggestions using a single input signal, and creating a rewritten version having a different tone or being shorter.

[0033] Examples of machine learning models that can be trained and used for producing suggestions include linear regression models, support vector machines, gradient boosting frameworks such as XGBoost, and neural networks.

[0034] Each of the processing functions described above can be based on collecting peruser data signals specifying interactions with each suggestion, thus forming a training dataset of historical personal interaction with applications, features, functions, or suggestions of the text processing platform. In one embodiment, training data comprises signals collected from user computers or user accounts for any of session data; the length of the text; the number of other suggestions; Writing context: what application the user is using (for example, browser extension, desktop, mobile editor); writing domain (charting app, document, email, business system); user goals (for example, formal or informal); user information (for example, whether the user is using a free or premium version of the text processing platform, what dialect the user uses such as British English, American English, or Canadian English), native language, how long the user has had an account. With a training dataset of this kind, the text-processing platform can be programmed to predict whether another suggestion is likely to be accepted. These functions can be supported by associating each suggestion with a position in a hierarchy of suggestions, and a value specifying what each suggestion improves.

[0035] User data signals can be processed using feature identification algorithms to produce feature values as output, which can be stored in a feature store in a digital storage device. Different sets of features can be extracted for purposes of different applications. Examples of features include features relating to a previous history of the interaction of the user with an application, text-related features based on the application of computational linguistics to an input text, by parsing the text to extract features indicating formality, fluency, or other attributes, user profile data specifying a demographic, job role, or industry, organizational profile, and/or stylometric features that capture a user's style of writing such as the usage of complex words, and/or a reading ease score.

[0036] Embodiments are programmed to protect user privacy by multiple techniques including but not limited to refraining from storing text that users enter; refraining from storing any values associated with the text, a user computer, or a user account that constitute personally identifiable information (PII) or indicate protected characteristics; processing data using deidentifying processes or other security controls; storing output suggestions only in encoded, tokenized, and/or encrypted forms so that the substance of the content is not stored.

[0037] 2. STRUCTURAL & FUNCTIONAL OVERVIEW

[0038] 2.1 DISTRIBUTED SYSTEM EXAMPLES

[0039] FIG. 1 illustrates a distributed computer system showing the context of use and principal functional elements with which one embodiment could be implemented.

[0040] In an embodiment, a computer system 100 comprises components that are implemented at least partially by hardware at one or more computing devices, such as one or more hardware processors executing stored program instructions stored in one or more memories for performing the functions that are described herein. In other words, all functions described herein are intended to indicate operations that are performed using programming in a special-purpose computer or general-purpose computer, in various embodiments. FIG. 1 illustrates only one of many possible arrangements of components configured to execute the programming described herein. Other arrangements may include fewer or different components, and the division of work between the components may vary depending on the arrangement.

[0041] FIG. 1 , and the other drawing figures and all of the description and claims in this disclosure, are intended to present, disclose and claim a technical system and technical methods in which specially programmed computers, using a special-purpose distributed computer system design, execute functions that have not been available before to provide a practical application of computing technology to the problem of machine learning model development, validation, and deployment. In this manner, the disclosure presents a technical solution to a technical problem, and any interpretation of the disclosure or claims to cover any judicial exception to patent eligibility, such as an abstract idea, mental process, method of organizing human activity, or mathematical algorithm, has no support in this disclosure and is erroneous. [0042] In the example of FIG. 1, computing device 102 is communicatively coupled via a network 120 to a text processor 140. In one embodiment, computing device 102 comprises a client-type computing device such as a personal computer, laptop computer, tablet computer, smartphone, or notebook computer. For purposes of illustrating a clear example, a single computing device 102, network 120, and text processor 140 are shown in FIG. 1, but practical embodiments may include thousands to millions of computing devices 102 distributed over a wide geographic area or over the globe, and hundreds to thousands of instances of text processor 140 to serve requests and computing requirements of the computing devices.

[0043] Computing device 102 comprises, in one embodiment, a central processing unit (CPU) 101 coupled via a bus to a display device 112 and an input device 114. In some embodiments display device 112 and input device 114 are integrated, for example, using a touch- sensitive screen to implement a soft keyboard. CPU 101 hosts operating system 104, which may include a kernel, primitive services, a networking stack, and similar foundation elements implemented in software, firmware, or a combination. Operating system 104 supervises and manages one or more other programs. For purposes of illustrating a clear example, FIG. 1 shows the operating system 104 coupled to an application 106 and a browser 108, but other embodiments may have more or fewer apps or applications hosted on computing device 102. For example, soft keyboards of mobile devices such as the iOS keyboard of APPLE mobile devices can integrate aspects of application 106 and input device 114 and the text processor 140 can interoperate with such a soft keyboard like an application.

[0044] At runtime, one or more of application 106 and browser 108 loads, or are installed with, a text processing extension 110A, HOB, which comprises executable instructions that are compatible with text processor 140 and may implement application-specific communication protocols to rapidly communicate text-related commands and data between the extension and the text processor. Text processing extensions 110A, 110B may be implemented as runtime libraries, browser plug-ins, browser extensions, or other means of adding external functionality to otherwise unrelated, third-party applications or software. The precise means of implementing a text processing extension 110A, HOB or to obtain input text is not critical provided that an extension is compatible with and can be functionally integrated with a host application 106 or browser 108. [0045] In some embodiments, a text processing extension 110A may install as a standalone application that communicates programmatically with either or both of the operating system 104 and with an application 106. For example, in one implementation, text processing extension 110A executes independently of application 106 and programmatically calls services or APIs of operating system 104 to obtain the text that has been entered in or is being entered in input fields that the application manages. Accessibility services or accessibility APIs of the operating system 104 may be called for this purpose; for example, an embodiment can call an accessibility API that normally obtains input text from the application 106 and outputs speech to audibly speak the text to the user, but use the text obtained by the accessibility service in the processes that are described for FIG. 2 and other sections herein.

[0046] In some embodiments, each text processing extension 110A, 110B is linked, loaded with, or otherwise programmatically coupled to or with one or more of application 106 and browser 108 and, in this configuration, is capable of calling API calls, internal methods or functions, or other programmatic facilities of the application or browser. These calls or other invocations of methods or functions enable each text processing extension 110 A, HOB to detect text that is entered in input fields, windows, or panels of application 106 or browser 108, instruct the application or browser to delete a character, word, sentence, or another unit of text, and instruct the application or browser to insert a character, word, sentence, or another unit of text.

[0047] Each of the text processing extensions IlOA, HOB is programmed to interoperate with a host application 106 or browser 108 to detect the entry of text in a text entry function of the application or browser and/or changes in the entered text, to transmit changes in the text to text processor 140 for server-side checking and processing, to receive responsive data and commands from the text processor, and to execute presentation functions in cooperation with the host application or browser.

[0048] As one functional example, assume that browser 108 renders an HTML document that includes a text entry panel in which a user can enter free-form text describing a product or service. The text processing extension HOB is programmed to detect user selection of the text entry panel, the entry of text, or changes in the text within the panel, and to transmit all such text changes to text processor 140. In an embodiment, each text processing extension 110A, 110B is programmed to buffer or accumulate text changes locally over a programmable period, for example, five seconds, and to transmit the accumulated changes over that period as a batch- to-text processor 140. Buffering or accumulation in this manner, while not required, may improve performance by reducing network messaging roundtrips and reducing the likelihood that text changes could be lost due to packet drops in the networking infrastructure. One commercial example of text processing extensions 110A, HOB is the GRAMMARLY extension, commercially available from Grammarly, Inc.

[0049] Network 120 broadly represents one or more local area networks, wide area networks, campus networks, or internetworks in any combination, using terrestrial or satellite, wired, or wireless network links.

[0050] In an embodiment, the text processor 140 comprises one or more server computers, workstations, computing clusters, and/or virtual machine processor instances, with or without network- attached storage or directly attached storage, located in any of enterprise premises, private data center, public data center and/or cloud computing center. Text processor 140 broadly represents a programmed server computer having processing throughput and storage capacity sufficient to communicate concurrently with thousands to millions of computing devices 102 associated with different users or accounts. For purposes of illustrating a clear example and focusing on innovations that are relevant to the appended claims, FIG. 1 omits basic hardware elements of text processor 140 such as a CPU, bus, I/O devices, main memory, and the like, illustrating instead an example software architecture for functional elements that execute on the hardware elements. Text processor 140 also may include foundational software elements not shown in FIG. 1, such as an operating system consisting of a kernel and primitive services, system services, a networking stack, an HTTP server, other presentation software, and other application software. Thus, text processor 140 may execute on a first computer, and text processing extensions 110A, HOB may execute on a second computer.

[0051] In an embodiment, text processor 140 comprises a change interface 142 that is coupled indirectly to network 120. Change interface 142 is programmed to receive the text changes that text processing extensions 110A, HOB transmit to text processor 140, and to distribute the text changes to a plurality of different checks 144A, 144B, 144C. To illustrate a clear example, source text 130 of FIG. 1 represents one or more text changes that text processing extension 110B transmits to change interface 142. In an embodiment, change interface 142 is programmed to distribute each and every text change arriving from a text processing extension 110A, 110B to all of the checks 144A, 144B, 144C, which execute in parallel and/or in independent threads.

[0052] Thus, in one embodiment, the text processor 140 may be programmed to programmatically receive a digital electronic object comprising a source text, a message with the source text, an application protocol message with the source text, an HTTP POST request with the source text as a payload, or using other programmed mechanics. In various embodiments, the first computer executes a text processor that is communicatively coupled to a text processor extension that is executed at the second computer and programmatically receives the digital electronic object comprising the source text via a message initiated at the text processor extension and transmitted to the text processor; and/or the text processor extension executes in association with an application program that is executing at the second computer, the text processor extension being programmed to automatically detect a change in a text entry window of the application program and, in response, to initiate the message; and/or the text processor executes in association with a browser that is executing at the second computer, the text processor extension being programmed to automatically detect a change in a text entry widget of the browser and, in response, to initiate the message.

[0053] Each of the checks 144A, 144B, 144C is programmed to execute a different form of checking or processing of a text change that has arrived. Example functions that checks 144A, 144B could implement include grammar checking, tone detection, and translation. In an embodiment, check 144C is programmed as a style check to implement style or personalization checks and suggestions, and therefore it is also denoted “style check 144” in this description. In an embodiment, style check 144 comprises a multi-class text classifier 146 coupled to summary suggestion instructions 148, which are coupled to ranking instructions 150; however, other machine learning models can be used. For example, an embodiment may use a number of individual text classifiers ensembled together, or targeted rules may be programmed to find relevant words and then coupled to a classifier to approve or reject whether the instance of a word is correct, thus using a coarse rule followed by ML-based filtering.

[0054] Furthermore, style check 144C is coupled to, or can access, a phrase store 170, which may be integrated with text processor 140 or implemented as separate storage. In an embodiment, phrase store 170 comprises a database, flat file system, object store, or another digital data repository that stores a large number of textual phrase suggestions, in association with category values or tags that specify a category or type of communication, text, or document in which the suggestions could be substituted. Thus, style check 144 and/or text processor 140 may be programmed for evaluating each particular source text unit among the plurality of source text units using a trained multi-class text classifier machine learning model and receiving a classification output from the multi-class text classifier that classifies each particular source text unit as a particular class of phrase among a plurality of possible classes of phrases. As further described herein for FIG. 2, in an embodiment, phrase suggestion instructions 148 are programmed, in part, to output a suggestion set 132 to transmit to text processing extension HOB.

[0055] FIG. 2 illustrates an example process flow or algorithm that can be programmed to implement an embodiment of auto-dismissal of suggestions as an aspect of personalization. FIG. 2 and each other flow diagram herein are intended as an illustration of the functional level at which skilled persons, in the art to which this disclosure pertains, communicate with one another to describe and implement algorithms using programming. The flow diagrams are not intended to illustrate every instruction, method object, or sub-step that would be needed to program every aspect of a working program, but are provided at the same functional level of illustration that is normally used at the high level of skill in this art to communicate the basis of developing working programs.

[0056] A process 200 of FIG. 2 initiates execution at step 201 in which the process receives text via a text processor extension and transmits the text to a text processor. For example, in the context of FIG. 1, one of the text processing extensions 110A, HOB receives a text from an application, browser, soft keyboard, or other source and transmits source content 130 via network 120 to change interface 142 of text processor 140. At step 202, the text is received by the text processor.

[0057] At step 204, process 200 is programmed to process the text to identify one or more corrections to the received text. For purposes of illustrating a clear example, step 204 refers to corrections, but in other embodiments, changes to the text can be based upon any of the user personalization factors, signals, and models that are described elsewhere herein, or any of the recipient, group, team, or audience personalization factors, signals and models that are described elsewhere herein, and correctness should be understood as merely one example goal or purpose of generating changes.

[0058] At step 206, process 200 is programmed to generate one or more suggestions based on the corrections. Suggestions can be based upon any of the user personalization factors, signals, and models that are described elsewhere herein, or any of the recipient, group, team, or audience personalization factors, signals, and models that are described elsewhere herein. At step 208, process 200 is programmed to submit the suggestions to a suggestion classifier for evaluation; the suggestion classifier can apply one or more additional levels of ranking, filtering, or other processing to achieve personalization. [0059] At step 210, process 200 is programmed to classify and score each suggestion and then, at step 212, to retrieve a predetermined threshold value and compare each of the suggestions to the threshold value. In this context, “predetermined” can mean digitally stored, programmed, or configured at any time asynchronously with respect to process 200. A threshold value can be digitally stored in the phrase store 170 or another data repository of personalization data, in association with or pointed to by records of a user account. That is, each user account or user has a threshold value for suggestions to control whether to show the suggestions in a user interface near the input text, or to suppress display or automatically dismiss the suggestions. A user account can be associated with a plurality of different threshold values for different categories of suggestions; example categories are shown in other sections and drawing figures herein. Each threshold value can be determined on a personalized basis based on any of the user personalization factors, signals, and models that are described elsewhere herein.

[0060] At step 214, process 200 is programmed to test whether the suggestion is ranked above the threshold value; if so, then control transfers to step 216 at which the process outputs the suggestion to the text processor extension of step 201 for presentation in a user interface. Then, at step 220, using the text processor extension, the process is programmed to update the text using the suggestion. Control may transfer to step 224 to update training data based on whether the suggestion is accepted or declined.

[0061] Alternatively, if the test of step 214 is negative, then at step 218, process 200 is programmed to automatically dismiss the suggestions from the user interface. At step 222, optionally, at the text processor, the suggestions having ranking values below the threshold are received so they can be shown to the user in response to a selection of a control function in the user interface.

[0062] FIG. 3 illustrates another example of process flow or algorithm that can be programmed to implement an embodiment. In an embodiment, process 300 is programmed for receiving, at step 310, under program control, data representing a first text sequence in a first language.

[0063] At step 320, process 300 is programmed for evaluating the first text sequence using a machine learning model, the first text sequence containing a plurality of source text units, the machine learning model having been trained using a first dataset, and generating a second dataset that comprises one or more corrected source text units corresponding to the plurality of source text units of the first text sequence. Text units can be words, clauses, sentences, or n- grams, in various embodiments.

[0064] At step 330, process 300 is programmed for transforming, using the machine learning model, the second dataset to yield a plurality of output suggestions. Step 330 also can be programmed for generating, using the machine learning model, a score corresponding to each of the plurality of output suggestions being scored based on one or more features. The features can be determined on a personalized basis based on any of the user personalization factors, signals and models that are described elsewhere herein.

[0065] At step 340, process 300 is programmed for displaying a graphical control element indicative of the first set of output suggestions, each suggestion of the first set of output suggestions corresponding to a score value above a predetermined threshold. With this approach, process 300 can be programmed to automatically boost and surface selected suggestions that otherwise would not have been shown to a user account, user computer, or user. At step 350, process 300 is programmed for modifying the first text sequence to result in creating and digitally storing, or displaying, a second text sequence in the first language.

[0066] FIG. 5A illustrates an example of software architecture that can be used in an embodiment. A personalization platform 500 can comprise one or more virtual compute instances and virtual data storage instances organized as a feature store 502, feedback storage 504, and UTC data storage 506. Each of the feature store 502, feedback storage 504, and UTC data storage 506 can comprise one or more tables in a relational database, a set of flat files, object storage, or other data storage. Data storage also can comprise a plurality of different personal and contextual signals 508. Examples of personal and contextual signals include a user profile 510, context data 514 such as a domain of communication or level of formality, personalized n-grams 512 based on entered text, and other signals.

[0067] The personalization platform 500 can comprise a plurality of software components 516 organized as sets of stored program instructions that implement an application programming interface (API) 518 for relevance and ranking; a model training and inference component 520; a feature extraction component 522 to determine user feature data and context data; and a service health monitoring component 524 that is programmed to monitor the platform 500 and report or store statistical data relating to performance attributes. The API 518 can be called by a variety of higher-order applications 526 including intelligence, business, developer, and marketing applications. [0068] Digitally stored signal data 508 from users, teams, suggestions, sessions, and documents provides the basis for evaluating machine learning models to determine specific suggestions that are personalized and contextually relevant. Example signals include:

[0069] User signals. Account age; free or premium account status; the primary language of the user; the secondary language of the user; Demographics; Job title; industry; Engagement history; Proficiency; Writing style; Reading style; Writing process; Personality; Network; Longer term goals; Knowledge, expertise.

[0070] User profile data 510 can specify a personal communication style, attributes of the user as a writer, and how the user uses the platform. The user profile is a central part of the various signals in the personalization platform and is made up of a few parts: Communication style: Word choice and vocabulary, English writing proficiency, sentence structure, punctuation, average document formality, length, or complexity, emoji usage, tones, personal “-isms”, etc. Attributes: Demographics, job or field of study, writing process, personality, network, long-term goals, knowledge, expertise, etc. Application history: How long they’ve been using the platform, which suggestions they engage with, what their writing and editing sessions look like, etc.

[0071] Team signals. Size; Teammates; Customers; Competitors; Org structure; Vertical; Shared goals; Key content; Brand voice.

[0072] Suggestion signals. Surface (inline); outcome (tone); rarity; average acceptance rate; average perception; attributed upgrades; quality; obviousness; clarity.

[0073] Session signals. Suggestion volume, suggestion interactions, container (comment), client type such as iOS Keyboard; application in which text is entered such as SLACK; site such as a website; user urgency level; user attention level; stage of writing.

[0074] Document signals. Length; goals that have been set; tones; audience; thread context; desired outcome; actual outcome; domain (academic); genre (medicine); subgenre (neurology); formality; technicality; subjects, topics; stakes.

[0075] Collectively, session signals and document signals can indicate context. Any or all the foregoing signals can be collected and added to a suggestion relevance model for use by a variety of applications.

[0076] Contextualization of output can apply to a plurality of different domains. For example, the user may be primarily focused on correctness, independent of user comprehension. Alternatively, the user context may be more important, including factors such as comprehension, stylistic choice, intentional rule-breaking, or prioritization. Or, audience context may dominate, including stylistic choice, intentional rule-breaking, or prioritization, as well as industry standards or corporation or brand standards. Finally, the context of a medium may be most important, again taking into account stylistic choice, intentional rule-breaking, or prioritization, as well as industry standards or corporation or brand standards. In this manner, the platform can be programmed to optimize not just for correctness, but for attributes or traits that matter to users or teams.

[0077] FIG. 5B illustrates an example data flow diagram showing how data values can be programmatically connected. In an embodiment, a personalization engine 501 comprises the personalization platform 500 (FIG. 5A) in combination with one or more machine learning models and digitally stored data, such as a general suggestion relevance model that has been trained on the signals and data previously described, digitally stored user profiles and team profiles, context data, domain-based knowledge graphs that represent terms, documents, web pages, and other knowledge of a particular enterprise, team, project, or other domain, specialized ML models for new user activation, conversion RL models, and other decision data such as digitally stored rules, heuristics, model limit parameters or “guardrails,” weight values, and default values. Examples of training features include features relating to a previous history of the interaction of the user with an application, text-related features based on the application of computational linguistics to an input text, by parsing the text to extract features indicating formality, fluency, or other attributes, user profile data specifying a demographic, job role, or industry, organizational profile, and/or stylometric features that capture a user's style of writing such as the usage of complex words, and/or a reading ease score. The personalization engine 501 is programmatically coupled, via an API or other programmatic communication, to one or more text processing products 530, user computers 102, and computational clients 532. Text processing products 530 can comprise a grammatical error correction check, an application, or system, a clarity check, a tone check, style guide checks, and others. The user computers 102 can be associated with individuals, teams, or other user groups. The computational clients 532 can comprise the text processing extensions 110A, HOB, a text editor, a soft keyboard, or an application interface.

[0078] In operation, in one embodiment, user computers 102 make changes to configuration parameters or other settings of applications, as path 542 indicates. User computers 102 also contribute activity signals and feedback signals to the personalization engine 501. Products 530 can receive all possible suggestions 546 to change a text from the personalization engine 501 based on providing a source text to the personalization engine and receiving suggestions, predictions, or classifications resulting from executing an inference stage of the ML models of the personalization engine over the source text.

[0079] After processing the source text using the inference stage of the ML models based on the signals and model training, the personalization engine 501 provides personalized suggestions 549 to clients 532 such as a browser extension, editor, keyboard, applications hosted under an operating system, or other target endpoints. Those clients 532 can communicate suggestions 548, using a graphical user interface specific to the clients and the products or applications, to user computers 102. The data flow thus forms a continuous loop in which feedback continuously improves personalized suggestions.

[0080] FIG. 6 illustrates an example distributed computer system showing the context of use and principal functional elements with which an embodiment could be implemented. In an embodiment, source content 130 (FIG. 1) or other live text from any of the sources previously identified is received via the change interface 142, implemented as an API at a networked server. Asynchronously, or in response to receiving the source content 130, a session feature computation component 147 identifies session feature data such as the type of external system from which the source content was received, the user task, the user’s language, or other signals relevant to a then-current text editing session of the user computer 102. Asynchronously, demographic data 602 is received by a data store 604 such as DATABRICKS and precomputed as feature data in an offline data store 502A. The session feature computation component 147 also can reference a grammar rule store or grimoire 610.

[0081] As user feedback signals are received, indicating acceptance of suggestions or decline signals, the feedback signals arrive in a feedback event stream 620 to enrich the precomputed features data. Feedback signals also can represent features relating to a previous history of the interaction of the user with an application, text-related features based on the application of computational linguistics to an input text, by parsing the text to extract features indicating formality, fluency, or other attributes, user profile data specifying a demographic, job role, or industry, organizational profile, and/or stylometric features that capture a user's style of writing such as the usage of complex words, and/or a reading ease score. In some embodiments, live text can be sampled to produce sampled session text 131 and replayed using a replay system 615 to generate added feedback events in the feedback event stream 620, providing a way to jump-start the enrichment of the offline store. The offline store 502A can be updated asynchronously to a pre-computed feature online store 502B to process queries from the change interface 142 for fast, real-time response to the live text input or source content 130. Both the source content 130 and feature data from the session feature computation component 147 are input to a personalization/relevance service equivalent to the personalization engine 501 (FIG. 5B) which evaluates the feature data against trained machine learning models to generate output suggestions.

[0082] A platform programmed according to these embodiments can implement personalization in a variety of ways. A text editing experience can become user-aware and context-aware by providing auto-dismiss of suggestions, auto-acceptance, and boosting of suggestions to inline presentation. Non-suggestion features can be made dynamic via suggestion configurations or toggles, dialects, dictionaries, style guides, bulk accepts, smart phrases, and snippets. Suggestions can be personalized to add clarity or tone revisions that adapt suggestions to the user’s writing voice.

[0083] Personalization also can enable the development of communication profiles, and coaching applications to build better writing habits and others. A dynamic communication profile can highlight a user’s unique voice and writing progress. For example, digitally stored data for attributes of a communication profile can be stored in a database record associated with a user account record and can specify attributes such as “optimistic communicator,” numbers of unique words, numbers of unclear sentences, and other. Personalized coaching can specify changes to user writing based on a profde and signal data specifying how past suggestions have been accepted or declined.

[0084] 2.2 DOWNSTREAM PERSONALIZATION

[0085] In some embodiments, the platform is programmed to implement downstream personalization, in which generic suggestions are first generated from machine learning models and then ranked in a personalized way.

[0086] INLINE BOOSTING. Downstream personalization can be programmed in one aspect to implement inline boosting in which relevant suggestions are boosted from an assistant panel to an inline presentation. FIG. 7 A illustrates an example graphical user interface that could be implemented in association with inline boosting. To illustrate an example embodiment, a graphical user interface 702 comprises an email composing window 704 in which a user computer 102 has entered draft text 706. Using the techniques that have been previously described, the checks of FIG. 1 have executed the inference stage of trained GEC ML models over the draft text and returned a suggestion set 132 that a text processing extension has rendered using underlining to highlight one or more suggestions 705 of changes to the text. Based on the scoring approach of FIG. 2, suggestion 708 to change the word “mandatory” has exceeded a threshold score and therefore qualified for emphasis.

[0087] In response, in an embodiment, the functional elements of style check 144C (FIG. 1) operating as described in FIG. 2 are programmed to automatically generate and transmit presentation instructions to render, at the user computer 102, a boost panel 710 relating to the suggestion 708. Boost panel 710 comprises an explanation 711, which can be obtained as a portion of the output from executing the ML models, that specifies why suggestion 708 has been boosted; in the example, the suggestion has been emphasized to assist the user in improving the receiver’s engagement with the text 706 based on past high-stakes conversations that the ML models have been trained on. The boost panel 710 further comprises a suggestion 712 showing a proposed change of “mandatory” to “necessary.” The label “necessary” is a default option among a plurality of options 714 that the ML models have output, as indicated by a visual emphasis applied to the option such as highlighting, boldface, or color. Each of the options 714 can be rendered as a selectable UI widget, such as a button widget. In response to input from user computer 102 to select one of the UI widgets, the boost panel 710 is programmed to visually emphasize the corresponding label of the UI widget. The boost panel 710 is programmed to display a plurality of action widgets 716 including but not limited to ACCEPT and DISMISS. In response to input from user computer 102 to select the ACCEPT action widget, the boost panel 710 is programmed to apply the suggestion, remove the visual emphasis of the suggestion 708, and transmit a feedback signal to the system.

[0088] Thus, inline boosting makes relevant suggestions more visible to the user, increasing the likelihood that the suggestions are accepted and will result in a better final text. [0089] AUTO-DISMISS. Downstream personalization can be programmed in one aspect to implement auto-dismiss in which irrelevant suggestions are moved to the end of a set of suggestions and graphical controls are provided to make bulk dismissal of the suggestions easier. In various embodiments, auto-dismiss can be programmed as a process of receiving suggestions that are normally generated by evaluating inference stage of the trained ML models over the source content 130 and then skipping program steps that otherwise would require rendering and displaying the suggestions as well as marking the suggestions in digital storage as suppressed. Additionally or alternatively the auto-dismiss programming can include reranking the suggestions to weigh one or more suggestions artificially high so that later processing steps will not select those suggestions for surfacing in the display. [0090] FIG. 7B illustrates an example graphical user interface that could be implemented in association with auto-dismiss. In the first user interface panel 720, the personalization platform 500 can be programmed to notify the user computer 102 that one or more suggestions have been hidden because they did not appear relevant based on known context or other personalization signals, as specified in notification 722. Notification 722 includes a selectable text panel or icon 724 that prompts the user to select the panel or icon if a review is desired. In an embodiment, if the panel or icon 724 is selected, then the platform 500 is programmed to automatically update the display to render a second user interface panel 726 to show the suggestions 728 that had been previously hidden with selectable options 730 to dismiss all the suggestions or review them individually. In an embodiment, auto-dismiss can increase an accept-to-look rate for suggestions.

[0091] 2.3 UPSTREAM PERSONALIZATION

[0092] Upstream personalization refers to applying personalization models at the time that the original text is drafted or created. Upstream personalization can be programmed to suggest edits or changes to a text that reflect an individual user’s specialized language.

[0093] FIG. 8A illustrates an example graphical user interface that could be implemented in association without and with upstream personalization and shows how the nature of suggestions may be different in upstream personalization. In an embodiment, source content 130 comprises the clause “We’ll produtcionize the model” and contains an apparent spelling error. In one approach without upstream personalization, evaluating the inference stage of trained ML models associated with the check 144 results in generating an output suggestion 132 that is rendered in a first UI panel 802 and suggests changing the specified word to “production.” A suggestion of this kind does not account for user context, enterprise knowledge, and/or specialized language. In another embodiment, with upstream personalization, evaluating the inference stage of trained and personalized ML models associated with the check 144 results in generating an output suggestion 132 that is rendered in a second UI panel 804 and suggests changing the specified word to “productionize.” A suggestion of this kind accounts for user context, enterprise knowledge, and/or specialized language. In an embodiment, panel 804 comprises explanation 806 that specifies a reason for the suggestion to notify the user computer 102 that the suggestion, which might appear incorrect to some readers, is based upon personalization. Both the first UI panel 802 and the second UI panel 804 can comprise active links, widgets, or labels 808, 810, 812 that are respectively programmed to signal the check 144C to dismiss the suggestion, add the suggestion to a personal dictionary, or open other tools or controls.

[0094] FIG. 8B illustrates two views of an interface in which fewer or more suggestions are provided on a personalized basis. The left side 820 of FIG. 8B illustrates suggestions in which signals and training have specified showing fewer false positive suggestions. In the example, processing the source content 130 has resulted in output suggestions 132 that have been rendered to visually emphasize the word “However” at 822 and a suggestion 824 that identifies the word and suggests adding a comma. The right side 830 of FIG. 8B shows a result of processing the same source content 130 with more suggestions 832 that are relevant, and acceptance of all such suggestions would result in a different style than for the left side 820 of FIG. 8B.

[0095] FIG. 8C illustrates an example in which the aggressiveness of rewrites is tailored to the user or session. In an embodiment, the left side 840 of FIG. 8C shows suggestions 844 that minimize changes in meaning, and an explanation 842 that the suggestions relate to clarity. In contrast, the right side 850 of FIG. 8C shows suggestions 852 that maximize conciseness, while still relating to clarity as indicated by explanation 842. Each of the left side 840 and right side 842 represent different examples, and each may comprise active links, widgets, or labels 846 that are programmed to signal the check 144C to dismiss the suggestion, rephrase the suggestion by re-running the ML models with different input, or open other tools or controls. Suggestions of each type can be generated based on personalization using the signals that have been previously described.

[0096] FIG. 8D illustrates an example of a graphical user interface in which personalized suggestions for a particular user can be displayed. In the example, a set of top suggestions for the user based on personalized style and content are displayed with selectable options to accept all or dismiss all. A graphical user interface 850 can comprise a message composition window 852 in which the user computer 102 has input source content 130 for evaluation. In response, evaluation of the source content 130 using check 144C has resulted in generating output suggestions 132 that have been rendered in window 852 as suggestion 854. Further, check 144C is programmed to generate presentation instructions to render, in the GUI 850, a top suggestions panel 856 comprising a list 858 of multiple suggestions, response widgets 860, and multiple individual suggestions 862. Each of the multiple suggestions in the list 858 can comprise an active icon or link such as a trashcan which, when selected, causes dismissing the associated suggestion. The response widgets 860 can comprise ACCEPT and DISMISS widgets which, when selected, cause the check 144C to accept and implement all the suggestions or dismiss all of them. Selecting the ACCEPT widget can cause re-rendering the composing window 852 to show all the suggestions in the text without visual emphasis.

[0097] The specific suggestions in list 858 can correspond to the source content 130 shown in FIG. 8E. FIG. 8E illustrates an example of a graphical user interface in which personalized suggestions for a particular audience can be displayed. Source content 130 comprises text in a message composing window 852 of an email application. Processing the source content 130 has resulted in visually emphasizing a portion 864 of the source content 130 corresponding to a suggestion 868 shown in a suggestion panel 866 having response widgets 860. In the example, in response to the selection of the three-line suggestion 864, the platform is programmed to generate and cause displaying a pop-up suggestion panel 866 containing a notification 867 that explains the suggestion in a personalized way. In this case, the panel specifies that “Your audience responds quickly to short emails” as an aspect of clarity, and provides a suggestion appropriate for the audience in which certain words are deleted.

[0098] 2.4 PERSONAEIZATION FOR PEATFORM MARKETING

[0099] Personalization models based on the foregoing also can be applied to the improvement of business outcomes for an organization that provides the specified platform. Personalization based on models of user action and the signals described herein can be applied at stages of the acquisition of a user or customer, activation, conversion, and/or retention. Personalization in customer acquisition can refer to: Advertise our personalization features to differentiate the product; Generate and hyper-target our ads and landing pages with our user and domain knowledge. Personalization in activation can refer to: From Day 1 as a customer, create the sense that the platform will personalize to your specific writing needs; Emphasize variety and impressiveness of the product in the first days. Personalization in conversion can refer to: Personalize our selling points and any “try before you buy” features like previews and trials to optimize conversion rates. Personalization in retention can refer to: Continue to make the product better tailored to the user the more they use it.

[0100] FIG. 9 is a two-part illustration of graphical user interfaces in which personalization can be used in promoting platform features. The left side 902 of FIG. 9 shows an interface having a message composing window 852 with source content 130, which has been processed to generate a visually emphasized portion 864 of text corresponding to a suggestion 908 in a suggestion panel 905 having a notification panel 906 and an action widget 910. In an embodiment, the notification panel 906 can display a notification indicating that check 144C is providing no-cost premium suggestions to a user who is not a premium service customer. [0101] In an embodiment, the right side 904 of FIG. 9 shows providing a similar suggestion in a customized domain such as a student essay. Source content 130 comprises visually emphasized portion 864 corresponding to suggestion 912 in suggestion panel 914. In this case, notification panel 906 can specify that the suggestion is a premium suggestion. The suggestions of each side 902, 904 of FIG. 9 are personalized based on the context of the user and the document.

[0102] 2.5 PERSONALIZATION BASED ON LEVELS OF USER INTENT

[0103] In an embodiment, the platform can be programmed to provide visual, graphical, selectable widgets which when selected cause activating different text editing and suggestion services that can vary based upon levels of user intent and/or stages of writing. FIG. 10A illustrates an example graphical user interface having a plurality of visual controls for selecting and invoking a plurality of different writing modes. In an embodiment, a GUI 1000 comprises source content 130, a control panel 1002, and a popup menu 1006 having a plurality of active links 1008. In the control panel 1002, example controls are labeled Research, Compose, Edit, and Autopilot, and can reflect whether the user is then currently at the research stage of a document with less need for corrective suggestions, composing a document such that a close review of text for suggestions is needed, editing a previously prepared document that may need only polishing, or operating without the need to carefully consider each selection.

[0104] In the example of FIG. 10A, user input has selected an Edit control 1004, and in response, the platform has generated and caused displaying the popup menu 1006 of options 1008 to enhance the text, rewrite it in a personalized style, rewrite in a brand voice, rewrite for inclusivity, rewrite for professionalism, rewrite for a target social media application such as INSTAGRAM, change the length, or change the tone. In other embodiments, other options could be offered depending on the input signals and training of relevant machine-learning models. In an embodiment, each of the options 1008 corresponds to a different trained ML model that has been trained using training data corresponding to the source text and transformed text based on the specified option. In an embodiment, selecting a particular option 1008 signals the check 144C to re-execute a trained ML model, corresponding to the selected option, over the source content 130 and to re-generate updated presentation instructions to cause updating the GUI 1000 with different suggestions based on an output of the model. In this way, the user computer 102 can select, use, test, or experiment with different kinds of editing options associated with style, voice, tone, length, document content, and others.

[0105] FIG. 10B illustrates the graphical user interface of FIG. 10A in which an Autopilot control has been selected in response to user input. In response to the selection of Autopilot control 1010, all suggestions are automatically applied to the text, and the interface then steps through each suggestion, highlighting the suggestion and requesting the user to confirm that the change is acceptable. For example, check 144C can be programmed to generate, for each suggestion in a set of output suggestions 132, a popup panel 1012 near each particular suggestion 1018 and comprising a plurality of active links, widgets, or labels 1014. In an embodiment, each of the active links, widgets, or labels 1014 corresponds to accepting the suggestion (e.g., LOOKS GOOD), dismissing the suggestion (e.g., a REVERT link), or accessing other controls or tools.

[0106] In this manner, a text can be bulk modified automatically with all suggestions that the machine learning models have determined should apply, but the user can reject one or all after review. Autopilot control thus facilitates efficiency in writing. The Autopilot control can be based upon a personalized machine learning model that has learned which suggestions each user wants to have corrected as they type.

[0107] FIG. 11 illustrates an example of a suggestion leading to a user request to customize writing style features to cause future suggestions to be different. When a suggestion is presented, such as changing “relevant” to “suitable,” an embodiment can be programmed to provide a visual control in the suggestion interface to open a customization interface. In an embodiment, the customization interface is programmed with a plurality of selectable options corresponding to domains of customization, such as dictionary management, language, writing style, and other features.

[0108] In the example of FIG. 11, assume that source content 130 (not shown) has been entered using an application or extension (not shown), and evaluation using check 144C has generated presentation instructions to present an output suggestion 132 in a popup suggestion panel 1102. Panel 1102 shows a visual representation 1104 of the suggestion, an explanation, and an active link 1106 to access one or more controls. The panel 1102 also can comprise a plurality of signal controls 1108 to provide feedback to check 144C to cause removing or flagging of the suggestion.

[0109] In an embodiment, selecting the active link 1106 causes check 144C to transmit presentation instructions to show a customization panel 1110. A top portion of panel 1110 comprises a plurality of tab links 1114, in which a Writing Style control 1112 associated with a Writing Style domain has been selected. In an embodiment, the Writing Style domain has a plurality of categories 1116 of customization controls including Clarity, Correctness, Delivery, and Engagement; in the example, “All Categories” has been selected, as indicated by a visual emphasis. In response, check 144C is programmed to display a visual table 1121 comprising a plurality of rows 1118 corresponding to suggestion types, and each row is associated with a visibility toggle widget 1120 and a category label 1122.

[0110] The operation of suggestions in these categories can be controlled by selecting whether different suggestion types are visible, corresponding to rows 1118, via user input to a plurality of visibility toggle widgets 1120 in the interface. When the interface of FIG. 11 is open, user input from a user computer can select and change any of the toggle widgets and save the result. The platform is programmed to use the values of the toggle widgets as filters in determining whether to show suggestions.

[0111] 2.6 SECURITY CONTROLS

[0112] Since embodiments rely in part upon collecting feedback signals from individual user accounts or users as text is entered and suggestions are evaluated, as well as other context signals from user computers, groups, teams, or other sources, the platform is programmed with security controls to support a sense of user trust and conformance to user expectations, as well as to conform to legal regulations. Signals received from user accounts or user computers are stored in secure data repositories using encryption. Storage systems are minimized and access controls are programmed to ensure that only authorized applications can request and receive access. Embodiments can implement security controls of several kinds:

[0113] User controls. Includes obtaining consent to collect and programming to enforce opt-out requests.

[0114] Data retention. Includes programming the platform to automatically enforce data deletion timelines and to retain only data that has legitimate business value or user value.

[0115] Security. Includes programming the platform to implement access controls, secure data storage systems, secure data transfer, and only acceptable use of data.

[0116] Data processing. Includes programming the platform to implement data deidentification, anonymization, sampling, and aggregation.

[0117] 3. IMPLEMENTATION EXAMPLE - HARDWARE OVERVIEW

[0118] According to one embodiment, the techniques described herein are implemented by at least one computing device. The techniques may be implemented in whole or in part using a combination of at least one server computer and/or other computing devices that are coupled using a network, such as a packet data network. The computing devices may be hard-wired to perform the techniques or may include digital electronic devices such as at least one application- specific integrated circuit (ASIC) or field programmable gate array (FPGA) that is persistently programmed to perform the techniques or may include at least one general purpose hardware processor programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the described techniques. The computing devices may be server computers, workstations, personal computers, portable computer systems, handheld devices, mobile computing devices, wearable devices, body-mounted or implantable devices, smartphones, smart appliances, internetworking devices, autonomous or semi-autonomous devices such as robots or unmanned ground or aerial vehicles, any other electronic device that incorporates hard-wired and/or program logic to implement the described techniques, one or more virtual computing machines or instances in a data center, and/or a network of server computers and/or personal computers. [0119] FIG. 4 is a block diagram that illustrates an example computer system with which an embodiment may be implemented. In the example of FIG. 4, a computer system 400 and instructions for implementing the disclosed technologies in hardware, software, or a combination of hardware and software, are represented schematically, for example as boxes and circles, at the same level of detail that is commonly used by persons of ordinary skill in the art to which this disclosure pertains for communicating about computer architecture and computer systems implementations.

[0120] Computer system 400 includes an input/output (I/O) subsystem 402 which may include a bus and/or other communication mechanism(s) for communicating information and/or instructions between the components of the computer system 400 over electronic signal paths. The I/O subsystem 402 may include an I/O controller, a memory controller and at least one I/O port. The electronic signal paths are represented schematically in the drawings, for example as lines, unidirectional arrows, or bidirectional arrows.

[0121] At least one hardware processor 404 is coupled to I/O subsystem 402 for processing information and instructions. Hardware processor 404 may include, for example, a general- purpose microprocessor or microcontroller and/or a special-purpose microprocessor such as an embedded system or a graphics processing unit (GPU) or a digital signal processor or ARM processor. Processor 404 may comprise an integrated arithmetic logic unit (ALU) or may be coupled to a separate ALU.

[0122] Computer system 400 includes one or more units of memory 406, such as a main memory, which is coupled to I/O subsystem 402 for electronically digitally storing data and instructions to be executed by processor 404. Memory 406 may include volatile memory such as various forms of random-access memory (RAM) or other dynamic storage device. Memory 406 also may be used for storing temporary variables or other intermediate information during the execution of instructions to be executed by processor 404. Such instructions, when stored in non-transitory computer-readable storage media accessible to processor 404, can render computer system 400 into a special-purpose machine that is customized to perform the operations specified in the instructions.

[0123] Computer system 400 further includes non-volatile memory such as read-only memory (ROM) 408 or other static storage devices coupled to I/O subsystem 402 for storing information and instructions for processor 404. The ROM 408 may include various forms of programmable ROM (PROM) such as erasable PROM (EPROM) or electrically erasable PROM (EEPROM). A unit of persistent storage 410 may include various forms of non-volatile RAM (NVRAM), such as FLASH memory, solid-state storage, magnetic disk, or optical disks such as CD-ROM or DVD-ROM and may be coupled to I/O subsystem 402 for storing information and instructions. Storage 410 is an example of a non-transitory computer-readable medium that may be used to store instructions and data which when executed by the processor 404 cause performing computer-implemented methods to execute the techniques herein.

[0124] The instructions in memory 406, ROM 408 or storage 410 may comprise one or more sets of instructions that are organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs including mobile apps. The instructions may comprise an operating system and/or system software; one or more libraries to support multimedia, programming or other functions; data protocol instructions or stacks to implement TCP/IP, HTTP or other communication protocols; file format processing instructions to parse or render files coded using HTML, XML, JPEG, MPEG or PNG; user interface instructions to render or interpret commands for a graphical user interface (GUI), command-line interface or text user interface; application software such as an office suite, internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games or miscellaneous applications. The instructions may implement a web server, web application server, or web client. The instructions may be organized as a presentation layer, application layer, and data storage layer such as a relational database system using structured query language (SQL) or no SQL, an object store, a graph database, a flat file system, or other data storage.

[0125] Computer system 400 may be coupled via I/O subsystem 402 to at least one output device 412. In one embodiment, output device 412 is a digital computer display. Examples of a display that may be used in various embodiments include a touchscreen display or a lightemitting diode (LED) display or a liquid crystal display (LCD) or an e-paper display. Computer system 400 may include other type(s) of output devices 412, alternatively or in addition to a display device. Examples of other output devices 412 include printers, ticket printers, plotters, projectors, sound cards or video cards, speakers, buzzers or piezoelectric devices or other audible devices, lamps or LED or LCD indicators, haptic devices, actuators or servos.

[0126] At least one input device 414 is coupled to I/O subsystem 402 for communicating signals, data, command selections, or gestures to processor 404. Examples of input devices 414 include touch screens, microphones, still and video digital cameras, alphanumeric and other keys, keypads, keyboards, graphics tablets, image scanners, joysticks, clocks, switches, buttons, dials, slides, and/or various types of sensors such as force sensors, motion sensors, heat sensors, accelerometers, gyroscopes, and inertial measurement unit (IMU) sensors and/or various types of transceivers such as wireless, such as cellular or Wi-Fi, radio frequency (RF) or infrared (IR) transceivers and Global Positioning System (GPS) transceivers.

[0127] Another type of input device is a control device 416, which may perform cursor control or other automated control functions such as navigation in a graphical interface on a display screen, alternatively or in addition to input functions. Control device 416 may be a touchpad, a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 404 and for controlling cursor movement on display 412. The input device may have at least two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane. Another type of input device is a wired, wireless, or optical control device such as a joystick, wand, console, steering wheel, pedal, gearshift mechanism or other type of control device. An input device 414 may include a combination of multiple different input devices, such as a video camera and a depth sensor. [0128] In another embodiment, computer system 400 may comprise an Internet of things (loT) device in which one or more of the output device 412, input device 414, and control device 416 are omitted. Or, in such an embodiment, the input device 414 may comprise one or more cameras, motion detectors, thermometers, microphones, seismic detectors, other sensors or detectors, measurement devices or encoders, and the output device 412 may comprise a special-purpose display such as a single-line LED or LCD display, one or more indicators, a display panel, a meter, a valve, a solenoid, an actuator or a servo.

[0129] When computer system 400 is a mobile computing device, input device 414 may comprise a global positioning system (GPS) receiver coupled to a GPS module that is capable of triangulating to a plurality of GPS satellites, determining and generating geo-location or position data such as latitude-longitude values for a geophysical location of the computer system 400. Output device 412 may include hardware, software, firmware, and interfaces for generating position reporting packets, notifications, pulse or heartbeat signals, or other recurring data transmissions that specify a position of the computer system 400, alone or in combination with other application-specific data, directed toward host 424 or server 430.

[0130] Computer system 400 may implement the techniques described herein using customized hard-wired logic, at least one ASIC or FPGA, firmware and/or program instructions or logic which when loaded and used or executed in combination with the computer system causes or programs the computer system to operate as a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 400 in response to processor 404 executing at least one sequence of at least one instruction contained in main memory 406. Such instructions may be read into main memory 406 from another storage medium, such as storage 410. Execution of the sequences of instructions contained in main memory 406 causes processor 404 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

[0131] The term “storage media” as used herein refers to any non-transitory media that store data and/or instructions that cause a machine to operation in a specific fashion. Such storage media may comprise non-volatile media and/or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage 410. Volatile media includes dynamic memory, such as memory 406. Common forms of storage media include, for example, a hard disk, solid state drive, flash drive, magnetic data storage medium, any optical or physical data storage medium, memory chip, or the like. [0132] Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that comprise a bus of VO subsystem 402. Transmission media can also take the form of acoustic or light waves, such as those generated during radio- wave and infrared data communications.

[0133] Various forms of media may be involved in carrying at least one sequence of at least one set of instructions to processor 404 for execution. For example, the instructions may initially be carried on a magnetic disk or solid-state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a communication link such as a fiber optic or coaxial cable or telephone line using a modem. A modem or router local to computer system 400 can receive the data on the communication link and convert the data to a format that can be read by computer system 400. For instance, a receiver such as a radio frequency antenna or an infrared detector can receive the data carried in a wireless or optical signal and appropriate circuitry can provide the data to VO subsystem 402 such as place the data on a bus. I/O subsystem 402 carries the data to memory 406, from which processor 404 retrieves and executes the instructions. The instructions received by memory 406 may optionally be stored on storage 410 either before or after execution by processor 404.

[0134] Computer system 400 also includes a communication interface 418 coupled to bus 402. Communication interface 418 provides a two-way data communication coupling to network link(s) 420 that are directly or indirectly connected to at least one communication networks, such as a network 422 or a public or private cloud on the Internet. For example, communication interface 418 may be an Ethernet networking interface, integrated- services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of communications line, for example an Ethernet cable or a metal cable of any kind or a fiber-optic line or a telephone line. Network 422 broadly represents a local area network (LAN), wide-area network (WAN), campus network, internetwork or any combination thereof. Communication interface 418 may comprise a LAN card to provide a data communication connection to a compatible LAN, or a cellular radiotelephone interface that is wired to send or receive cellular data according to cellular radiotelephone wireless networking standards, or a satellite radio interface that is wired to send or receive digital data according to satellite wireless networking standards. In any such implementation, communication interface 418 sends and receives electrical, electromagnetic or optical signals over signal paths that carry digital data streams representing various types of information.

[0135] Network link 420 typically provides electrical, electromagnetic, or optical data communication directly or through at least one network to other data devices, using, for example, satellite, cellular, Wi-Fi, or BLUETOOTH technology. For example, network link 420 may provide a connection through network 422 to a host computer 424.

[0136] Furthermore, network link 420 may provide a connection through network 422 or to other computing devices via internetworking devices and/or computers that are operated by an Internet Service Provider (ISP) 426. ISP 426 provides data communication services through a worldwide packet data communication network represented as Internet 428. A server computer 430 may be coupled to Internet 428. Server 430 broadly represents any computer, data center, virtual machine, or virtual computing instance with or without a hypervisor or computer executing a containerized program system such as DOCKER or KUBERNETES. Server 430 may represent an electronic digital service that is implemented using more than one computer or instance and that is accessed and used by transmitting web services requests, uniform resource locator (URL) strings with parameters in HTTP payloads, API calls, app services calls, or other sendee calls. Computer system 400 and server 430 may form elements of a distributed computing system that includes other computers, a processing cluster, server farm or other organization of computers that cooperate to perform tasks or execute applications or services. Server 430 may comprise one or more sets of instructions that are organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs including mobile apps. The instructions may comprise an operating system and/or system software; one or more libraries to support multimedia, programming or other functions; data protocol instructions or stacks to implement TCP/IP, HTTP or other communication protocols; file format processing instructions to parse or render files coded using HTML, XML, JPEG, MPEG or PNG; user interface instructions to render or interpret commands for a graphical user interface (GUI), command- line interface or text user interface; application software such as an office suite, internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games or miscellaneous applications. Server 430 may comprise a web application server that hosts a presentation layer, application layer, and data storage layer such as a relational database system using a structured query language (SQL) or no SQL, an object store, a graph database, a flat file system or other data storage.

[0137] Computer system 400 can send messages and receive data and instructions, including program code, through the network(s), network link 420, and communication interface 418. In the Internet example, a server 430 might transmit a requested code for an application program through Internet 428, ISP 426, local network 422, and communication interface 418. The received code may be executed by processor 404 as it is received, and/or stored in storage 410, or other non-volatile storage for later execution.

[0138] The execution of instructions as described in this section may implement a process in the form of an instance of a computer program that is being executed, and consisting of program code and its current activity. Depending on the operating system (OS), a process may be made up of multiple threads of execution that execute instructions concurrently. In this context, a computer program is a passive collection of instructions, while a process may be the actual execution of those instructions. Several processes may be associated with the same program; for example, opening up several instances of the same program often means more than one process is being executed. Multitasking may be implemented to allow multiple processes to share processor 404. While each processor 404 or core of the processor executes a single task at a time, computer system 400 may be programmed to implement multitasking to allow each processor to switch between tasks that are being executed without having to wait for each task to finish. In an embodiment, switches may be performed when tasks perform input/output operations, when a task indicates that it can be switched, or on hardware intermpts. Time-sharing may be implemented to allow fast response for interactive user applications by rapidly performing context switches to provide the appearance of concurrent execution of multiple processes simultaneously. In an embodiment, for security and reliability, an operating system may prevent direct communication between independent processes, providing strictly mediated and controlled inter-process communication functionality.

[0139] In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent scope of the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction.