CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Pat. App. No. 63/252,231 filed on October 5, 2021, the entire contents of which are hereby incorporated by reference.

FIELD

[0002] The present disclosure generally relates to devices, systems, and methods related to identifying the impact (e.g., cognitive or motivational impact) on audiences of certain content creation constructs, including framing styles, rhetoric, topics, and the like, potentially also including content constructs that take the form of images.

BACKGROUND

[0003] There remains a need for improved techniques to identify the likely impact of content on particular audiences. Words common to a specific domain if used over time, for example, can serve to activate implicit associations and related behaviors of which the creator and the audience may be unaware. As a result, content constructed for one explicit purpose may cue implicit associations that counter or undercut that purpose. Because these cognitive and behavioral processes can be habitual and subconscious, there remains a need for a diagnostic and remedial process, e.g., to identify and annotate a variety of content constructs and their likely cognitive, behavioral, and other effects to creators, which can enable creators and the like to better align their intended motivational and other effects with their likely ones.

SUMMARY

[0004] The present teachings include content processing techniques (e.g., via a content processing platform hosted by a remote computing resource) to analyze source content for determining an impact of the content on an audience. Specifically, a content processing platform may use a weighted pattern matching machine learning classification algorithm (WPMMLCA) to identify and measure the impact of patterns uncovered within the source content. The patterns may be deliberately constructed to achieve a goal, e.g., to motivate an audience. The WPMMLCA may also or instead identify habitual/unintentional content patterns that can impact the audience. Identified patterns may relate to, for example, narrative frame (a cognitive shortcut that cues often implicit associations), context, lexicon, concept, language, structure, sequence, sentiment, and/or intended audience. The content processing platform may further provide recommendations and suggestions to alter/remediate source content based upon likely cognitive, motivational and other impact(s) of the subject content and characteristics of an audience.

[0005] In an aspect, a method disclosed herein may include: receiving source content for presentation to an audience; determining one or more weights for use by a weighted pattern matching machine learning classification algorithm (WPMMLCA) based on at least a context related to the source content and the audience, the one or more weights related to a measurable impact of content; normalizing the source content by annotating with one or more labels; analyzing the normalized source content including the one or more labels using the WPMMLCA to identify one or more patterns related to a narrative frame and/or lexicon of at least a portion of the source content; and, determining, using the WPMMLCA and the one or more weights, the likely measurable impact of the portion of the source content based on the one or more patterns related to the narrative frame and/or lexicon.

[0006] In an aspect, a computer program product for determining impact of source content on an audience may include computer executable code embodied in a non-transitory computer readable medium that, when executing on one or more computing devices, performs the steps of: receiving source content for presentation to an audience; determining one or more weights for use by a weighted pattern matching machine learning classification algorithm (WPMMLCA) based on at least a context related to the source content and the audience, the one or more weights related to a measurable impact of content; normalizing the source content by annotating with one or more labels; analyzing the normalized source content including the one or more labels using the WPMMLCA to identify one or more patterns related to a narrative frame of at least a portion of the source content; and, determining, using the WPMMLCA and the one or more weights, the measurable impact of the portion of the source content based on the one or more patterns related to the narrative frame.

[0007] In an aspect, a system disclosed herein may include: a data network; a user device coupled to the data network; and a remote computing resource coupled to the data network and accessible to the user device through the data network. The remote computing resource may include a processor and a memory, the memory storing code executable by the processor to perform the steps of: receiving source content from the user over the data network, the source content for presentation to an audience; determining one or more weights for use by a weighted pattern matching machine learning classification algorithm (WPMMLCA) based on at least a context related to the source content and the audience, the one or more weights related to a measurable impact of content; normalizing the source content by annotating with one or more labels; analyzing the normalized source content including the one or more labels using the WPMMLCA to identify one or more patterns related to a narrative frame of at least a portion of the source content; and, determining, using the WPMMLCA and the one or more weights, the measurable impact of the portion of the source content based on the one or more patterns related to the narrative frame.

[0008] Implementations — such as one or more of the aforementioned method, computer program product, and system — may include one or more of the following features. The measurable impact may include at least one of a motivational impact, an emotional impact, and a cognitive impact. The present teachings may further include providing one or more measurables related to the measurable impact of the portion of the source content. The one or more measurables may include at least one of a quantity and a type of one or more narrative frames included in the portion of the source content. The one or more patterns may be related to at least one of consistency, tone, sequence, lexicon, argumentation, and ideation. At least one of the one or more labels may be based on a grammar structure of the source content that defines one or more of a person, a noun, and an actor in relation to a verb. The present teachings may further include: associating a confidence score to at least one of the one or more labels, and using the confidence score as a factor when analyzing the normalized source content. The present teachings may further include triggering a notification for review of the at least one of the one or more labels when the associated confidence score is below a predetermined threshold. The present teachings may further include analyzing the audience to produce a motivation score related to a magnitude of motivation for one or more members of the audience, the motivation score at least in part used as a criterion for determining the one or more weights. Analyzing the audience may include identifying one or more of demographic information and a political affiliation thereof. The motivation score may be based on feedback from the one or more members of the audience. The feedback may be based on an impression formed related to predetermined power-framed language. The present teachings may further include comparing the context and the audience to one or more predetermined contexts and audiences included in a matrix, and selecting the one or more weights at least in part based on a similarity of the context and the audience to the one or more predetermined contexts and audiences included in the matrix. The present teachings may further include quantifying the narrative frame. The present teachings may further include producing a report including the quantified narrative frame. The present teachings may further include analyzing the narrative frame and providing one or more recommendations to alter the narrative frame. The one or more recommendations may include alternative content for the portion of the source content. The alternative content may include alternative text. The alternative content may include edits selected to alter the measurable impact of the portion of the source content. The edits may include a new sequencing of text in the portion of the source content. An output of the WPMMLCA may include a table including the one or more labels and portions of the source content annotated with the one or more labels. The present teachings may further include selecting the one or more labels for normalizing the source content, where selection of the one or more labels is based on at least one of the context and the audience. The present teachings may further include associating a label of the one or more labels to a different label of the one or more labels based on at least one of a topic, a frame, an identity, a sector, and a sample size. Annotating with the one or more labels may utilize natural language processing. Normalizing the source content may further include one or more of reformatting and word stemming. The present teachings may further include parsing the source content based on an author thereof. The present teachings may further include parsing the source content based on a narrative perspective of the source content. The context may include one or more of a goal of the source content, a topic covered in the source content, and an attribute of an originator of the source content. The context may include the attribute of the originator of the source content, where the attribute includes at least one of branding, standing, a domain or sector related to the originator, and demographic information. The source content may include one or more of text, an image, audio, and a video. The source content may include one or more of an email, a webpage, a letter, a memorandum, a newsletter, an article, a book, a donation appeal, and promotional material. Receiving the source content may include receiving metadata related to the source content. The metadata may include one or more of a date, a time, a description, a hyperlink, a title, an intended audience, an intended effect, an open rate, and an unsubscribe rate.

[0009] These and other features, aspects, and advantages of the present teachings will become better understood with reference to the following description, examples, and appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The foregoing and other objects, features and advantages of the devices, systems, and methods described herein will be apparent from the following description of particular embodiments thereof, as illustrated in the accompanying drawings. The drawings are not necessarily to scale or exhaustive, emphasis instead being placed upon illustrating the principles and functions of the devices, systems, and methods described herein. In the drawings, like reference numerals generally identify corresponding elements.

[0011] Fig. 1 shows a system for determining impact of source content on an audience, in accordance with a representative embodiment.

[0012] Fig. 2 is a flow chart of a method for determining an impact of source content on an audience, in accordance with a representative embodiment. [0013] Fig. 3 is an example workflow for determining impact of source content on an audience, in accordance with a representative embodiment.

[0014] Fig. 4 is an example workflow for determining impact of source content on an audience, in accordance with a representative embodiment.

[0015] Fig. 5 is an example workflow for determining impact of source content on an audience, in accordance with a representative embodiment.

[0016] Fig. 6 is an example workflow for determining impact of source content on an audience, in accordance with a representative embodiment.

[0017] Fig. 7 is an example workflow for determining impact of source content on an audience, in accordance with a representative embodiment.

[0018] Fig. 8 is an example workflow for determining impact of source content on an audience, in accordance with a representative embodiment.

[0019] Fig. 9 is an example workflow for determining impact of source content on an audience, in accordance with a representative embodiment.

DETAILED DESCRIPTION

[0020] The embodiments will now be described more fully hereinafter with reference to the accompanying figures, in which preferred embodiments are illustrated. The foregoing may, however, be embodied in many different forms and should not be construed as limited to the illustrated embodiments set forth herein. Rather, these illustrated embodiments are provided so that this disclosure will convey the scope to those skilled in the art.

[0021] All documents mentioned herein are hereby incorporated by reference in their entirety. References to items in the singular should be understood to include items in the plural, and vice versa, unless explicitly stated otherwise or clear from the text. Grammatical conjunctions are intended to express any and all disjunctive and conjunctive combinations of conjoined clauses, sentences, words, and the like, unless otherwise stated or clear from the context. Thus, the term “or” should generally be understood to mean “and/or” and so forth.

[0022] Recitation of ranges of values herein are not intended to be limiting, referring instead individually to any and all values falling within the range, unless otherwise indicated herein, and each separate value within such a range is incorporated into the specification as if it were individually recited herein. The words “about,” “approximately” or the like, when accompanying a numerical value, are to be construed as indicating a deviation as would be appreciated by one of ordinary skill in the art to operate satisfactorily for an intended purpose. Similarly, words of approximation such as “about,” “approximately,” “likely,” or “substantially” when used in reference to physical characteristics, should be understood to contemplate a range of deviations that would be appreciated by one of ordinary skill in the art to operate satisfactorily for a corresponding use, function, purpose, or the like. Ranges of values and/or numeric values are provided herein as examples only, and do not constitute a limitation on the scope of the described embodiments. Where ranges of values are provided, they are also intended to include each value within the range as if set forth individually, unless expressly stated to the contrary. The use of any and all examples, or exemplary language (“e.g.,” “such as,” or the like) provided herein, is intended merely to better illuminate the embodiments and does not pose a limitation on the scope of the embodiments. No language in the specification should be construed as indicating any unclaimed element as essential to the practice of the embodiments.

[0023] In the following description, it is understood that terms such as “first,” “second,” “top,” “bottom,” “up,” “down,” and the like, are words of convenience and are not to be construed as limiting terms unless specifically stated to the contrary.

[0024] In general, the devices, systems, and methods disclosed herein relate to analysis of source content to determine one or more impacts of the source content on an audience. Specifically, in certain aspects, one or more weighted pattern matching machine learning classification algorithms (WPMMLCAs) may be used to identify and measure the impact of patterns within at least a portion of the source content. Patterns identified or analyzed by the WPMMLCA may relate to a narrative frame of at least a portion of the source content. Recommendations and/or suggestions may be provided to alter at least a portion of the source content based upon a measured impact of the analyzed content and characteristics of an audience, such as a target audience.

[0025] It will be generally understood that, the content processing approaches described herein may involve the application of cognitive and motivational science to better understand the likely impact that certain narrative framing styles, contexts, rhetoric, topics, etc. may have on diverse audiences, creators, and others. One example of an impact on an audience is the motivation to act. The motivational impact of content may be modified by a host of factors, including sentiment. Thus, the present teachings may involve utilizing research that characterizes contemporary social dynamics and hypothetical causal and correlative links with respect to motivation (and/or other drivers), in order to automatically detect one or more likely impacts of content analyzed in view of this research.

[0026] The devices, systems, and methods described herein may be used to systematically observe, measure, and explore patterns within content, and to demystify the effects that a certain narrative frame and language may have on audiences to whom it is addressed or presented. Thus, the present teachings may systematize the detection of patterns in source material for determining an impact thereof.

[0027] Absent the teachings described in the present disclosure, content creators rely on content construction processes that can often include frames and other content characteristics common to a given domain, the habitual use of which can become subconscious and/or autonomic, and may entail conceptual dissonances or adverse emotional and motivational effects likely to be inconsistent with the content creators’ conscious intent. So for example, a content creator touting the power of women, may unintentionally or autonomically render that power in a common frame or familiar lexicon which, by framing that power in the negative, actually undercuts it and thereby may occasion measurably adverse cognitive and/or motivational effects. For example, the phrase “women are disproportionately underrepresented in the U.S. Congress” — while a familiar narrative formula in created content in this domain — can in effect reinforce (rather than counter) women’s implied powerlessness and depress engagement.

[0028] The teachings set forth herein generally address autonomia and its unintended motivational and other consequences in content creation across different domains by varied creators for diverse audiences. To this end, the present teachings may include a process by which narrative frames common to a given domain can be identified, their conceptual/lexical content annotated, their underlying sentiments, associations, or other referents measured, and their likely motivational effects mapped in relation to standard scientific measures. In an aspect, the process aims to put content creators across domains in the driver’s seat where construction is concerned, ensuring they can more consciously deploy framing, linguistic or other techniques, and more actively ensure their outputs align with their intent conceptually or in terms of desired motivational outcomes or otherwise.

[0029] It will be understood that the terms “narrative frame” or “narrative framing” (and the like) are used herein to describe an underlying contextual environment or frame of reference of content. A narrative frame (e.g., powerless women or a photo of powerless women) can cue a set of implicit or explicit associations in the brains and bodies of both creators of content and consumers of that content. A narrative frame may be a general context of the content, such as a theme, tone, emotion, setting, conceptualization, and so forth. Thus, a narrative frame may generally include how a set of content constructs impact someone producing the content and/or how content constructs interact with someone consuming the content. In this context, and unless expressly stated to the contrary, it will be understood that a “consumer” of content includes someone that reads, watches, sees, reviews, and the like, the content; similarly, “consuming” the content may include reading, reviewing, watching, listening, seeing, hearing, understanding, and so on, the content. A narrative frame may also or instead include a manner of approaching or rendering a topic and/or issue. A narrative frame may also or instead include how content is presented, e.g., in written, oral, or imagistic format in a manner that creates a measurable impact on someone consuming the content. That is, a narrative frame may include rhetorical devices used to convey information emphasizing specific elements of that information. As such, a narrative frame may include a pattern such as a linguistic and/or imagistic pattern that can create a likely impact, and/or a narrative frame may be detectable through identification of such a pattern.

[0030] A narrative frame may be intentionally constructed to create an effect or impression on an audience. For example, and generally in the context of embodiments of the present teachings, source content may be created with a narrative frame or other construct(s) intended to motivate an audience.

[0031] An example of a narrative frame that includes a negative association is “people of color are less likely to get bank loans.” This statement includes a deficit frame. If revised, such a deficit frame’s potentially adverse motivational impact may be reduced or removed, such as by stating, “banks are less likely to give loans to people of color.” The re-frame in this example redresses the association of the deficit (less likely) with the “people of color” identity and associates it instead with a judgment made by the bank.

[0032] By way of example of narrative frames, an “asset orientation” versus a “deficit orientation” will be described in the context of community engagement. This example can be found in “Comparison Between Asset and Deficit Based Approaches,” Engaged Scholar Module 4, University of Memphis, available at https://www.memphis.edu/ess/module4/page3.php, and hereby incorporated by reference. The example goes as follows — often, people begin work with communities by “needs assessments” that identify problems and focus on weaknesses of a community. Unfortunately, this methodology includes a deficit frame and reinforces the idea that the “community” has or is “the problem” and thus should be the focus of attention. That is, the continual focus on the “problem” may lead to individuals feeling as if “all they have are problems” or communities believing that “all they have are deficits.” As a result of this habitual negative mindset, a widespread belief or stereotype may start to develop about the individuals, organization, or community being depressed, dysfunctional, or just filled with problems. This negative or “deficit” mindset can make it difficult to get people motivated to take action to make positive change. Thus, the narrative frame in the form of a deficit frame can affect motivation.

[0033] Furthering the community engagement example from the University of Memphis described above, an asset based approach may — in contrast — foster engagement by shifting the focus from “what’s wrong with us” to “what’s right with us.” An “asset” or “powerframed” approach assumes that, even though there may be problems with or in a person or community, sometimes very serious problems, untapped resources and capacities also exist therein and are inherent in every individual, organization, or community. These “plusses” or “assets” can be put to use to improve current conditions — i.e., use of an asset frame can be motivating. To discover, affirm, and/or cue such (or patterns of such) underutilized assets and untapped potential are hallmarks of an asset based approach to community work.

[0034] Additionally, or alternatively, a narrative frame may be unintentionally deployed, such as when content creation cues a pervasive and/or a predominant cultural narrative or set of frequent associations. For example, content created regarding who has power in society may unintentionally cue an underlying or unconscious association of power with a particular gender or race, in the case of the United States, for example, power is most affirmatively associated with maleness and whiteness. The cognitive and motivational weight of such commonplace narrative frames may be subtle; they may operate autonomically on both a creator and an audience. By way of example, a dominant or common narrative frame that repeatedly affirmatively associates maleness, whiteness, and powerfulness may subconsciously (or consciously) evoke implicit or explicit negative associations between non-white men and women to relative degrees along perceived racial characteristics and power with correspondingly negative effects. Sources for further exploration regarding the status quo association of white men with power include Warner, Judith, et al., “The Women’s Leadership Gap,” Center for American Progress (November 20, 2018) available at https://www.americanprogress.org/issues/women/reports/2018/l l/20/461273/womens- leadership-gap-2/, and “Women in the Workplace 2021,” McKinsey & Company (September 27, 2021) available at https://www.mckinsey.com/featured-insights/diversity-and- inclusion/women-in-the-workplace.

[0035] Additional nonexclusive examples of a narrative frame include an obstacle frame, a disparity frame, a stereotype, a power frame, and a variety of modifiers or contextualizers that attach to these standard frames. An obstacle frame may cast the challenges, suffering, and/or suppression that a person or group may face as part of that person or group’s identity, rather than an adverse context faced by people of that identity. A disparity frame may be used to show that a person or group is disproportionately underprivileged, oppressed, and/or otherwise subject to injustice. Disparity frames, while potentially factually sound, can emotionally reinforce negative stereotypes of a person or group as being, for example, “disempowered” or “victimized.” Stereotypes — the most obvious of which can typically be avoided in the creation of most content, e.g., through adherence to social norms and civil behavior — may still find use in subtler forms. For example, subtle stereotypes can be presented as compliments, while surreptitiously invoking harmful associations or the like. Power frames center around a person or group’s strengths and achievements, and thus it will be understood that this type of framing may be desirable — and, as such, more power framed content be an output of the present teachings. Typically, power framing does not obscure the problems a person or group faces, but ensures that those barriers do not through repeat association come to define the person or group. Power framing may unbind oppression from identity and may move an audience to see a person or group’s power as natural, not anomalous. This can be particularly advantageous in cognitive and behavioral terms when the person or group being invoked is not typically presented in a power framed manner, which is unfortunately often the case for certain groups.

[0036] Fig. 1 illustrates a system for determining impact of source content on an audience, in accordance with a representative embodiment. In general, the system 100 may include a networked environment where a data network 102 interconnects a plurality of participating devices and/or users in a communicating relationship. The participating devices may, for example, include any number of user devices 110, remote computing resources 120, and other resources 130.

[0037] The data network 102 may be any network(s) or intemetwork(s) suitable for communicating data and information among participants in the system 100. This may include public networks such as the Internet, private networks, telecommunications networks such as the Public Switched Telephone Network or cellular networks using third generation (e.g., 3G or IMT-2000), fourth generation (e.g., LTE (E-UTRA) or WiMAX- Advanced (IEEE 802.16m)), fifth generation (e.g., 5G), and/or other technologies, as well as any of a variety of corporate area or local area networks and other switches, routers, hubs, gateways, and the like that might be used to carry data among participants in the system 100.

[0038] Each of the participants of the data network 102 may include a suitable network interface comprising, e.g., a network interface card, which term is used broadly herein to include any hardware (along with software, firmware, or the like to control operation of same) suitable for establishing and maintaining wired and/or wireless communications. The network interface card may include without limitation a wired Ethernet network interface card (“NIC”), a wireless 802.11 networking card, a wireless 802.11 USB device, or other hardware for wired or wireless local area networking. The network interface may also or instead include cellular network hardware, wide area wireless network hardware or any other hardware for centralized, ad hoc, peer-to-peer, or other radio communications that might be used to connect to a network and carry data. In another aspect, the network interface may include a serial or USB port to directly connect to a local computing device such as a desktop computer that, in turn, provides more general network connectivity to the data network 102. [0039] The user devices 110 may include any devices within the system 100 operated by one or more users 101 for practicing the techniques as contemplated herein. Specifically, the user devices 110 may include any device for creating, preparing, editing, receiving, and/or transmitting (e.g., over the data network 102) source content 140 and information related thereto such as a context 142, a narrative frame 144, a narrative perspective 146, metadata 148, and other content 149, as well as any outputs 162 in the system 100. Similarly, the user devices 110 may include any device for creating, preparing, editing, receiving, and/or transmitting (e.g., over the data network 102) other data or files in the system 100, such as normalized source content 150, a label 152, a pattern 154, output 162, and so on as described herein. The user devices 110 may also or instead include any device for managing, monitoring, or otherwise interacting with tools, platforms, and devices included in the systems and techniques contemplated herein. Thus, it will be understood that the user devices 110 may include client devices of a client receiving a service including the present teachings, and/or managerial or administrator devices by a provider of a service including the present teachings, where, for example, such managerial or administrator devices provide a user 101 (who, in this example, is a manager, administrator, employee, service provider, or the like) access to backend systems such as the remote computing resource 120. And, in general, the user devices 110 may be coupled to the data network 102, e.g., for interaction with one or more other participants in the system 100.

[0040] By way of example, the user devices 110 may include one or more desktop computers, laptop computers, network computers, tablets, mobile devices, portable digital assistants, messaging devices, cellular phones, smart phones, portable media or entertainment devices, or any other computing devices that can participate in the system 100 as contemplated herein. As discussed above, the user devices 110 may include any form of mobile device, such as any wireless, battery-powered device, that might be used to interact with the networked system 100. It will also be appreciated that one of the user devices 110 may coordinate related functions (e.g., analyzing an audience 170, determining weights 156, and so on) as they are performed by another entity such as one of the remote computing resources 120 or other resources 130.

[0041] Each user device 110 may generally provide a user interface, such as any of the user interfaces described herein or otherwise known in the art. The user interface may be maintained by a locally executing application on one of the user devices 110 that receives data from, e.g., the remote computing resources 120 or other resources 130. In other embodiments, the user interface may be remotely served and presented on one of the user devices 110, such as where a remote computing resource 120 or other resource 130 includes a web server that provides information through one or more web pages or the like that can be displayed within a web browser or similar client executing on one of the user devices 110. The user interface may in general create a suitable visual presentation for user interaction on a display device of one of the user devices 110, and provide for receiving any suitable form of user input including, e.g., input from a keyboard, mouse, touchpad, touch screen, hand gesture, or other use input device(s).

[0042] A user 101 may be an individual and/or an organization. A user 101 may include a creator of the source content 140 or someone associated with an organization or the like that created the source content 140. A user 101 may also or instead be affiliated with, or otherwise have an interest in, the source content 140. A user 101 as described in the context of the system 100 may also or instead include a person or entity associated with a provider of a service that includes the present teachings, such as an administrator or the like.

[0043] The audience 170 is described in further detail below and elsewhere herein, but will be understood to generally include any consumer or recipient, or would-be consumer or recipient, of the source content 140 and/or output 162 of the system 100, which may include revised content that is derived from analysis of the source content 140. The audience 170 may be an intended audience and/or an unintended or unknown audience of the source content 140. By way of example, the source content 140 may include promotional material intended to generate funds for a user 101, such as donations for a non-profit organization or the like. Continuing with this example, in this manner, the audience 170 could include members of the organization, previous donation providers, the general public or a subset thereof, and so on.

[0044] The remote computing resources 120 may include, or otherwise be in communication with, a processor 122 and a memory 124, where the memory 124 stores code executable by the processor 122 to perform various techniques of the present teachings. More specifically, a remote computing resource 120 may be coupled to the data network 102 and accessible to the user device 110 through the data network 102, where the remote computing resource 120 includes a processor 122 and a memory 124, where the memory 124 stores code executable by the processor 122 to perform the steps of a method according to the present teachings.

[0045] The remote computing resources 120 may also or instead include data storage, a network interface, and/or other processing circuitry. In the following description, where the functions or configuration of a remote computing resource 120 are described, this is intended to include corresponding functions or configuration (e.g., by programming) of a processor 122 of the remote computing resource 120, or a processor 122 in communication with the remote computing resource 120. In general, the remote computing resources 120 (or processors 122 thereof or in communication therewith) may perform a variety of processing tasks related to determining the impact of source content 140 on an audience 170 as discussed herein. For example, the remote computing resources 120 may manage information received from one or more of the user devices 110, and provide related supporting functions such as normalizing the source content 140, storing and/or implementing the WPMMLCA 160, communicating with other resources 130, generating, receiving, storing, and/or transmitting data, and the like. The remote computing resources 120 may also or instead include backend algorithms that react to actions performed by a user 101 at one or more of the user devices 110. The backend algorithms may also or instead be located elsewhere in the system 100, and may for example include the WPMMLCA 160.

[0046] The remote computing resources 120 may also or instead include a web server or similar front end that facilitates web-based access by the user devices 110 to the capabilities of the remote computing resource 120 or other components of the system 100. A remote computing resource 120 may also or instead communicate with other resources 130 in order to obtain information for providing to a user 101 through a user interface on the user device 110, or otherwise. Where the user 101 specifies certain criteria for source content processing and the like, this information may be used by a remote computing resource 120 (and any associated algorithms, such as the WPMMLCA 160) to access other resources 130. Additional processing may be usefully performed in this context such as recommending certain data processing operations and techniques.

[0047] A remote computing resource 120 may also or instead maintain, or otherwise be in communication with, a database 126 of content such as the WPMMLCA 160 and/or code associated therewith, source content 140, normalized source content 150, labels 152, patterns 154, outputs 162 of the present teachings (e.g., a report, a recommendation, revised content, and so on), rules or settings for implementation in the present teachings, and so on. For example, in certain aspects, annotated content (e.g., normalized source content 150) and raw content (e.g., the source content 140) may be stored in a database 126 such as a relational database, where such a database can be used for training the WPMMLCA 160 or the like. For example, training data stored in the database 126 may include raw content and annotations that are segmented into training sets and test sets, where a training process can be run against the training sets to test their accuracy with the test sets, and where the results of the training and testing can be reviewed. The database 126 may also or instead include computer executable code configured to implement any of the computer implemented methods as described herein.

[0048] The system 100 may further include an interface for users 101 at the user devices 110, or another participant in the system 100 such as an administrator, to utilize the content of such a database 126. Thus, in one aspect, a remote computing resource 120 may include a database 126 including an WPMMLCA 160, and the remote computing resource 120 may act as a server that provides a platform for selecting, configuring, and using a WPMMLCA 160, and/or providing supporting services related thereto.

[0049] A remote computing resource 120 may also or instead be configured to manage access to certain content (e.g., for an enterprise associated with a user 101 of the user device 110). In one aspect, a remote computing resource 120 may manage access to a component of the system 100 by a user device 110 according to input from a user 101.

[0050] The other resources 130 may include any resources that may be usefully employed in the devices, systems, and methods as described herein. For example, the other resources 130 may include without limitation other data networks, human actors (e.g., programmers, researchers, annotators, editors, analysts, and so forth), sensors (e.g., audio or visual sensors), data mining tools, computational tools, data monitoring tools, and so forth. The other resources 130 may also or instead include any other software or hardware resources that may be usefully employed in the networked applications as contemplated herein. For example, the other resources 130 may include payment processing servers or platforms used to authorize payment for access, content or feature purchases (e.g., certain files, source content 140, normalized source content 150, WPMMLCAs 160, outputs 162, and so on), or otherwise. In another aspect, the other resources 130 may include certificate servers or other security resources for third-party verification of identity, encryption or decryption of data, and so forth. In another aspect, the other resources 130 may include a desktop computer or the like co-located (e.g., on the same local area network with, or directly coupled to through a serial or USB cable) with one of the user devices 110 and/or remote computing resources 120. In this case, the other resource 110 may provide supplemental functions for the user device 110 and/or remote computing resource 120. Other resources 130 may also or instead include supplemental resources such as scanners, cameras, printers, input devices, display devices, and so forth.

[0051] The other resources 130 may also or instead include one or more web servers that provide web-based access to and from any of the other participants in the system 100. While depicted as a separate network entity, it will be readily appreciated that the other resources 130 (e.g., a web server) may also or instead be logically and/or physically associated with one of the other devices described herein, and may, for example, include or provide a user interface for web access to a remote computing resource 120 or a database 126 in a manner that permits user interaction through the data network 102, e.g., from a user device 110.

[0052] It will be understood that the participants in the system 100 may include any hardware or software to perform various functions as described herein. For example, one or more of the user device 110 and the other resources 130 may include a memory 124 and a processor 122, although these components are shown on the remote computing resource 120.

[0053] The various components of the networked system 100 described above may be arranged and configured to support the techniques described herein in a variety of ways. For example, in one aspect, a user device 110 connects through the data network 102 to a server (e.g., that is part of one or more of the remote computing resource 120 or other resources 130) that performs a variety of processing tasks related to determining impact of source content 140 on an audience 170. For example, the remote computing resource 120 may include a server that hosts a website that runs a platform for determining impact of source content 140. More specifically, a user 101 associated with the user device 110 and having appropriate permissions for using the system 100 may use the user device 110 to transmit source content 140 over the data network 102 to the remote computing resource 120 for utilizing components of the system for determining impact of source content 140, such as the WPMMLCA 160. As described herein, the source content 140 may include a context 142, a narrative frame 144 related to at least a portion of source content 140, a narrative perspective 146, and/or metadata 148 (e.g., a date, a time, a description, a hyperlink, a title, etc.). The remote computing resource 120 may receive the source content 140 from the user 101 over the data network 102 for processing thereof. Processing the source content 140 may include normalizing the source content 140. For example, normalizing the source content 140 when in the form of a document may include breaking the document down into paragraphs, sentences, words, images or other media, or any other useful parsing of a document depending on its contents. Processing the source content 140 may also or instead include annotating with one or more labels 152. The normalized source content 150 (shown, for example, in Fig. 1 as including both a label 152 and a pattern 154) may include one or more files, a table, text, and so on. The remote computing resource 120 may further transmit the normalized source content 150 to one or more of the database 126, the user 101, the other resources 130, and so on. Processing the source content 140 may also or instead include using the WPMMLCA 160 for identifying one or more patterns 154 within the source content 140 and/or the normalized source content 150, where such operations may take place on the remote computing resource 120 and/or another component of the system 100.

[0054] A user 101 may provide one or more additional inputs, via user device 110, for use by the WPMMLCA 160, and/or one or more additional inputs may be derived from a user 101. For example, one or more weights 156 may be provided by user 101, where a user 101 may be an administrator or the like that evaluates an impact of the source content 140 and/or an audience 170. The user 101 may also or instead include a member of a test audience (and thus it will be understood that the user 101 may themselves be a member of the audience 170) evaluating the impact of source content 140 or otherwise providing an input. Thus, in some embodiments, weights 126 may be determined based upon other feedback provided by the audience 170 and/or the user 101. In another example, one or more goals 172 may be provided in relation to the source content 140 or a particular intended audience 170. Inputs such as weights 156 or goals 172 may, in addition or instead, be stored in the database 126 for access by the remote computing resource 120, a user 101, or others as desired.

[0055] In an aspect, the remote computing resource 120 may receive the source content 140 from a user 101 over a data network 102 for processing and analysis thereof. The remote computing resource 120 may, upon receiving the source content 140, normalize the source content 140 to create normalized source content 150. The remote computing resource 120 may then analyze the normalized source content 150. Analyzing the normalized source content 150 may include analyzing a particular aspect of the content, such as a narrative frame thereof, e.g., by determining a measurable impact of at least a portion of the normalized source content 150, and/or identifying a pattern 154 within the normalized source content 150. The remote computing resource 120 may further provide one or more outputs 162 of the analysis of the normalized source content 150.

[0056] It will be understood that the processing (e.g., normalizing) and/or analysis of the source content 140 may be conducted using a WPMMLCA 160. It will further be understood that the analysis of the source content 140 by the WPMMLCA 160 may occur anywhere in the system 100, such as locally at a user device 110, or remotely at a remote computing resource 120 and/or a web platform hosted by same. For example, the remote computing resource 120 may transmit the normalized source content 150 and the WPMMLCA 160 to the user device 110 over the data network 102, where a user 101 operating the user device 110 may then analyze the normalized source content 150 using the WPMMLCA 160. Also or instead, the remote computing resource 120 may analyze the normalized source content 150 using the WPMMLCA 160, and then the remote computing resource 120 may transmit output 162 from the analysis to the user device 110 over the data network 102.

[0057] It will be understood that the WPMMLCA 160 may include one or more of a multiple variety of algorithms that are trained against prepared, labeled data sets. Some of these algorithms may include, for example, linear and logistic regressions, decision trees, naive Bayesian, k-nearest-neighbors, k-means, random forest, and others. For the formation / training of the WPMMLCA 160, algorithms may be exposed to both explicitly labeled, as well as computed features.

[0058] The WPMMLCA 160 may be trained on labeled data — e.g., labeled data originally prepared manually and/or through computer-implemented means. It will be understood that the WPMMLCA 160 may be substituted or supplemented by another machine learning classification algorithm in certain aspects. In general, an algorithm used in the present teachings, such as the WPMMLCA 160, may follow standard natural language processing practices; involving the creation of testing groups, and using both supervised and unsupervised learning processes (where it will be understood that this may not necessarily be reduced to semisupervised; instead, the present teachings may use both rather than combining the approaches into a single algorithm). The present teachings may also or instead implement a “Mixture of Experts” approach (see https://en.wikipedia.org/wiki/Mixture_of_experts, which is incorporated by reference herein), as well as expert-augmented machine learning (see https://www.pnas.org/content/! 17/9/4571, which is incorporated by reference herein), among others. Ultimately, testing may involve checking a series of machine-learning classified text segments against those annotated manually or otherwise through the use of a fully -trained algorithm. The aggregate precision and accuracy of that result may then be reviewed. Should the expansion of a testing set struggle to increase accuracy and precision, individual samples may be retested with a series of algorithms that are trained in a similar fashion, but also trained on smaller and more specific subsets of content. For example, testing may utilize an algorithm trained exclusively on annotations tagged as power frames that include an aggregated “black” identity. In this manner, the use of and cued associations between the identity (“black” in this example) and power would be calculated and its likely effects on, for example, the mindset or motivations, of the consumer would be identified. It will be understood that such an algorithm may necessarily diverge from a more generic near-equal algorithm in order to appropriately register context linked to common stereotypes or other signals specifically affected by the identity to which they are linked.

[0059] The general similarity between the test content and the training set may produce a confidence score (see Amit Mandelbaum and Daphna Weinshall, “Distance-based Confidence Score for Neural Network Classifiers,” available at https://arxiv.org/pdf/1709.09844.pdf, which is incorporated by reference herein). This confidence score may operate as a measurable, testable metric defining how similar a tested text chunk is to observed, annotated chunks. When that score indicates low confidence — meaning it is relatively hard to establish similarity to existing patterns, that feedback may be returned to put it into a disambiguation workflow. Disambiguation may involve otherwise processing (e.g., manually processing) a given piece of content, typically by fully annotating the content. Confidence scores can be used to trigger quality assurance processes as well. In some aspects, generative adversarial networks (see https://en.wikipedia.org/wiki/Generative_adversarial_network , which is incorporated by reference herein) may be utilized in order to help improve certainty. This may involve generating content (e.g., sentences of text) that appear, to the model, equivalent or similar to the test case. This may involve annotating that content (e.g., the sentences) as necessary. This may allow a relatively large sample set of grammatically (or otherwise) similar content segments to be disambiguated, widening and deepening a classification algorithm’s understanding of how different parts of the pattern affect the label 152 applied in the present teachings.

[0060] Thus, the WPMMLCA 160 may be tested for quality assurance purposes, where such testing may involve a manual and/or automated (i.e., computer-implemented) review, and where such testing may involve the checking of labels 152 provided within annotated content. Further, the selection of what material to test may be automated in the present teachings. This process may include randomly selecting segments of content classified by the machine learning classification algorithms of the present teachings for quality assurance purposes. This process may also or instead include randomly testing itself against existing annotated data (by taking a large section of annotated data, hiding the annotations and reannotating using the classification system, where, adherence to known, accurate annotations evaluates accuracy). Whatever degree of variance is determined to be unacceptable may trigger further quality assurance testing of annotations (e.g., manually) and in-depth review of the algorithm itself. Testing of annotated content can aid in the identification of any biases, mislabeling, and/or low confidence patterns in the present teachings.

[0061] Fig. 2 is a flow chart of a method for determining impact of source content on an audience, in accordance with a representative embodiment. The method 200 may utilize any of the systems or platforms described herein, e.g., the system 100 described with reference to Fig. 1. In general, the method 200 represents a technique for processing and analyzing source content to understand an impact that the source content may have on an audience. It will be understood that the “audience” may include any as described herein, and in some implementations may include an author / creator / producer of content, a content provider, and the like.

[0062] As shown in step 202, the method 200 may include receiving source content. As described in more detail herein, it will be understood that this step 202 may further include receiving source material including content and associated metadata such as an audience for consuming the content. For example, receiving source content may include receipt by a computing device (e.g., a remote computing resource such as any described herein) from a user uploading the source content onto a system or platform for determining impact of source content on an audience, or otherwise receiving the source content from a user, e.g., via email or the like. Stated otherwise, the source content may be received via an uploading of the source content onto a web-based platform or similar. The source content may also or instead be affirmatively retrieved by a remote computing resource or the like that operates such a system or platform, or a portion thereof — e.g., source content may be retrieved directly from a user’s computing device, from a database containing a plurality of source content, and/or retrieved from a publicly-available source such as the internet. In certain aspects, the source content is provided to an affiliate of a platform for determining impact of source content on an audience, where the affiliate uploads the source content for processing on the platform.

[0063] The source content may include any content intended for an audience. For example, and as discussed herein, the source content may include one or more of text, an image, audio, a video, and the like. Also or instead, in exemplary embodiments, the source content may include an email, a webpage, a social media post, a letter, a memorandum, a piece of legislation, a research paper, a newsletter, an article, a book, a donation appeal, promotional material or other advertising or marketing-related material, or at least a portion of any of the aforementioned content types. In another embodiment, the source content may include a library of content from one or more sources.

[0064] The source content may further include metadata, where such metadata may be related to the source content. The metadata may be received in addition to the source content, either together (e.g., in the same file or otherwise at the same time, such as where the metadata is embedded therein or appended thereto) or at another time for association with the source content. In certain embodiments, the metadata may include one or more of a date, a time, a description (e.g., description of images, a summary, a note, and so on), a hyperlink, a title, an intended audience, an intended effect, a sent rate (e.g., for email), an open rate, an unsubscribe rate, statistics or metrics related to consumption of content, and the like. For example, the source content may be a series of promotional emails that were distributed to members of a mailing list, and the metadata related to the series of emails may include open rates and unsubscribe rates for particular emails in the series, email addresses associated with open and unsubscribe actions, demographic information associated with the email addresses, and the like.

[0065] As shown in step 204, the method 200 may include analyzing an audience, such as an intended, known audience, and/or expected audience of the source content. By way of example, analyzing the audience may include an analysis that produces a score, for example, a motivation score related to a magnitude of motivation for one or more members of the audience. The score may be used, at least in part, as a criterion for determining a weight related to the audience (other criteria may also or instead be used). In certain aspects, analyzing the audience may include identifying information about one or more of the audience members, such as demographic information (e.g., age, gender, race, nationality, economic status, etc.) and/or political affiliation. The score may be based on publicly-available information and/or privately collected information, such as feedback from one or more members of the audience. By way of further example, such feedback from a member of the audience may be based on an impression formed related to predetermined framed language — e.g., one or more members of an audience may be presented with content having a predetermined narrative frame (e.g., the content may include predetermined power-framed language), where feedback is based on an impression formed related to the predetermined narrative frame. Further details regarding the possible effects of an audience on the analyses in the method 200 are described below.

[0066] As shown in step 206, the method 200 may include determining one or more weights for use by a WPMMLCA or the like, where it will be understood that the WPMMLCA may include any as described herein. The weight(s) may be based on, among other criteria, a context related to the source content and/or a context related to a particular audience or a segment of an audience. The context of the source content upon which a weight may be based may include, for example, one or more of a goal of the source content, a topic covered in the source content, an attribute of an originator of the source content, and so on. Examples of attributes of an originator of the source content may include one or more of branding, standing, a domain or sector related to the originator, demographic information, and the like. By way of example, the originator of the source content may be a non-profit organization with a goal of wilderness preservation and the reduction of human impact on the environment, where a goal of the source content may be fundraising. In this example, if the non-profit organization is relatively unknown to the audience, then less weighting may be assigned to mentions of the organization within the source content, while greater weighting may be assigned to mentions of other organizations with higher audience recognition such as known-polluters of the environment.

[0067] It will be understood that, in the context of the present teachings, an “originator” of the source content generally refers to a person, entity, or otherwise that provided the source content for analyses thereof. Thus, the originator of the source content may include the creator of the source content, or the originator of the source content may instead include a provider of the source content that was created by another. It will further be understood that the techniques described herein may take into account the originator of the source content as defined herein and/or the creator of the source content when the originator is not the creator, e.g., where attributes of one or more of the foregoing are used for weighting and/or pattern identification by a WPMMLCA or the like. It will further be understood that source content may include content attributable to a person, entity, document, or the like (e.g., quoted language) that are different from the originator, where attributes of one or more of the foregoing may be used for weighting and/or pattern identification by a WPMMLCA or the like. [0068] A weight may be related to a measurable impact of content on an audience, e.g., a numeric value related to a measurable impact of content on a particular audience. Weighting the impact of content may be guided by social science research. In some embodiments, a weight may be a single value. In other embodiments, a weight may be a ratio, a multiple, a sum, and/or a difference of more than one value. In general, weights may indicate the effectiveness of different content to motivate a particular audience in a desired way, or to otherwise have a measurable impact. In this manner, in certain aspects, the measurable impact includes a motivational impact. It will be further understood that the measurable impact may also or instead include one or more of the following that can occur as a result of consuming content: an emotional response (how a member of an audience feels after consuming content — e.g., happy, angry, sad, powerful, depressed, etc.), a physiological response (e.g., an effect on heart rate, respiratory rate, blood pressure, blink rate, facial expression changes, body language responses that can be conscious or subconscious, and so on), a cognitive or processing slowdown prompted by unfamiliar/novel frames (or vice-versa), a motivational response (e.g., what action(s) a member of an audience would be motivated to perform upon or after consuming content, where such actions may possibly include further research or study, volunteering, donating, working, exercising, shopping, communicating with others, meditating, and so on), and the like. A weight may also or instead be related to a sequencing within the source content (e.g., a sequence of terms used in the source content).

[0069] By way of example, members of a test audience may complete a survey after being presented with a piece of motivational political content. The survey results may be coupled with information about the respondents’ political affiliations to determine a weight, such as a magnitude of motivation, related to one or more audience segments. Weights may also or instead be determined based on one or more contexts of the content presented to the audience. In an example of such a survey of complex or compound content, the source content presented can include a portion having a context related to power and a portion having a context related to disparity. The test audience can include members having political affiliations of two different political parties — party A and party B. The study can be used to determine weights (e.g., magnitudes of motivation), specifically of: Weight 1 — party A affiliates presented content with power-related content; Weight 2 — party B affiliates presented content with power-related content; Weight 3 — party A affiliates presented content with disparity-related content; and Weight 4 — party B affiliates presented content with disparity -related content.

[0070] An example of the results of such a test audience is included below in Table 1 :

Table 1

[0071] For the results shown in Table 1, predetermined content was provided to two groups — those with a political affiliation of “Democrat,” and those with a political affiliation of “Republican” — where the predetermined content included content, based on research, hypothesized to have a certain disparity frame and a certain power frame. The magnitude of motivation may be a statistically significant result of participants scoring their own sense of motivation on a scale created by a research team or the like. Based on these example results, the present teachings may prioritize the use of material resembling the power framed variant test material in order to maximize overall motivation due to the higher magnitude of self-reported motivation after consuming power-framed content.

[0072] The above example may be used to form a matrix that can be utilized to select weights for use by the WPMMLCA based on an audience. And, in this manner, the method 200 may include comparing the context and the audience to one or more predetermined contexts and audiences included in a matrix, and selecting one or more weights at least in part based on a similarity of the context and the audience to one or more predetermined contexts and audiences included in the matrix. It will be understood that, in such a matrix, the input factors can be dynamic, meaning that they can change based on one or more of context, the audience, social reforms, political and/or societal shifts, demographics changes, and so on. In other words, because external outputs can be unpredictable, the matrix can shift over time as a technique to attempt to address this unpredictability. Regardless, a matrix such as that described herein may be referred to algorithmically by the WPMMLCA when source content substantially matches (to a predetermined extent) the context/frame in the matrix, and an audience resembles the audience in the matrix (again, to a predetermined extent). Specifically, the WPMMLCA may use the stored values in the matrix as a reference point. For the example discussed above, the algorithm may simply observe a higher number of appearances and confirm that the power variant is more successful at motivation. And in this manner, should the question be posed, the algorithm (or an output thereof) can represent that the most successful way to demotivate Republican-affiliated individuals is to use a disparity frame, where this is made possible by applying stored values as a scalar weight or ratio weight for use by the WPMMLCA. [0073] Therefore, it will be understood that an intention of the originator of the source content may be an input for the present teachings, and more specifically, for the WPMMLCA, e.g., in selecting weights or otherwise performing its analyses. For example, a factor used as an input may include whether the source content is intended to motivate an audience to donate money, or to reinforce a brand identity, and so on. Thus, an input may include an intention or goal of an originator/provider of source content. An input may also or instead include an identity of an originator/provider of source content, or of an audience. Such input and similar inputs may be provided with the source content, e.g., in step 202 of the method 200.

[0074] As shown in step 208, the method 200 may include selecting one or more labels, e.g., for use in normalizing the source content as described herein. Selection of the labels may be based upon a context of the source content, a goal of the source content, a sentiment associated with the source content (e.g., positive or negative feelings, and the like), a characteristic of an audience (e.g., an intended audience), and the like. For example, if a goal of the content is to motivate the purchase of a product, labels may be applied to terms describing the product or the company selling the product.

[0075] Selecting one or more labels may utilize natural language processing and/or other similar techniques, for example, when the source content includes text. In this manner, a label may be based on a grammar structure of the source content, e.g., a grammar structure that defines one or more of a person, a noun, and an actor in relation to a verb. It will be further understood that, in the context of selecting a label and/or in any of the analyses described herein, analysis of compound or modified frames is included within the scope of this disclosure. For example, terms separated by adjectives can be recognized (e.g., through machine learning techniques or otherwise) and appropriately labeled and/or otherwise processed such that the meaning of such compound language structures is accurately processed using the present teachings.

[0076] It will be understood that a label may relate to a characterization (e.g., a term such as beautiful, studious, maternal, etc.), a power frame, a sexualization, a negative tone, implicit biases, a growth mindset, a disparity frame, an obstacle frame, a stereotype, and the like. And in this manner, it will be understood that a label may include a keyword or the like that relates to one or more of the above-provided examples or otherwise to a narrative frame.

[0077] As shown in step 210, the method 200 may include parsing the source content. For example, text-based source content may be parsed by aggregating grammatical structures that describe, e.g., persons, nouns, or actors in relation to a specific verb. Such parsing may be used particularly for certain verbs that impact motivation and/or that occur in motivational patterns. Also or instead, source content may be parsed based on an author thereof. Additionally or alternatively, parsing the source content may be based on recitations from perspectives of a first person, a second person, and a third person. In other words, parsing the source content may be based on a narrative perspective of the source content. In certain aspects, parsing the source content may include identifying and/or grouping content within the source content such as text, images, quotations, citations, and the like. It will be understood that parsing the source content may be part of the normalization processes described herein, and vice-versa.

[0078] As shown in step 212, the method 200 may include normalizing the source content, such as by annotating with one or more labels. For example, at least one of the labels may be used to tag or otherwise identify grammatical structures, parts of speech, data types (e.g., text, image, audio, etc.), languages, rhetoric, topics, entities, narrative frames, concepts, identities, sectors, organizations, and the like. Annotating with the labels may utilize natural language processing and/or other computer-automated techniques. In embodiments, normalizing the source content may be performed by a WPMMLCA as described herein.

[0079] Normalizing the source content may also or instead include extracting or grouping specific types of data based upon a type of source content. For example, text and images from the source content may be separated or grouped together. Similarly, certain portions of content can be identified for particular analysis thereof; for example, whether text includes a link, a title, a formal name, a headline, an organization, and so on, may affect the analysis thereof. Normalizing the source content may also or instead include one or more of reformatting, word stemming, tokenization of content, synonym mapping, identification and/or removal of stop words, creation of lemmas, named entity recognition, the creation of n-grams, and the like. Normalizing the source content may also or instead include associating and/or grouping a label with a different label, where such an association may be based on at least one of a topic, a frame, an identity, a sector, a sample size, and the like. By way of example, normalizing the source content may involve the linking of annotations such that contiguous text and/or non-contiguous text can be associated, the latter of which can allow for concepts expressed at one portion of the source content to be associated with references to these concepts elsewhere in the source content. In this manner, the WPMMLCA may map the relational nature of topics included within the source content, and the like.

[0080] Part of the normalizing process and/or part of a quality assurance check or otherwise may include associating a confidence score to at least one of the labels, and using the confidence score as a factor when analyzing the normalized source content as described herein. That is, the confidence score may be used by the WPMMLCA when analyzing the normalized source content. More specifically, the confidence score may indicate a similarity to labeled content to known or existing patterns. For example, a low confidence score may indicate that labeled content has a low degree of similarity to existing patterns. And by way of further example, a confidence score below a predetermined threshold may trigger a notification for review of the label associated therewith. The review may be computer implemented and/or may include a manual review. Thus, in one embodiment, a notification may be triggered when a label has a confidence score below a predetermined threshold. The notification may further trigger specific actions, such as an additional processing step applied to the source content, or the use of a specific WPMMLCA (or setting and/or adjustment thereof) for analysis.

[0081] Therefore, normalizing the source content may include identifying different topics covered in the material and/or the topography of the material, where this information may be used in conjunction with attributes of one or more of the originator/provider of the source content and the audience (e.g., the branding and positioning of the originator/provider of the source content in their domain/sector). The normalization process may thus involve the creation and/or application of a series of annotation labels and/or word-matching filters as described herein. For example, the WPMMLCA may determine that it should specifically identify every annotation that includes a specific product sold by the originator of the source content or other entity associated therewith, or a project the entity is working on. Other examples can include tagging every reference to a specific grantee of a foundation by scanning for the name of that grantee (or otherwise tagging entities), adding specific labels to annotations that pertain to a grantee, identifying all instances of certain terms, and so forth. By way of specific example, for a specific analysis, the WPMMLCA may be trained to identify all instances of the word “violence” in source content (e.g., once the source content has been stemmed and lemmatized). Continuing with this example, because it may be far more complicated to determine every case where a violent action is described without identifying the agent of the action (person or people who acted violently), using the pattern matching and weighting capabilities of the present teachings (as progressed/trained through the addition and processing of more and more data), the identification of “violent + violent agent” may be automated, e.g., by aggregating grammar structures that define persons, nouns, and/or actors in relation to verbs that often describe violent actions, and the present teachings may further include defining concrete patterns in those tightly defined grammatical constructs.

[0082] Normalizing the source content may also or instead include disambiguating content within the source content. This disambiguation may include some of the aforementioned techniques, including contextualizing, sorting, grouping, annotating, labeling, tagging, flagging, filtering, extracting, and the like.

[0083] An example of how portions of text within source content may be normalized through labeling is — “Power - 3 ^rd Person” — which may indicate a compound concept, i.e., that a portion of the source content so labeled is power framed, and that it occurs in material stemming from someone other than the main author of the source content. More simplistic labeling is also or instead possible, such as a binary label regarding whether a portion of source content is power framed or not (or otherwise framed or not).

[0084] As shown in step 214, the method 200 may include analyzing the normalized source content using the WPMMLCA to identify one or more patterns within the source content. The analysis may at least in part include use of the labels in identifying such a pattern. Thus, the present teachings may include the use of a WPMMLCA to classify content based on patterns observed in previously annotated examples (e.g., via training of the WPMMLCA), to measurably improve the efficiency of this investigative process.

[0085] A pattern identified by the WPMMLCA may be related to a narrative frame of at least a portion of the source content, as the term “narrative frame” is defined herein, or more generally as any identifiable trait or attribute that could form a pattern or inform an analysis of a pattern. A pattern identified by the WPMMLCA may also or instead include a linguistic pattern, such as where the source content includes text and/or audio. A pattern identified by the WPMMLCA may also or instead relate to a rhetorical device and/or a narrative perspective. For example, the pattern may relate to one or more of a consistency, a tone, a sequence (e.g., of words, imagery, presentation of topics, listing of items/persons/entities, and so on), an argumentation, an ideation, and so on. By way of example, the WPMMLCA may analyze each sentence in source content that includes text or audio, and/or associated building blocks thereof, where changing the framing, such as by changing a sequencing or use (or similar) of such structures, can affect a pattern identifiable by the WPMMLCA.

[0086] An input for analysis by the WPMMLCA may include one or more tables, where such tables may be processed through the use of a simple word search, lemmatization, the creation of n-grams, and others. The tables may thus be used to aggregate data for analysis by the WPMMLCA. In this manner, the tables (as input to the WPMMLCA for analysis) may include an output of the normalization described herein.

[0087] Pattern detection can be performed by structured, intuitive pattern recognition analysis. This may entail relating grammatical constructs, actors, verbs, sentiment, tense, and other features. As an example, a common written method to explore “discrepancies” or numerical imbalances is to refer to something as “over-represented” or “under-represented.” This relies on both some general expectation that there be a natural or intrinsic balance between two groups. An intentional, or perhaps “supervised” pattern matching process may focus on the presence of: two numbers of differing value, the use of “over-“ or “under-,” or the grammatical patterns of “group + numerical expression & alternate group + alternate numerical expression.” [0088] The patern matching process may also or instead be “unsupervised,” relying explicitly on algorithms to detect paterns based on the labeled or otherwise computed features. This unsupervised pattern matching may draw an extremely wide range of features and characteristics into review. An illustrative example of unsupervised learning applied in a fully separate field might be the use of algorithms to evaluate health features to detect cancer prior to traditional signals. In that example, traditional signals represent intentional analysis (supervised learning) while a system observing as many features as possible may draw entirely different features, both detecting cancer earlier and advancing our understanding of the underlying causes of cancer. An example in this context is Konstantina Kourou, et al., “Machine learning applications in cancer prognosis and prediction,” Computational and Structural Biotechnology Journal, Vol. 13, pp. 8-17 (2015).

[0089] As shown in step 216, the method 200 may include determining a measurable impact of the portion of the source content based on one or more paterns related to the narrative frame or lexicon. The measurable impact may be determined by the WPMMLCA, e.g., at least in part using the weights determined above in the method 200. The measurable impact may relate to, for example, a motivational impact, an emotional impact, a cognitive impact, a behavioral change, and so on. By way of example: in the case where an elementary school has drafted an apology leter to their community due to the closure of their after-school program, the presence of profanity may signal a likelihood that the leter will not be seen as “authentically apologetic.” And thus, a measurable impact may include a negative emotional response by a consumer of this content, and/or a lack of motivation to forgive, donate, volunteer, etc.

[0090] As shown in step 218, the method 200 may include providing one or more measurables related to the measurable impact of the portion of the source content that is analyzed by the WPMMLCA. In this context, a measurable may include, for example, a type of narrative frame included in the source content, a numerical count of different narrative frames identified in the source content, a statistically significant indicator, a score related to the impact and/or narrative frame, and so on. In this manner, one or more measurables may include at least one of a quantity and a type of one or more narrative frames included in at least the portion of the source content that is analyzed by the WPMMLCA. By way of example, to help illustrate the likely efficacy of an advertising campaign, a short list enumerating the number of ideologically- aligned celebrities may be useful. Another example can include providing how many spoken lines of dialogue are delivered by Latinx actors.

[0091] As shown in step 220, the method 200 may include analyzing the narrative frame. The analysis, or an output thereof, may include quantifying the narrative frame, e.g., for inclusion in a report or the like. The narrative frame may be quantified, by way of example, by one or more of a score, grade, and the like. An example includes a general “positivity” rating according to concepts covered, sentiment, and/or other patterns expressed in content, e.g., a community garden outreach letter. By way of further example, this may include a raw count of how many times a campaign slogan is repeated in the entire corpus of a political candidate’s television interviews.

[0092] Nothing included herein should be understood to foreclose the application of the described process to images as distinct from or in conjunction with words. Image processing is the sine qua non of cognition and motivation, as pictures or images can activate conscious and unconscious associations more quickly than words. The analysis can thus include examination of words alone or pictures alone or words and images together.

[0093] The analysis, or an output thereof, may include providing one or more recommendations or suggestions to alter the narrative frame. Such recommendations or suggestions may include possible alternative content for the portion of the source content that was analyzed by the WPMMLCA. And such alternative content may include alternative text, alternative images, alternative arrangement of text/imagery, and so on.

[0094] As shown in step 222, the method 200 may include providing one or more outputs based on the analysis by the WPMMLCA, some of which have already been explained above such as providing a recommendation to alter a narrative frame and/or lexicon or sequencing of source content. An output of the method 200 may also or instead include a report, e.g., a report that quantifies the narrative frame and/or that identifies certain portions of the source content that are framed in a certain manner. Also or instead, an output of the method 200 may include at least one of the following: one or more recommendations to alter the narrative frame; one or more recommendations of alternative content for at least a portion of the source content, e.g., alternative text for an excerpt of content; edits to the content selected to alter the measurable impact of the portion of the source content; a table including labels and the portions of the source content annotated with the labels; and so on. Output may also or instead include new content to replace and/or supplement the source content, such as rearticulating language of at least a portion of the source content to alter the measurable impact thereof.

[0095] In order to produce the outputs, the WPMMLCA or similar may utilize the parsed and/or normalized source content, which may be restructured into one or more tables. The algorithm may apply rules for filtering these tables and extracting aggregate statistics or the like related to an intended outcome of the analysis, such as quantifying a narrative frame. By way of example, an output can include aggregating power frames within the source content under an arbitrary grouping of topics (where topics may be associated with annotations). For example, the present teachings might aggregate “Commerce” and “Business” tagged annotations to create a view relevant to an investigation plan goal of “how are power frames represented in the subject of Economy.” Or perhaps “crime,” “criminal justice,” and “policing” in an investigation plan goal of “how many power framed annotations are related to arrests.”

[0096] A review of the output may be conducted manually and/or through computer- implemented techniques. By way of example, a review can trigger the creation of a new label and restart an annotation process to observe that label. If necessary, source content can be reannotated, e.g., going through the process of: updating annotations, processing content and annotations, generating output, aggregating data in-line with an investigation plan, and rereviewing output.

[0097] It will be understood that the present teachings may include a computer program product that implements one or more steps of the method 200. For example, in an aspect, a computer program product disclosed herein includes computer executable code embodied in a non-transitory computer readable medium that, when executing on one or more computing devices, performs the steps of: receiving source content intended for an audience; determining one or more weights for use by a WPMMLCA based on at least a context related to the source content and the audience, the one or more weights related to a measurable impact of content; normalizing the source content by annotating with one or more labels; analyzing the normalized source content including the one or more labels using the WPMMLCA to identify one or more patterns related to a narrative frame of at least a portion of the source content; and determining, using the WPMMLCA and the one or more weights, the measurable impact of the portion of the source content based on the one or more patterns related to the narrative frame.

[0098] It will be understood that the present teachings may include a system configured to implement one or more steps of the method 200, where such a system may be the same or similar to any as described herein, e.g., with reference to Fig. 1. For example, in an aspect, a system disclosed herein includes a computing resource including a processor and a memory, the memory storing computer executable code embodied in a non-transitory computer readable medium that, when executed, causes the processor to perform the steps of: receiving source content intended for an audience; determining one or more weights for use by a WPMMLCA based on at least a context related to the source content and the audience, the one or more weights related to a measurable impact of content; normalizing the source content by annotating with one or more labels; analyzing the normalized source content including the one or more labels using the WPMMLCA to identify one or more patterns related to a narrative frame of at least a portion of the source content; and determining, using the WPMMLCA and the one or more weights, the measurable impact of the portion of the source content based on the one or more patterns related to the narrative frame. [0099] Figs. 3-9 show example workflows for determining impact of source content on an audience, in accordance with representative embodiments.

[0100] Specifically, Fig. 3 shows an example intake and annotation of source content in the form of a collection of documents. In this example, the system receives a collection of documents for processing. Such processing may include annotating and/or tagging the documents as a batch, and performing some normalization such as breaking the documents down into paragraphs, sentences, words, as well as preserving certain contextual clues regarding the purpose of the document(s) itself as well as a particular section thereof. For example, this may include identifying the “Abstract Summary” of a research paper, the methods and discussion sections, as well as the links to related material. The metadata for this process may vary by collection. To preserve the character-based position of the start and end of a segment and for flexibly including various semantic or other types of tagging, an extensible data format may be used. An example might include: collect! on id, document ! d, tags (polymorphic join to complex field of “tags”), start position, stop position, transformations (a brief explanation of the various techniques applied to “clean” the document), and so on. Tags may include frames, source, language, purpose, speaker, concept, stereotype, sentiment, etc. Transformations and processing may include word stemming, lemmatization, the stripping of non-standard characters, etc. The annotation may allow for the tagging of document sections based on start position and stop position along with the association of various tags and/or their sequencing within the content. In the example embodiment of Fig. 3, the output of processing the documents may include certain collections, labeled “Collection 1” and “Collection 2,” which may include any of the foregoing metadata, tags, labels, annotations, etc.

[0101] Fig. 4 shows an example of quality assurance (QA) review of annotated documents, e.g., according to specific workflows and the definitions of tags, shown as “Collection 1” in this figure. QA may involve reviewing the association of tags to various sections of the documents, indicated, for example, by start and stop position. Additional QA processes may run based on the output of the system’s classification algorithms. One of the metrics may be a computed “confidence” (based on the mechanism of machine learning classification). Should this confidence dip below an acceptable level, QA may be triggered to review a wide band of low to moderate confidence scores. That is to say, a lot of sentences tagged as possibly matching the feature may be reviewed to confirm or deny matching, and thus improve precision. Furthermore, this workflow may include QA for the system, analysis, annotations, and algorithms as deemed necessary. Once QA is performed, an output may include a refined document or collection, shown in the figure as “New Collection 1.” [0102] Fig. 5 shows an example process by which research may be analyzed. Analysis may be performed, for example, on existing research or on research methodologies. This may include identifying additional testable variables and outcomes, and ultimately developing novel social and psychological research tools (such as Implicit Association Tests (IATs)) to explore an impact (e.g., the motivational impact) of specific content through linguistic and/or rhetorical patterns on specific audience cohorts. Through the use of this system’s landscape of features — such as identities, context, rhetoric, framing, linguistic patterns, topic, domain (among others), and the like — the system may be able to analyze extant social science research and infer cases where multiple feature variants are aggregated as one, and cases where only a single feature variant is observed (to the exclusion of others) that are not explicitly intended by the research case. For example, it may be appropriate to aggregate “all children” when testing if “children enjoy candy.” However, should it be determined that “candy” was only represented with images of licorice, the initial study’s design may come into question. This analysis may serve to inform research study design (e.g., new research methodologies and strategies), as well as to qualify the results of other studies insofar as their testing design is shown to be compatible with the intended use of the results.

[0103] Fig. 6 shows an example of how IATs and other social science research can be leveraged to “score” linguistic patterns according to the scientific results. For example, if Pattern A causes more people to wish to purchase a product or vote for a candidate than Pattern B, those results may be processed into a technical weighting system. This weighting system may represent the improved efficacy of Pattern A when evaluating content with the explicit outcome goal of, in this example, motivating purchasing or voting. This process may produce a WPMMLCA as described herein.

[0104] This process may involve the use of social science research results as weighting scores or the like. For example, testing mechanisms may return aggregate results indicating that a specific rhetorical argument is more motivationally effective than another. With an appropriate number of respondents, the difference between two semantically equivalent but stylistically different pieces of text may be statistically significant. For example, should Pattern A result in an aggregate 3.1/7 on the motivation scale and Pattern B result in an aggregate 3.8/7, the system may weight Pattern A as -81% as effective for the “motivation” vector (3.1 / 3.8 = 0.81) when bounded by the context of the study. In this case, the “out of 7” scale refers to a standard social science qualitative scale of strongly disagree / disagree / slightly disagree / neutral / slightly agree / agree / strongly agree, where the weight of each response increases in increments of 1, for example. Other qualitative rating scales may use a similar pattern with different verbs. This weighting may thus allow the system to appropriately emphasize Pattern B when targeted for motivation within the specific topic / context. For example, Pattern B may better motivate participants to write a letter to their senator about climate change.

[0105] Fig. 7 shows an example of a content editing system. Specifically, this figure shows a text editing system whereby a user explicitly defines their goal relevant to their application (e.g., to motivate readers to purchase an item) and the user’s text is analyzed according to the WPMMLCA (e.g., classifiers with functional efficacy weights may form one component of the analysis). A provided output may be options to substitute content (e.g., Pattern B with Pattern A) in pursuit of the stated goal.

[0106] This may represent a text/image-editing interface whereby sentences and sentence fragments are identified as matching (or resembling) patterns understood by the system. Should the user then specifically indicate the domain/topic, as well as the intent of the document (for example, to motivate readers to write to their senator about climate change), the system may apply its classification algorithms and weighted variables to represent areas where the user may be able to, for example, switch from using Pattern A to Pattern B to better motivate readers. This may be heavily dependent on a generative adversarial network (GAN) (see https://en.wikipedia.org/wiki/Generative_adversarial_network , which is incorporated by reference herein) trained against annotated data. Alternatively, the system may apply additional supplemental logic to existing GANs in order to filter programmatic outputs to select a subset that matches the needs of the system, rather than creating its own GAN.

[0107] Fig. 8 shows an example of the use of classification algorithms to segment audiences. Specifically, this figure shows classification algorithms used to help segment audiences based on which patterns are most effective for members of the audience. For example, perhaps Pattern A is most effective with college-age Americans; and perhaps Pattern B is most effective with people who live on the upper west side of Manhattan that frequently donate to a certain institution.

[0108] In this context, the system may be capable of ingesting information regarding audience engagement and actions (or some other metrics, shown as user data in the figure), and then segmenting that audience or collection of users (e.g., grouping an audience into sections) based on which content was most effective at promoting which other actions. For example, a segment of newsletter recipients may have opened more newsletters with titles referencing climate change. That section would predictably be identified as having greater likelihood of opening emails regarding climate change. This information can be used to represent, to the user of the application, the likely effects of drafting their next newsletter around climate change or some other topic. Should the user choose another topic, the system may anticipate the audience specifically interested in climate change to be less interested. That tradeoff may be represented using rough statistical likelihoods of engagement, as well as further computed statistics regarding other content goals, for example, an intent to raise funds. In the case of a fundraising goal, aggregate fundraising data can be used to roughly predict fundraising results based on past performance of topics, language, and patterns. A stream of contemporary data may be used to improve the viability of that prediction, such as analyzing social media sentiment regarding a topic, and the like. Should the user desire it, the system can even look for patterns within their own funding data, such as attempting to connect it to the content of appeals, historical social media sentiment, historical and legislative context, and the like. For consumer-oriented product and service-oriented users, historic user data may have similar efficacy as other contemporary data, such as when analyzing advertising campaigns, sales, or base product information, which can all provide meaningful contextual data for this system.

[0109] Thus, in this manner, it will be understood that one or more of the algorithms described herein may be used to identify patterns in data (e.g., source content and/or user/audience interactions therewith or other similar metrics) and make one or more predictions as to likely outcomes based on an identified pattern or other insight garnered from the data by the algorithm. It will further be understood that, with ample data and refinement, the algorithm(s) can function autonomously to perform such tasks. Such predictions may be related to, by way of example, how an audience and/or user will perceive and/or react to source content — e.g., the impact of the source contact on the user and/or actions that the user may take in response to the source content. Predictions may be made based on one or more of the source content or a portion thereof, a source and/or creator of the source content, characteristics of the audience, similar audience results (e.g., perceptions, actions, impact, etc.), and the like. Similarly, rules may be created based on identified patterns, where such rules may be related to making a prediction related to the source content — e.g., IF ‘x’ conditions are met THEN certain ‘y’ predictions are made. This can be done using iterative assessments that can highlight efficacy and provide feedback to adjust the system (e.g., to identify whether more information or context would be beneficial to a system).

[0110] Fig. 9 shows an example of the organization of tags or topics. Specifically, this figure shows an example of the organization of tags or topics into collections used to segment documents, e.g., in an effort to profile rhetoric, language, and/or concepts used within that collection. That is, where source content is a document, the system may be able to use the topics evident in the document, alongside the patterns, targeted and real audiences, and the publishing date of a document (among other data points, shown in the figure as metadata) to help organize a user’s content along several vectors (e.g., collection 1, collection 2) that help provide insight into trends (successful or otherwise) within a user’s library of content. [oni] Some potential additional iterative applications of the techniques and systems described herein will now be described.

[0112] The present teachings may include the development of social and psychological research tools (such as Implicit Association Tests (IATs)) to explore the behavioral and motivational impact of specific linguistic and rhetorical patterns on specific audience cohorts.

[0113] The present teachings may include the embedding into a multi-channel communications platform — e.g., leveraging the present teachings in relation to emails, text, imagery, social media campaigns, and the like. This platform could leverage many of the techniques described herein within an ever-growing contextualized understanding of a user’s business/goals and communications.

[0114] The present teachings may include the use of the described techniques as a text annotating “plug-in” or application. For example, this may include a content filter for an organization of documents or images or the like within a company, website, organization, personal corpus, and the like. The present teachings may include performing a meta-analysis of past communication and/or outreach campaigns to attempt to determine the strategies that were cognitively and/or motivationally successful and to determine how to reproduce their success.

[0115] The present teachings may include the use of the described techniques to take a “snapshot” of contemporary discourse as it relates to a specific sector, subject, and/or issue. This could afford users with a vision into the dynamics of a specific conversation/discourse and, ultimately, may help forecast public response to an issue, e.g., insofar as the public response to the issue follows any of the identified patterns.

[0116] The present teachings may include political and/or corporate risk assessment and triage as it relates to the public perception of specific issues and topics for an individual and/or organization or even how the company perceives or is perceived itself. For example, how does a company respond to claims of misogyny? And more specifically, the present teachings could assist with regard to an organization’s or individual’s communications strategy and triage.

[0117] The following provides some additional context for techniques and processes mentioned elsewhere in this document.

[0118] An annotation system process may include: (1) research gathered or commissioned; (2) documents curated, “cleaned,” prepared for input into annotation tools; (3) annotated according to tags, grammatical structures, named entities; (4) flow into a database; and (5) quality assurance.

[0119] An annotation system outputs may include:

[0120] 1. Contextual analysis — e.g., how does the presence of the word “unusually” affect the rating of the word “brilliant” in the sentence: “This woman is unusually brilliant.” [0121] 2. Topic / sector analysis — e.g., what are relevant concepts in business, politics, education?

[0122] 3. Name Entity Recognition — e.g., who or what are important named entities in a sample (e.g., Dr. Martin Luther King carries significant contextual & connotative meaning, while lesser-known figures may not).

[0123] 4. Review of contexts, images, lexicon, and stereotypes commonly found in sampled content.

[0124] The present teachings may include framing detection:

[0125] 1. Annotated documents may be sub-set into collections based on tags (e.g., all annotations with reference to the USA can be a subset; all annotations expressing first person perspective can be a subset).

[0126] 2. A collection may be used to train a specific classification algorithm for framing (e.g., an obstacle framing algorithm may be trained against only annotations with first- person material).

[0127] 3. Various algorithms may be combined using Mixture of Experts or the like, e.g., hierarchically.

[0128] Generative models may be used, such as an algorithm that can produce alternatively framed text with equivalent semantic and denotative meaning (possibly leveraging bespoke GPT-3 (Generative Pre-trained Transformer-3) implementation or the like). Grammatical Clustering may be used (component of generative models), such as through: (1) tagged text isolated, organized around expressed grammatical patterns; (2) within each grammatical segment: identify parts of speech, topics, entities, frames; (3) algorithm trained on a collection of equivalent/similar grammatical segments.

[0129] System infrastructure may include: cluster of microservices; graph engine; security; QA evaluation authority. Information used may include: articles; research papers; newsletters; books; metadata. Code categories may include: natural language processing; model training; analysis and deployment pipelines.

[0130] A generalized feature roadmap (e.g., outputs) of the present teachings may include one or more of: (1) show a user how they can change a document in order to, e.g., increase donation yield; (2) show a user how they can change a document in order to help a cause (e.g., advancing the positive perception of women’s power); and (3) show a user how they can change a document in order to motivate an audience.

[0131] The present teachings may include ontological learning: https://en.wikipedia.org/wiki/Ontology_leaming (incorporated by reference herein). [0132] The present teachings may include a lexical discovery phase: word extraction - tokenization; Synsets - synonym mapping. This may include: (1) generating a list of terms from Document A; (2) mapping terms to synonyms; and (3) finding a similar “Document B” using an expanded set of similar words.

[0133] The present teachings may include a contextual analysis phase, which may include: categorization - e.g., vector-based classification; Sense2Vec - e.g., semantic similarities; similarity detection - e.g., find similar using Jaccard’s Index or the like.

[0134] Example using Jaccard’s Similarity Index to discover the similarity of two documents:

[0135] A = {0,1, 2, 5, 6}

[0136] B = {0,2, 3, 4, 5, 7, 9}

[0137] Solution: J(A,B) = |AAB| / |AUB| = | {0,2,5} | / 1 {0, 1,2, 3, 4, 5, 6, 7, 9} | = 3/9 = 0.33.

[0138] The present teachings may include general natural language processing techniques such as one or more of: extraction; tokenization of content; removal of stop words; creation of lemmas; and custom named entity recognition.

[0139] The above systems, devices, methods, processes, and the like may be realized in hardware, software, or any combination of these suitable for a particular application. The hardware may include a general-purpose computer and/or dedicated computing device. This includes realization in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices or processing circuitry, along with internal and/or external memory. This may also, or instead, include one or more application specific integrated circuits, programmable gate arrays, programmable array logic components, or any other device or devices that may be configured to process electronic signals. It will further be appreciated that a realization of the processes or devices described above may include computer-executable code created using a structured programming language such as C, an object oriented programming language such as C++, or any other high-level or low-level programming language (including assembly languages, hardware description languages, and database programming languages and technologies) that may be stored, compiled or interpreted to run on one of the above devices, as well as heterogeneous combinations of processors, processor architectures, or combinations of different hardware and software. In another aspect, the methods may be embodied in systems that perform the steps thereof, and may be distributed across devices in a number of ways. At the same time, processing may be distributed across devices such as the various systems described above, or all of the functionalities may be integrated into a dedicated, standalone device or other hardware. In another aspect, means for performing the steps associated with the processes described above may include any of the hardware and/or software described above. All such permutations and combinations are intended to fall within the scope of the present disclosure.

[0140] Embodiments disclosed herein may include computer program products comprising computer-executable code or computer-usable code that, when executing on one or more computing devices, performs any and/or all of the steps thereof. The code may be stored in a non-transitory fashion in a computer memory, which may be a memory from which the program executes (such as random-access memory associated with a processor), or a storage device such as a disk drive, flash memory or any other optical, electromagnetic, magnetic, infrared, or other device or combination of devices. In another aspect, any of the systems and methods described above may be embodied in any suitable transmission or propagation medium carrying computer-executable code and/or any inputs or outputs from same.

[0141] The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings.

[0142] Unless the context clearly requires otherwise, throughout the description, the words “comprise,” “comprising,” “include,” “including,” and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in a sense of “including, but not limited to.” Additionally, the words “herein,” “hereunder,” “above,” “below,” and words of similar import refer to this application as a whole and not to any particular portions of this application.

[0143] It will be appreciated that the devices, systems, and methods described above are set forth by way of example and not of limitation. For example, regarding the methods provided above, absent an explicit indication to the contrary, the disclosed steps may be modified, supplemented, omitted, and/or re-ordered without departing from the scope of this disclosure. Numerous variations, additions, omissions, and other modifications will be apparent to one of ordinary skill in the art. In addition, the order or presentation of method steps in the description and drawings above is not intended to require this order of performing the recited steps unless a particular order is expressly required or otherwise clear from the context.

[0144] The method steps of the implementations described herein are intended to include any suitable method of causing such method steps to be performed, consistent with the patentability of the following claims, unless a different meaning is expressly provided or otherwise clear from the context. So, for example performing the step of X includes any suitable method for causing another party such as a remote user, a remote processing resource (e.g., a server or cloud computer) or a machine to perform the step of X. Similarly, performing steps X, Y, and Z may include any method of directing or controlling any combination of such other individuals or resources to perform steps X, Y, and Z to obtain the benefit of such steps. Thus, method steps of the implementations described herein are intended to include any suitable method of causing one or more other parties or entities to perform the steps, consistent with the patentability of the following claims, unless a different meaning is expressly provided or otherwise clear from the context. Such parties or entities need not be under the direction or control of any other party or entity, and need not be located within a particular jurisdiction.

[0145] While particular embodiments have been shown and described, it will be apparent to those skilled in the art that various changes and modifications in form and details may be made therein without departing from the spirit and scope of this disclosure and are intended to form a part of the invention as defined by the following claims, which are to be interpreted in the broadest sense allowable by law.