BRIGGS, Jason Rex (4364 Town Center Blvd, Suite 320El Dorado Hills, California, 95762, US)
|WHAT IS CLAIMED IS:
1 A method, comprising: providing a user interface; receiving information relating to one or more models; and displaying a visual representation of at least one of the models in relation to at least one other model.
2. The method of claim 1, wherein the model comprises a cause and effect relationship.
3. The method of claim 1, further comprising sharing at least one of the models.
4. The method of claim 1, further comprising connecting at least one of the models to at least one other model.
5. The method of claim 1, further comprising receiving one or more collaborations on at least one model.
6. The method of claim 1, further comprising creating at least one consensus model.
7. The method of claim 1, further comprising storing explanatory information relating to at least one model.
8. A computer readable storage medium comprising program instructions stored thereon, wherein the program instructions are computer-executable to implement a method comprising: providing a user interface; receiving information relating to one or more models; and displaying a visual representation of at least one of the models in relation to at least one other model.
9. A system for determining an apportionment of marketing resources, comprising:
a data memory coupled to the CPU; and
a system memory coupled to the CPU, wherein the system memory is configured to store one or more computer programs executable by the CPU, and wherein the computer programs are executable to implement a method comprising: providing a user interface; receiving information relating to one or more models; and displaying a visual representation of at least one of the models in relation to at least one other model.
10. A method, comprising : receiving, by a computer system, information relating to two or more models; linking, by the computer system, at least one of the models with at least one other of the models; and displaying, by the computer system, a graphical representation of at least one of the linked models in relation to at least one other of the linked models.
11. The method of claim 10, wherein at least one model is linked to at least one other model automatically.
12. The method of claim 10, wherein linking at least one of the models with at least one other of the models comprises executing an integration algorithm.
13. The method of claim 10, wherein changing an input in at least one of the linked models changes at least a portion of the output of at least one other of the linked models.
14. The method of claim 10, wherein the graphical representation comprises at least a first icon corresponding to a first one of the models and a second icon corresponding to a second one of the models, wherein the size of at least the first icon and the second icon varies as a function of impact of the corresponding model.
15. The method of claim 10, wherein the graphical representation comprises at least a first connecting symbol corresponding to a first link between at least two of the models and a second icon corresponding to a second link between at least two of the models, wherein the size of at least the first connecting symbol and the second connecting symbol vary as a function of the strength of the corresponding link. 16. The method of claim 10, further comprises receiving a selection of at least one of the linked models.
17. The method of claim 16, wherein receiving the selection of the at least one linked model comprises a user selecting an icon on a display representing the at least one linked model.
18. The method of claim 16, further comprising:
receiving input from a user to change at least one variable for the selected model; and determining at least one output based on the received input. 19. The method of claim 18, further comprising displaying at least a portion of the at least one output on a graphical display.
20. The method of claim 18, further comprising:
receiving input from a user to change the selected model; and
determining at least one output based on the received change to the selected model.
21. The method of claim 20, further comprising displaying at least a portion of the at least one output on a graphical display.
22. The method of claim 10, wherein at least one of the linked models is created in a different program than at least one other of the linked models.
23. The method of claim 10, wherein at least a portion of the information for at least a first one of the models is provided by a first user, wherein at least a portion of the information for at least a second one of the models is provided by a second user.
24. The method of claim 10, wherein at least a portion of the information for at least a first one of the models is provided by a first user over a network, wherein at least a portion of the information for at least a second one of the models is provided by a second user over the network.
25. The method of claim 10, further comprising reconciling at least a portion of the output from at least one of the linked models with at least a portion of the output from at least one other of the linked models.
26. The method of claim 25, wherein reconciling the at least a portion of the output from the linked models comprises a best-fit estimation.
27. The method of claim 10, further comprising creating at least one consensus model from at least two of the linked models.
28. The method of claim 10, further comprising the computer system optimizing at least a portion of the output for at least one of the linked models.
29. The method of claim 10, wherein at least one of the models comprises a cause and effect relationship.
30. The method of claim 10, further comprising sharing at least one of the models.
31. The method of claim 10, further comprising storing explanatory information in connection with at least one of the models.
32. The method of claim 10, wherein at least one of the models is linked to at least one other of the models by way of a Bayesian model.
33. The method of claim 10, wherein at least one of the models is linked to at least one other of the models using artificial intelligence.
34. A method, comprising: receiving, by a computer system, information relating to one or more models; generating, by the computer system, a public version of at least one of the models; providing access to a public version of to two or more users; and receiving input from at least one user for at least one of the models, wherein at least a portion of the input is different than the public version; and generating, by the computer system, a private version of at least one of the models based on the input received from the at least one user.
35. The method of claim 34, wherein at least one of the models is a crowd source model, wherein the public version includes input from at least two users.
36. The method of claim 34, wherein the at least one public version comprises at least one reconciliation of input from at least two different users.
37. The method of claim 36, wherein the at least one reconciliation comprises a curve fit.
38. The method of claim 34, wherein generating the public version of at least one of the models comprises:
providing one or more normative templates to a user;
receiving input from the user based on at least one of the normative templates;
generating at least a portion of the output for the public version based on the information received based on the at least one normative template.
39. A system, comprising:
a memory coupled to the CPU;
a decision support component configured to display information from one or more models; a decision tools component configured to provide modeling information from one or more models to the decision support component; and
a data stream component configured to provide information to the decision support component.
40. The system of claim 39, wherein the decision support component is configured to display information from two or more models.
41. The system of claim 39, wherein the decision support component is further configured to connect two or more models with one another.
42. The system of claim 40, wherein the decision support component is configured to automatically connect two or more models with one another.
1. Field of the Invention
The field of the invention generally relates to the creation, connection, and integration of models, user interfaces for models, and tools for collaborating on models.
2. Description of Related Art
Many tools have been developed to provide users with the ability to use models to solve complex analytical problems. An issue that confronts many users of these tools is that their complexity can be intimidating, limiting their utility. In addition, existing tools often lack the ability to share cause and effect relationships in a way that allows them to be readily accessible to the layperson. Existing tools often operate as standalone systems, and may not connect to one another or enable different models to be related to one another. Existing tools may also lack the ability to create a consensus model in a collaborative setting. In addition, existing tools may not store explanatory information associated with mathematic data and often do not provide a visual interface for working with mathematical models.
In various embodiments, a system and method may implement visually oriented object modeling. Systems and methods as described herein may provide a visual vocabulary and a user interface that provides users with visual tools to assist their understanding of underlying mathematic models.
According to one embodiment, a system for decision making includes a decision support component, a decision tools component, and a data stream component. The decision support component can display information from one or more models. The decision tools component can provide modeling information from one or more of the models to the decision support component. The data stream component can provide information to the decision support component. In some embodiments, the decision support component can connect two or more models with one another.
According to one embodiment, a method includes providing a user interface, receiving information relating to one or more models, and displaying a visual representation of at least one of the models in relation to at least one other model. The models may include a cause and effect relationship. In some embodiments, the models are shared. In some embodiments, one or more of the models are connected to at least one other model. In certain embodiments, one or more collaborations are received for a model. In various embodiments, a mark-up language may be used that readily allows users who lack training in the use of complex models to read and interact with such models in a manner that allows them to more readily access the solutions and outputs provided by such models.
BRIEF DESCRIPTION OF THE DRAWINGS
Advantages of the present invention will become apparent to those skilled in the art with the benefit of the following detailed description and upon reference to the accompanying drawings in which:
FIG. 1 illustrates one embodiment of a system for decision-making including a visually oriented modeling system;
FIG. 2 is a flow diagram illustrating one embodiment of implementing decision tools and decision support in a modeling system;
FIG. 3 is a flow diagram illustrating one embodiment of implementing data streams in a modeling system;
FIG. 4 illustrates a linkage of a system with a content management system that allows the storage of explanatory text and graphics according to one embodiment;
FIG. 5 illustrates an embodiment of a user interface that can be used to implement weight- indexed scoring;
FIG. 6 illustrates one embodiment of a display for a modeling system including multiple scenarios for a set of television shows;
FIG. 7 illustrates an embodiment of a display for supporting optimization;
FIG. 8a-8c illustrate displays of tabular results of various models according to one embodiment;
FIG. 9 illustrates a cause and effect graphical interface according to one embodiment; FIG. 10a illustrates a display of a zoom into Model for "Make or Break" according to one embodiment;
FIG. 10b illustrates a display of a zoom into "Portfolio Model" according to one embodiment;
FIG. 10c illustrates a display of a zoom into "Marketing Spend Optimization Model" according to one embodiment;
FIG. 11 illustrates a plot by a system that includes two models that both predict ratings;
FIG. 12 illustrates storing explanatory information with mathematical data according to one embodiment. While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. The drawings may not be to scale. It should be understood, however, the drawings and detailed description provide examples of the output of the invention, and that the drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but to the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present invention as defined by the appended claims.
DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS
In various embodiments, systems and methods are provided for handling the integration of models, and delivering a visual interface that allows one to zoom into a model, or zoom out and see the connection with other models. In various embodiments, a system and method implements visually oriented object modeling. In some embodiments, collaboration tools are provided that allow an individual to examine the consensus view of a cause and effect relationship versus their own individual view.
As used herein, a model may include a defined set of relationships between variables. The model itself can contain different scenarios (which are a range of input values that produce different output variables). In addition, a user may want to compare different models which may have similar input variables, and output variables, but has different relationships connecting variables.
In some embodiments, computer based simulation is used to communicate trade-offs, test what-ifs, understand likely outcomes and assess possible risks. In some embodiments, visually oriented mathematic models, and the ability to allow these models to easily connect with one another, are implemented. Various embodiments may allow users of the models to share results in a public space to create consensus views (in some embodiments, while maintaining their private models with their own sets of assumptions/inputs).
In some embodiments, the integration of models is handled by using new statistical forms and pattern recognition. A system may deliver a highly usable visual interface that allows one to zoom into a model, or zoom out and see the connection with other models.
In some embodiments, a system may include crowd source models. In some embodiments, a system allows multiple users to create a "normative" estimate for the system to use. The system may address how to deal with multiple different estimates for the same parameters, especially when some are better estimates than others (quality parameter) or some are more recent than others. Some may use different underlying cures (an S versus a Log may give similar answers at some place, and very different ones at the inflection points - even if both has the same overall "fit" to the data"). In some embodiments, the system deals with "averaging" different "curve" fits by pushing through inputs, to create outputs of the various models and then use SPLINE to "re-fit" the best fit off the output of all the different models. The system may allow a high level of user control so that one can maintain their own curve fit types, inputs and assumptions, or can use crowd sourced estimates. Because each model is a separate "object", the user can choose to use their own personal models in some areas, and crowd source models in other areas.
In some embodiments, a system stores meta data associated with mathematical models.
The system may plug into a content management system to save "textual" observations - such as best practice recommendations, or explanation between "good and bad" results, and a tie into optimization.
In some embodiments, a system and method includes ways of handling the integration of models and achieving a highly usable visual interface to "zoom" in and out of different connected models, while at the same time collaborating with others to examine the consensus view of a cause and effect relationship versus their own view. In various embodiments, a system and method may include or implement:
1. sharing cause and effect relationships in a complex system in a way that is accessible to the layperson.
2. automatically connecting models to one another (as a solution to the "stand-alone" problem, for example).
3. collaborating and creating consensus models automatically.
4. storing explanatory information with the mathematic data.
In some embodiments, a system for supporting decision making includes a graphical user interface that enable visually representing information from two or more models. In one embodiment, a system includes a decision support component, and decision tools component, and a data stream component. Each of the components may include one or more modules. Components a and modules of a system may be implemented in hardware, software, or a combination thereof. In some embodiments, each of the components in a system is implemented as a software layer. For example, a system for decision making may include a decision support layer, a decision tools layer, and a data stream layer. In some embodiments, the system is object-oriented. FIG. 1 illustrates one embodiment of a system for decision making including a visually oriented modeling system. System 100 includes visually oriented modeling system 102, client systems 104, and databases 106. Visually oriented modeling system 102 is connected to client systems 104 over network 108. Visually oriented modeling system 102 may access data on databases 106.
Visually oriented modeling system 102 includes decision support layer 110, decision tools layer 112, and data stream layer 114. Decision support layer 110 includes graphical interface module 116 and cross-model connection module 118. Graphical interface module 120 may generate a graphical user interface display on client systems 104 over network 108 and to local user systems (local user systems are omitted for clarity). In some embodiments, a visually oriented interface allows users to 'drill down' (i.e. double click) on objects to see the underlying data sets, model relationships, and/or decision support module input fields.
In some embodiments, two or more different models are connected or related with one another. Connection or integration of models may be carried out manually, automatically, or a combination thereof. In some embodiments, model cross-connection module 118 connects two or more models with one another. In one embodiment, a user specifies the inputs and outputs and maps the relevant inputs and outputs across models. The decision support tool then determines which technique or techniques to use to combine models (for example, agent based, or linear programming).
Decision tools layer 1 12 includes predictive modeling system 124. In some
embodiments, predictive modeling system 124 is integrated with agent-based rules or linear program optimizations.
System 100 includes database management system 140 and databases 142. Database management system 140 and databases 142 may be remote from visually oriented modeling system 102. Database management system 140 is coupled to network 106. In some embodiments, visually oriented modeling system 102 can access data on databases 142. Visually oriented modeling system 102 may use data from database 142 alone, or in combination with, data from databases 110.
Data stream layer 114 includes data exchange module 130. Data exchange module 130 may access databases 110 to serve as a global data exchange and repository.
In some embodiments, decision support layer 110 accesses base data repositories and feeds, and accesses the visual interactive decision support models in separate modules.
Clients 104 may each include user systems 150. User systems 150 can access visually oriented modeling system 102 over network 108. Network 108 may be, for example, the Internet. In some embodiments, user systems 150 access visually oriented modeling system 102 by way of a personal computer system with a browser.
FIG. 2 is a flow diagram illustrating one embodiment of implementing decision tools and decision support in a modeling system. Flow diagram 180 includes blocks 182 and connection arrows 184. In some embodiments, a flow diagram is displayed to a user. The user may click on blocks or connections of the display to view information concerning a particular element, model, or connection. For example, the user may click on the Marketing Spend Optimization block to view information concerning the Marketing Spend Optimization model, such as equations, models feeding the Marketing Spend Optimization model, and input data sources. As illustrated by the arrows, key decisions 186 may be fed by decision flow and tools 188 may be fed by data streams 190 and models 192.
FIG. 3 is a flow diagram illustrating one embodiment of implementing data streams in a modeling system. Flow diagram 200 includes data streams 202 and models 204. In some embodiments, a flow diagram of data streams for a modeling system is displayed to a user. The user may click on blocks or connections of the display to view information concerning a particular element, model, or connection.
Table 1 shows an example of data stream content for models according to one embodiment.
What Description Who Provides When
Program Analysis of ratings and Research Quarterly as part of marketing Tracking audience trends measuring Department planning
the strength and momentum
Celebrity Q- Analysis of the appeal of Direct feed. Annually updated, available S cores celebrities among different automatically within the population targets modeling system
Profit Model Breakdown of the revenue Marketing & Updated annually as part of sources from different Finance business planning
programs, included a ratings
forecasting model from ad
Marketing Regression Model includes a May be provided Updated quarterly
Mix Model ratings forecasting directly in the
component, though the modeling system.
primary purpose is budget
Creative Quantitative pre-testing of Research Ad Hoc, as needed. Results Testing advertisements (primarily Department, loaded into the modeling television ads) upon request system by research
from marketing department.
In- Three analysis are included Research Within six weeks after
Market/Post in this class of research: 1) Department for premiere.
Premiere actual ratings versus every show
Evaluation forecast, 2) actual situation
factors versus forecast and
3) optional tracking of
consumer awareness, intent
to tune, etc. Model Integration
Referring again to FIG. 1, at decision tools layer 112, the system may use various techniques for modeling, and various techniques to integrate models together. Each of the models may draw from decision tools feeds and arrays in data streams layer 114. In one embodiment, system 100 implements one or more of the following:
1. Multivariate statistical model
2. Agent Based Modeling (for example, as a basis of connection)
3. Linear programming optimization (for example, linear programming optimization of independent curves, footed to consistent time scales)
4. Process flow integration
5. Weighted Indexed Scoring
Embodiments of techniques for modeling, and integrating models with one another, are described further below.
1. Multivariate statistical models
In some embodiments, multivariate statistical models are used to form an underlying system of equations between visual objects in the modeling system. The multivariate models may be, for example, models of consumer response. The response may be in the form of time series data or cross sectional data. In a marketing context, time series models may be historical marketing mix models, and the cross sectional models may be consumer-based respondent models.
As an example, a statistical model of the economy may be connected by the system to a model of consumer behavior in purchasing a new car. The historical time series model of the economy may be a regression model. The regression model for the historical time series may include variables to predict disposable income, wealth distribution, gas prices, and commodity prices based on a wide range of economic and geo-political variables, such as the presence of war in key regions, economic growth rates of countries such as China, unemployment rates in the US, etc. The consumer response model may also be a regression model. In the regression model for the consumer response, multivariate analysis may be used to predict the choice of car someone will purchase based on the price of the vehicle, the type of the vehicle, the miles per gallon of the vehicle, the disposable income, gas prices, amount of advertising, and type of advertising.
2. Agent-Based Modeling. In some embodiments, a modeling system implements agent-based modeling. Agent
Based Modeling ("ABM") may include computational techniques that use non-statistical techniques to program in sets of rules in how "agents" behave and interact. In some embodiments, ABMs are embedded in a visually oriented modeling system in the form of rule sets at the decision tools layer.
As one example, for the statistical model types (time series and cross sectional described above), the system may integrate these models as follows:
The statistical models may be converted to an ABM by identifying the mechanism of interaction in the statistical model. These interactions may be programmed into the agents, and simulations run until the central tendency of the most common outcome of the ABM aligns with the statistical model outcome. The ABM, at this point, may provide similar output as the statistical model in terms of predicting outcomes such as disposable income, gas prices, commodity prices etc. The outcomes now, however, can be generated by agents. In some embodiments, the agents have additional characteristics.
In some embodiments, ABM rules sets are taken from, and integrated with, models using different units of analysis. For example, a "most granular common denominator" may be identified in both models such that the models can be connected. In this case, a "consumer" agent, which represents the "disposable income" variable in the first model, and makes an automotive purchase selection in the second model is identified as the "most granular common denominator". The system may integrate the model such that the agent with the disposable income also makes the purchase selection. With the models integrated, changes in the one model can now influence outcomes in the other. For example, if war breaks out, in the "economy model", it may be immediately in the system how the outbreak directly influences vehicle selection.
3. Linear Programming.
In some embodiments, a system uses linear programming (LP) optimization of independent models to create an integrated model.
As an example, a pharmaceutical company may have several different business/marketing models. Each of the models may be designed to analyze marketing spending, and the influence of new prescriptions and renewal prescriptions. Some of the models may use statistical regression to evaluate the relationship between media spending and media weight in direct-to- consumer advertising to new prescriptions and renewal prescriptions. Another model may use exposed and control "design of experiments" - for example, assessing the difference in new and renewal prescription between those getting a email newsletter versus those getting control treatment of no email newsletter. Still other models may use expert estimation of the likely outcome. For example, estimating the right investment to make in adding "detailing" staff to periodically brief doctors on new studies related to the prescriptions. In some embodiments, expert estimation may be implemented in a spreadsheet that applies a rule-based estimate of additional prescriptions per detailing staff.
In one embodiment, each of the models described above are implemented in linear programming optimization models. In the special case where the diverse models all produce the same outcome variable (in this case, new prescriptions and renewal prescriptions), linear programming may result in a more efficient system model integration. (In another embodiment, the models are converted into an ABM simulation, as described above. ABM may, however, be computationally intensive compared to linear programming optimization models.)
Each model may be run such that the outcome is scaled to a consistent time period (sales of new and renewal over a month, for example). Each model is run to generate "outcome predictions" over a range of spending. The outcome predictions from each different model may then be analyzed by the system to generate a new set of response functions, using linear programming. The models may be combined into a single model, for example, to allow for integrated media planning.
4. Process Flow Integration
In some embodiments, a system uses process flow integration to create an integrated model. As an example, a TV network may have several models that influence decisions such as: whether to launch a new TV show, or not; how much to spend behind a TV show; and how much money the network can expect to make back from the TV show in terms of advertising sales. In some embodiments, integration of models includes a process flow in which certain decisions over-ride all other decisions. For example, in the process flow, the CEO of the company may strategically decide that it is important to keep a certain star or director happy by funding marketing support of a TV show that otherwise would not merit such spending (based purely, for example, on the analytic models predicted ratings — or the analytic models predicting responsiveness of consumers to advertising.) A "make or break" decision may precede the evaluation of the rest of the integrated models in a hierarchical basis. FIG. 4 illustrates one example of a process flow integration with over-rides. In the example shown in FIG. 4, "make or break" decision model 210 feeds into the marketing spend priorities, and comes before (in a way that over-rides) Portfolio model 212. Portfolio model 212 comes before (and constrains the options) of Marketing Spend model 214.
5. Weighted Indexed Scoring In some embodiments, a system integrates models using weight indexed scoring. Weight indexed scoring may be used in cases where there are a few different outcome variables. For example, the variables may be: (1) a TV show's strength relative to other TV shows, (2) the TV show's fit with the TV Network and (3) the profit potential of the TV show. Weighted Indexed Scoring may be computationally more efficient than ABM, and visually easier for business managers to understand.
FIG. 5 illustrates an embodiment of a user interface that can be used to implement weight indexed scoring. Display 240 includes graphs 242 and panel 244. Panel 244 includes sliders 246 and input boxes 248. Sliders 246 and input boxes 248 may serve as input devices for different variables. Each of the sliders can be positioned from 1 to 5. The data for all of the variables in a model may be indexed within a 1 to 5 range (any range can be used, with 1 to 5 used in this case and 0 to 100 being another index choice). The weighting may refer to the degree to which each factor contributes to the total score on the dimension of strength, and on the dimension of fit. Weighted total 250 may be determined from all of the inputs for a category (in this example, strength or fit).
In some embodiments, outputs for user-inputs on a display are dynamically determined and displayed. For example, in the embodiment shown in FIG. 5, profit may be a third dimension. Profit may be generated from a statistical model. The system may bring profit into an integrated model. In the embodiment shown in FIG. 5, profit is represented by the size of each of bubbles 256 on X-Y graph 258. As various inputs are changed by the user (for example, by the user moving sliders 246), or based on changes to the inputs from external data for the models, the outputs (in this case, profit), may be altered. For example, if a user moves the Wide Appeal slider for Sinbad from 1 to 5, the size of the bubble representing profit corresponding to Sinbad may increase the size.
In some embodiments, a modeling system brings together human judgment input and modeled input. The mixed inputs may be used in decision making. In certain embodiments, human judgment input and modeled input are normalized on a common scale. For example, in the embodiment shown in FIG. 5, the locks may represent variables that are generated by models, and then translated into a 1-5 indexed score (see the visual interface configuration below). The variables without locks may be generated by human judgment. Human judgment may be easier to estimate using a scale (in this case 1 to 5) and in a "comparison" framework.
In some embodiments, a modeling system presents a comparison of two or more scenarios in a single display. FIG. 6 illustrates one embodiment of a display for a modeling system including multiple scenarios for a set of television shows. Display 280 includes scenarios 282 and 284. Each scenario and show (for example, Sinbad Scenario 1, Sinbad
Scenario 2) may be associated with a set of sliders 286 and input boxes 288.
Although two scenarios are shown in FIG. 6 for illustrative purposes, a system may, in various embodiments, model and display any number of scenarios simultaneously.
In some embodiments, a display the images on a display graphical are arranged or selected to show the importance of a particular factor in the model. For example, in the embodiment shown in FIG. 5, the weighted score may be graphically shown to the right, such that those factors that matter more to the outcome have a higher weight, and a longer bar chart compared to those that matter less. The list of inputs is also rank ordered based on the weight score.
FIG. 7 illustrates an embodiment of a display for supporting optimization. Display 300 includes input area 302, graphs and tables area 304, and block diagram 306. A user may change inputs in input area 302. The system may automatically update graphs and other information in graphs and tables area 304. The user may also click on boxes or arrows in block diagram 306 to access information on factors that influence the ratings.
In some embodiments, the thickness of arrows or distance from ratings is used to show the importance a factor. For example, those factors with thicker lines or closer to the ratings may be relatively more important than other factors.
In some embodiments, multivariate regression models, ABM, or linear programming are used in conjunction with Process Flow. In this case, the two different rating models may both use statistical techniques, and predict the same outcome variable (ratings). In some embodiments, the system combines the ratings models using linear programming. In another case, a Portfolio model uses a wider range of outcome variables to determine how well a TV show fits with the networks brand and the aspiration of the executive for the network brand. The models that feed the Portfolio model can be combined with ABM. The models can also be integrated using a weighted-indexed scoring.
Cause & Effect Graphical Interface (CEGI)
In some embodiments, a graphical interface for a system includes a Cause & Effect Graphical Interface ("CEGI"). The interface may make the visualization of cause and effect data models easy to see, navigate, edit, and examine what-if scenarios (based on changing different inputs).
The following examples illustrate a company's marketing mix modeling using non-web- based approaches. Further examples illustrate using web-based visual CEGI. For context, the users of the tool may be marketing executives that have to make decisions about budget allocation, and forecast the return on investment of various marketing efforts.
In some existing systems, the visualization of marketing mix models has only been delivered in graphical presentation programs and spread sheets, but this has been limiting in that users may not be able to easily share models, and put in different inputs, or compare scenarios. Users cannot interact with their models to see the results of changing assumptions. In some embodiments herein, a system puts complex information into a web based application to deliver a dynamic user interface.
In some embodiments, a CEGI shifts from a spread sheet and/or statistical package math output to a web based interface. In certain embodiments, the user can build a chart and add labels. In certain embodiments, only cause and effect charts are used, connecting them to the math formulas so a user can see the cause and effect relationship directly. The system may provide a graphical way to show how variables relate to the outcome.
In some embodiments, a system allows a user to:
1. See a measure of the importance of the variable in the overall model
2. Drill down and see the more details of a relationship (response curve and interaction terms)
3. Change the assumptions of impact, cost, etc and see the outcome of changes (such as a bell distribution that shows the range of outcomes and their probability)
4. Add/delete variables
In certain embodiments, a system allows a user to add in:
1. Size of impact of "marketing object" (size of a box relative to others)
2. Strength of relationship (width of an arrow)
3. Interaction effects (arrows among circles)
EXAMPLE 1 : Marketing Mix Models For Pharmaceutical Company
Current output is in a spreadsheet, and shows one variable causing an impact on another. For example, in this output, it may be found that online advertising may influence revenue, based on a design of experiments analysis (illustrated in FIG. 8a). FIG. 8A illustrates a display 70 of results including table 72 and x-y graph 74. In a completely separate model, it may be found that Television advertisements for the brand impact profit, based on an econometric statistical analysis (illustrated in FIG. 8b). FIG. 8B illustrates a display 80 of results including table 82 and x-y graph 86. In a separate model, there may be the conversion of someone searching online, and clicking over to the website, and then subsequently buying the product - in this case, getting a prescription from a doctor (illustrated in FIG. 8c). FIG. 8C illustrates a display 90. In this example, however, it is not shown whether TV advertising increases the propensity to search, or how the two might be related. The two activities are not connected, because the models are separate.
Furthermore, the relationship between the amount of TV advertising and the amount of profit is shown mathematically as a log function. But, the implications of the formula and the difference between an s-curve or linear relationship may not be altogether clear to a non- mathematician.
FIG. 9 illustrates one embodiment of a CEGI. This example may be contrasted with examples shown in FIGS. 8a-8c. FIG. 9 includes outcome summary 322 and model relationship diagram 320. In FIG. 9, the relationships are shown graphically, with the user able to quickly see and specify relationships, add data (and have the system calculate the relationships automatically) and compare models with others. Furthermore, the linkage into the content management system may annotate relationships to provide the explanation of what a log function means to the layman in terms of making marketing spend decisions.
The "What You See Is What You Get" (WYSIWYG) interface may be designed for business people who can define data relationships, and graphically see them, but are not mathematicians.
By clicking on the graphical relationships, to test different scenarios/value the user can see how the outcome changes.
In some embodiments, the system creates a collaborative tool that multiple users can access and edit and see the different assumptions and outcomes based on different people's assumptions, as well as a roll-up of the consensus assumption. A CEGI shows the range of different inputs, and the outcomes.
In an illustrative embodiment, elements of the Cause & Effect Graphical Interface "CEGI" include:
1. Login Page
2. Home Page Work Space
i. Normative Templates, predefined by category, and saved models ii. Editing (Add/Delete/Change)
iii. Layer explorer
3. What If Scenario (Set-up/Output) and Optimization 4. Role Management
5. Admin/ User Home Page
i. Saving models as "private" versus "sharable"
1. The login page may contain the following elements:
1.1 Field for user name
1.2 Field for password
2. Home Page Work Space
2.1 The home page may provide easy navigation to the elements needed to build the cause and effect model. Building a cause and effect model can start from selecting a pre-defined model, or starting from a blank canvas. Therefore, the options include...
2.1.1 Selecting a pre-defined template, and then editing it.
2.1.2 Or, selecting from "public" models and editing it.
2.1.3 Or, building a new model from a blank canvas -
As users work on a model, they can edit the value within a model by creating new scenarios, or they can edit that model itself, by creating different cause and effect connections between input and output variables.
2.1.4 Edit Value/ Create new scenario
2.1.5 Edit Model/Create new cause and effect relationships between variables
The user can then save the model in the user's own private list, or choose to share it as a public model.
2.2 Normative Template, predefined by category, and saved models
1. A pull down of predefined normative templates may be available. These may be labeled "Normative Models"
2. A pull down of saved models may be available. These may be labeled "Saved Models".
The saved models may be subject to viewing based on the user settings (which default to "private")
3. The initial list of pull down options may include, for example, "Automotive, Pharma, CPG).
4. The user may have the option of saving their model to the "Saved Model list"
5. As the user scrolls over the list of model objects, an algorithm matches to other similar models and recommends potential additional models to plug in or connect.
2.3 Editing the Model (Add/Delete/Change) 1. The user is able to add/delete or change a model. Note: This is NOT the same as simply changing values within an existing model. (To see changing values within a model, see scenarios).
3. What-if Scenario and Optimization
1. The user can create, edit, delete "scenarios". A scenario is the same model, but with different input value, for different objects within the model. For example, a user might want to compare the amount of profit that the model predicts based on spending $1 million in TV advertising versus spending $10 million in advertising.
2. Optimization is having the tool itself produce the highest output possible for the least input. Optimization is a specific feature of scenario, where the system produces the input and output value, rather than the user editing the input values to see "what-if I change this value.
3. Within the tool, the user may be able to zoom in, and explore the model.
4. Role Management— This page may allow the user to assign certain functionality to the user to determine what functionality they will have access to when they login.
1. Admin User— Admin has access to view everything, edit anything, approve changes to models.
2. Account User— Access to view/edit everything related to them (their specific
5. Admin/ AE User Home Page— When an internal user logs into CEGI, they may have a
different home page that allows them to get to the following information:
1. Access Role Management
2. View any of the reports for a specific account. The Admin user would select a
specific account and be able to see everything the account user is able to see.
In some embodiments, a CEGI turns what has existed in mathematical formulas into an easy to use graphical interface. The CEGI may include algorithms to create curves, and to integrate different models. In contrasting FIG. 9 with FIGS. 8a-8c, it may be observed that the system shown in FIG. 9 has combined several different models. In the spread sheets, there was a separate model for TV and base volume, a separate model for Online banners, a separate model for website, a separate model for search, and a separate model for direct mail and email. These models made their own estimates of their impact on profit, but did not interact dynamically with each other. In some embodiments, a system uses various algorithms to synchronize and fit impact to outcome variables (in this case, profit). A second attribute of the system illustrated in FIG. 9 is the graphical interface that may make the relationships easy to see. They are represented by arrows, and the user can click on the arrows and see the underling shape of relationship (linear, exponential, log, s-curve, u-shaped, etc.). The CEGI also allows the user to see the range of estimates from crowd-sourced data (in this case, it is shown as a bell distribution, and the user's estimates are slightly higher than the consensus estimates. The user can easily change the estimate using the sliding arrow). While the interface itself may change, the features of combining models providing a graphical interface of cause and effect relationships, and integrating crowd sourced data and individuals estimates may be seen. Not shown in FIG. 9 is the integration of the system with content management system. (That feature is shown in FIG. 5, however.)
Examples of how a user might use the output of the system include the following: In this example, television advertisements for the brand impact three different objects. Television advertisements may impact search (more people search after seeing an ad for this particular brand) and television advertisements may impact website visits (some people read the URL in the television ad, and directly visit the website). Television advertisements may also directly impacts profit (some people directly buy the product as a result of television advertising). In addition, the relationship between the amount of television advertising and the amount of profit is shown in the curves, which can be seen connected to the arrow between television and profit. The relationships are shown graphically, with the user able to quickly see and specify relationships, add data (and have the system calculate the relationships automatically) and compare models with others.
In certain embodiments, a system connects meta data as part of the model "object". These meta data can relate to explanation of the type of relationship between two variables (such as a linear relationship versus an s-shape curve). Or, they can be the difference between a high and low value on a particular input, or the underlying quality of the input data. For example, for a marketer using the entertainment system framework, online advertising might show a bifurcated range of impact. In addition, the user might want to know what the difference is between a lower modality (very little impact) and a higher modality (strong impact). FIG. 9 shows an example of how the linkage of a system described above with a content management system allows the storage of explanatory text and graphics linked to specific aspects of a model.
EXAMPLE 2: Entertainment Company Program Promotion Decision Making
To further illustrate, the following example is based on the entertainment category where there are several different models that are not integrated, and are not easy for the non- mathematician business person to use. In this embodiment, the system may include: 1. sharing cause and effect relationships in a complex system in such a way that is accessible to the layperson.
2. automatically connecting models to one another (as, for example, as solution to the "stand-alone" problem.
3. collaborating and creating consensus models automatically.
4. storing explanatory information with the mathematic data.
In some embodiments, a node chart is used to show the connection of different disparate models. In some embodiments, integration algorithms are used to connect the model. The user can "zoom" into any of the models to enter value, or to run different scenarios.
FIG. 10a illustrates a display of a zoom into Model for "Make or Break" according to one embodiment. Table 340 includes a tabulation of "make" considerations and "break" considerations.
FIG. 10b illustrates a display of a zoom into "Portfolio Model" according to one embodiment. Display 360 includes tables 362 and x-y graphs 364.
FIG. 10c illustrates a display of a zoom into "Marketing Spend Optimization Model" according to one embodiment. Display 380 includes tables 382 and graphical display 384.
In some embodiments, the system allows a user to store the model in its native form (as shown in FIGS. 10a, 10b, and 10c). In addition, the system may allow the user to use the CEGI (such as described in the previous example) to make the mathematical relationships easier for the layman to see and understand at a glance.
In some embodiments, the system connects models to one another. In various embodiments, configuration methods, automated methods, or combinations thereof, may be used by a system to link models. Once linked, a user can make changes in one model and see the changes cascade across the other models. The user can also choose to create scenario so that a range of inputs can be considered and evaluated. Finally, the user can turn to the system to optimize the system and can work across multiple models to find the ideal inputs to maximize the desired output.
The system may implement different methods to reconcile different models estimating the same value, as in this case with Ratings. In FIG. 9, for example, Ratings is estimated in the Marketing Spend Optimization Model, as well as in its own separate ratings model (managed by different departments within the organization). The user can specify whether these two values should be left separate (generating two scenarios) or reconciled. In the event the user selects reconciliation, the system may automatically run iterations to the ranges of convergence and divergence and the best estimation fit among the models.
In some embodiments, pattern recognition includes a Linear Programming fitting process using SPLINES. By plotting outputs and refitting a new equation using linear programming, a pattern of output from one model may be recognized as a new equation. The new equation may be integrated with other models.
In some embodiments, a modeling system allows for a determination of outcomes based on an iterative process. For example, in some embodiments, a system may allow several inputs to be changed by a user. As the input variables are changed, the live rating achieved (for example, the top right corner of FIG. 5 may change accordingly. In some embodiments, a modeling system records the outputs and the inputs and plots them. The system may iterate through possible combinations to plot a range of input/outputs. In some instances (for example, when the system is comparing multiple models that are measuring the same outcome and many of the same inputs), the system may create a map of the situations that produce similar outputs, and the situations where the outputs diverge. Since each input may be its own dimension, the calculation may use dot product distances to measure convergence versus divergence.
FIG. 1 1 illustrates a plot by a system that includes two models that both predict ratings. One of the models produces a set of points 400. The other model produces a set of points 402. Both models may use the time of day when the program airs as part of the model. While the models may generally converge, there may a wider divergence when it comes to the prime -time hour (the peak area). In this case, the model that produces points 400 suggest that prime time alone has a larger impact on the actual ratings. The model that produces points 402 indicates that prime time is important, but not as important as the other model. Such deviation may occur, for example, in a case where points 402 are produced from a model that includes marketing spending separately from the model for points 400. Thus, the plot may reveal that if marketing spend is separated out, and advertising is not used to support television programming, the prime time ratings will be lower.
In some embodiments, a graphical display allows users to add additional variables and draw connections to one or more outputs. As one example, the opportunity to allow a variable may included in the marketing spend optimization tab with the ratings node shown in FIG. 7. For example, a comedy channel may want to add a variable for "celebrity". Celebrity is not a factor included in the normative model for entertainment. To add the celebrity variable, the user may click on a "+" symbol and answer some questions about the new variable. For example, the user may specify the expected relationship that the variable has on the outcome variable (in this case, ratings). The user would provide some data points for the celebrity score, and outcome ratings, as well as other relevant data details. From the data provided by the user, the system may determine the best method for integration. The celebrity variable may be included as part of an integrated model.
In some embodiments, a system uses a normative model as a starting point (for example, before users edit the model to their own needs). Users can edit the normative model to address the user's particular needs. The normative model may be based, for example, on what influences ratings for TV shows. If a user is in charge of marketing for a food channel, the user may add new variables related to diet trends (Southbeach, low-carb, etc.). In addition, such a user may add in celebrity scores, since the chefs have an audience draw based on the strength of their celebrity.
In some embodiments, a visually oriented modeling system is used for simulation based on user input. For instance, an automotive marketer might want to ask "what if the economy gets worse" or "what if gas prices fall $0.20 a gallon" or "what if we improved the creative strength of our advertising". For example, referring to FIG. 7, a user may change inputs and evaluate the effect of the changes in input on the outcomes. In addition, the user can graphically select factors to assess the effect of the factor on an outcome. For example, in the embodiment shown in FIG. 7, a user may click on situational factors or marketing factors to see additional variables that can be changed to see how the factors affect the outcome.
In some embodiments, a modeling system may be connected to a user model. For instance, a user may develop their own statistical model for predicting ratings. In certain embodiments, the user-generated model is connected as a black box. The modeling system may not have the actual equations for how inputs into the user model produce outputs. Instead, the modeling system may provide inputs to the user model and read the outputs from the user model. In some embodiments, inputs are made and outputs read by way of an API. The modeling system may create a plot of a range of possible inputs and the corresponding outputs. In some embodiments, the modeling system fits its own mathematical model to the input/output relationship. In one embodiment, system may read the output and fit a curve (for example, using linear programming). In certain embodiments (for example, in cases where there are a high degree of interactions among variables), the modeling system uses a neural network. The neural network may determine how to reproduce similar output so that the modeling system can incorporate the user model into the system.
In some embodiments, a system implements crowd sourcing. In some embodiments, crowd sourcing includes the system gathering and automatically combining human judgment input. For example, in the embodiment shown in FIG. 5, items without locks may be fed by human judgment. In one embodiment, a decision support tool aggregates and averages multiple human judgment inputs (crowd sources) (for example, various users 250 shown in FIG. 1).
Based on the multiple human inputs, the system may determine a likely average. As another example, a user may over-ride the current pricing on media (in the marketing factors link, within the graphic). The changes may be stored and used to calculate new estimates of CPM based on the pricing trends gathered through human input (for example, crowd sourcing).
In some embodiments, a modeling system creates a consensus model. The consensus model may be created by running a range of scenarios within two or more models and plotting the outputs as a distribution. The distributions may be overlaid to compare similarity and differences. The system may choose the most common (for example, consensus) outputs where there is general agreement on the outcome. The system may note where models diverge widely (for example, outside of a pre-determined limit) in the expected outcome. Areas with divergence can be flagged as higher risk situations with less predictability.
In certain embodiments, the system stores and shares models in a public space, while maintaining one's own private version of a model (or the values within a model). This allows different people working in the same field to benefit from the overall "consensus view" without revealing individuals own data (or having access to others individual data). This aspect may encourage collaboration and make creating consensus models more efficient.
In certain embodiments, the system can store explanatory information with the mathematic data. Figure 12 illustrates storing explanatory information with mathematical data according to one embodiment. In this example, the person examining online advertising impact curve will get concise explanation of "best practice checklist" as well as "worst practices" to explain the two bifurcated distribution. This process may not be automatic. In some embodiments, a notation feature is included in a system to provide context explanation. By conceiving of models as objects, the system may, in some embodiments, expand the capability of attaching meta data to mathematical models.
Therefore, if the marketing mix model the user was examining included magazine, and the program is a returning series, the marketer may get a different set of text recommendation than if it was a new TV series and only included TV and Online advertising. In this way, a system may connect context-specific text to the user as situational inputs are adjusted.
In another example embodiment, a middle school student may obtain a model from the public domain, add their opinion of how things might work differently, and see the impact on the output (which may occur almost immediately). The system may achieve user friendliness, and availability of "public" models, that can include "private" inputs, or curves, so that one can test hypothesis, or upload their own data (e.g. while it might get aggregated into the crowd source norm, a user may maintain his or her private figure rather than the norm figure).
In certain embodiments, a system stores and shares models in a public space, while maintaining one's own private version of a model (or the values within a model).
In certain embodiments, information that can be represented as a system of interactive equations can be uploaded into a system. This can range from marketing mix models, as the examples in this document have shown, to weather patterns, to fantasy baseball team draft selection models. Mathematical models and their dynamic manipulation may be made much more accessible to every person.
In an embodiment, a form of Bayesian modeling and artificial intelligence is used to connect models. Bayesian modeling may include using prior estimates to influence a prediction. In one embodiment, a normative model (the priors in the Bayesian model) may be used to manage predictions.
In some embodiments, a system may make complex analytics more accessible to a layman by re-imagining mathematical models as objects that can be designed from the beginning to work with other objects (models). This makes it easier to integrate disparate models, to attach meta data (from a content management system) and to allow users to share information ranging from whole model objects, to specific data values, or data relationships.
Embodiments of the system and method described herein may have an advantage in being relatively easy to use for the layman. In some embodiments, however, the system may not include a statistical engine in and of itself. Users may develop their own model "objects" using their own statistical packages. The users can then upload these models into the system and connect them with other models, and can compare their models to others who are modeling the same phenomenon. The system may create consensus views using an integration algorithm. The system may include a focus on interactive and visual modeling to assist the user in more rapidly developing and improving their own hypothesized relationships, a complete system of interactive models - previously visually and analytically separate.
In some embodiments, a user interface includes a navigation element. For example, in the embodiment shown in FIG. 7, display 302 includes navigation slider 308. Although navigation slider 308 is shown only in FIG. 7 for clarity, a navigation element may, in various embodiments remain visible in all of the displays of a system (for example, FIGS. 5-7). In some embodiments, navigation slider 308 is displayed to the left of each of the various displays. In various embodiments, the systems and methods described herein can be used in a wide variety of applications and fields, for example, wherever a mathematical relationship is known, or can be estimated. As used herein, a mathematical relationship includes a "cause and effect" as in, for example, spending more on TV advertising causes sales to increase. There are many forms the relationship can take, and in various embodiments the system may be designed to handle any linear, or non-linear functional form. In addition, the system may be designed to integrate and connect different models.
In some embodiments, the system is used in business and social science settings where multiple independent models exist, along with opinion (which can be estimated in terms of mathematical relationships). Because the systems and methods may include collaboration aspects, they may also be used where a person would like to compare their perspective to that of others.
Although various systems and uses described above relate to marketing applications, the systems and processes described above can be used for procedures and practices in any field or endeavor. Examples of other applications for a visually oriented modeling system include manufacturing, medical care, transportation management, health promotion programs, and disease prevention programs.
Computer systems may, in various embodiments, include components such as a CPU with an associated memory medium such as Compact Disc Read-Only Memory (CD-ROM). The memory medium may store program instructions for computer programs. The program instructions may be executable by the CPU. Computer systems may further include a display device such as monitor, an alphanumeric input device such as keyboard, and a directional input device such as mouse. Computer systems may be operable to execute the computer programs to implement computer-implemented systems and methods.
A computer system may allow access to users by way of any browser or operating system.
Embodiments of a subset or all (and portions or all) of the above may be implemented by program instructions stored in a memory medium or carrier medium and executed by a processor. A memory medium may include any of various types of memory devices or storage devices. The term "memory medium" is intended to include an installation medium, e.g., a Compact Disc Read Only Memory (CD-ROM), floppy disks, or tape device; a computer system memory or random access memory such as Dynamic Random Access Memory (DRAM), Double Data Rate Random Access Memory (DDR RAM), Static Random Access Memory (SRAM), Extended Data Out Random Access Memory (EDO RAM), Rambus Random Access Memory (RAM), etc.; or a non- volatile memory such as a magnetic media, e.g., a hard drive, or optical storage. The memory medium may comprise other types of memory as well, or combinations thereof. In addition, the memory medium may be located in a first computer in which the programs are executed, or may be located in a second different computer that connects to the first computer over a network, such as the Internet. In the latter instance, the second computer may provide program instructions to the first computer for execution. The term "memory medium" may include two or more memory mediums that may reside in different locations, e.g., in different computers that are connected over a network. In some embodiments, a computer system at a respective participant location may include a memory medium(s) on which one or more computer programs or software components according to one embodiment may be stored. For example, the memory medium may store one or more programs that are executable to perform the methods described herein. The memory medium may also store operating system software, as well as other software for operation of the computer system.
The memory medium may store a software program or programs operable to implement embodiments as described herein. The software program(s) may be implemented in various ways, including, but not limited to, procedure-based techniques, component-based techniques, and/or object-oriented techniques, among others. For example, the software programs may be implemented using ActiveX controls, C++ objects, JavaBeans, Microsoft Foundation Classes (MFC), browser-based applications (e.g., Java applets), traditional programs, or other technologies or methodologies, as desired. A CPU executing code and data from the memory medium may include a means for creating and executing the software program or programs according to the embodiments described herein.
Further modifications and alternative embodiments of various aspects of the invention will be apparent to those skilled in the art in view of this description. Accordingly, this description is to be construed as illustrative only and is for the purpose of teaching those skilled in the art the general manner of carrying out the invention. It is to be understood that the forms of the invention shown and described herein are to be taken as examples of embodiments. Elements and materials may be substituted for those illustrated and described herein, parts and processes may be reversed or omitted, and certain features of the invention may be utilized independently, all as would be apparent to one skilled in the art after having the benefit of this description of the invention. Changes may be made in the elements described herein without departing from the spirit and scope of the invention as described in the following claims. The words "include", "including", and "includes" mean including, but not limited to.