AND NETWORK ACTIVITY

BACKGROUND

[0001] The present application discloses technology which is used to help a business keep a computer based production environment operating efficiently and with good performance. The "production environment" could be any of many different things. In some instances, the production environment could be a networked system of computer servers that are used to run an online retailing operation. In another instance, the production environment could be a computer system used to generate computer software applications. In still other embodiments, the production environment could be a computer controlled manufacturing system. Virtually any sort of production environment that relies upon computers, computer software and/or computer networks could benefit from the systems and methods disclosed in this application.

[0002] As computer-based production environments scale up and become larger, performance can decline. It becomes increasingly difficult to keep all portions of the system operating efficiently. There are many software

applications that have been designed to monitor a production environment, and to report on key metrics and events. However, the data and reports generated by such monitoring applications can themselves be difficult to comprehend. It can be difficult to use such data and reports in a meaningful manner to restore peak performance. Also, when problems and issues arise in such a production environment, it can be very difficult for a system administrator to identify the root causes of the problems or issues based on the data and reporting provided by such a monitoring application.

For all the above reasons, there is a need for additional technology that can monitor the activity in a production environment, and identify the root causes of problems and issues as they arise. There is also a need for technology that can proactively identify problems as they arise, and which can take steps to mitigate or solve the problems without the need for human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 is a block diagram illustrating various elements of a production environment assistant;

[0004] Figure 2 is a block diagram illustrating various elements of a data collection unit;

[0005] Figure 3 is a block diagram illustrating various elements of a data collection and transformation unit;

[0006] Figure 4 is a block diagram illustrating various elements of a metrics unit;

[0007] Figure 5 is a block diagram illustrating various elements of an evaluation unit;

[0008] Figure 6 is a block diagram illustrating various elements of an incident unit;

[0009] Figure 7 is a block diagram illustrating various elements of a notification unit;

[0010] Figure 8 is a block diagram illustrating various elements of an active inspector system;

[0011] Figure 9 is a block diagram illustrating various elements of a remediation unit;

[0012] Figure 10 is a block diagram illustrating various elements of a user interface system;

[0013] Figure 1 1 is a flowchart illustrating steps of a method of collecting data from client systems;

[0014] Figure 12 is a flowchart illustrating steps of a method of storing received client data into various data repositories;

[0015] Figure 13 is a flowchart illustrating steps of a method of calculating various metrics from collected client data; [0016] Figure 14 is a flowchart illustrating steps of a method of analyzing data to determine if an incident has occurred;

[0017] Figure 15is a flowchart illustrating steps of a method of reporting incidents that have occurred;

[0018] Figure 16 is a flowchart illustrating steps of a method of actively monitoring a client's systems to acquire data and to determine whether a predefined incident has occurred; and

[0019] Figure 17 is a flowchart illustrating steps of a method of taking remedial action to correct problems or issues with a client's system.

DETAILED DESCRIPTION

[0020] Figure 1 illustrates various elements of a production environment assistant 100 which receives or obtains data from a client's production

environment, which analyzes that data to determine whether issues or problems may be occurring, and which reports on any identified problems or issues. The production environment assistant 100 may also take remedial action to cure or mitigate such issues or problems.

[0021] The production environment assistant includes a data collection unit 200 which is responsible for receiving or obtaining data from a client's production environment. The data collection unit 200 would typically receive data via application programming interfaces (APIs) which have been installed and configured on the client's systems. The APIs would be configured to

automatically send certain types of data to the data collection unit 200 on a periodic or continuous basis. The data being sent by the APIs to the data collection unit 200 could include data points representative of various

measurements of a client's production environment, as well as event data relating to events which have occurred on the client's production environment.

[0022] The data could relate to operations performed by computer applications or programs, to the computer systems and networks themselves, and also other data related to the client's business. For example, the data being reported to the data collection unit 200 could include statistical data or information relating to business activity occurring on the client production environment, such as information relating to sales or usage of the client's production environment. Virtually any type of data relevant to a client's

production environment could be reported to the data collection unit 200 via one or more APIs installed on the client's systems.

[0023] The production environment assistant 100 also includes a data transformation and storage unit 300. The data transformation and storage unit 300 receives data from a client's production environment, and transforms and enriches the data and loads that data into a data queue. The data

transformation and storage unit 300 could also act to store received or obtained client data into one or more data repositories.

[0024] The production environment assistant 100 also includes a metrics unit 400. The metrics unit 400 receives or acquires data relating to a client's production environment, and then calculates various metrics using that raw data. Such calculations can include (but are not limited to) different statistical equations and algorithms, as well as outlier and anomaly algorithms. The metrics data is then stored in a metrics repository.

[0025] The production environment assistant 100 further includes an evaluation unit 500. The evaluation unit obtains or acquires data relating to a client's production environment and analyzes the data to determine if a predefined incident has occurred or is occurring on the client's production

environment. The evaluation unit 500 could apply traditional analysis techniques, as well as artificial intelligence based analysis techniques.

[0026] The production environment assistant 100 also includes an incident unit 600. The incident unit 600 is notified by the evaluation unit whenever a predefined incident is determined to have occurred. Such incidents are stored in an incident database, which can be searched via a query unit.

[0027] The production environment assistant 100 further includes a notification unit 700, which reports incidents to client and system administrators. The notification unit 700 can act through various different communication channels to deliver a notification to a client or system administrator. [0028] The production environment assistant 100 further includes an active inspector system 800. The active inspector system 800 configures and runs individual active inspectors, each of which is setup to monitor a single client's production environment for the occurrence of a particular issue or problem. An active inspector may also be configured to take remedial action in an attempt to correct an identified problem or issue.

[0029] The production environment assistant 100 further includes a remediation unit 900. The remediation unit 900 is configured to take steps to correct or mitigate a problem or issue with the client's production environment when such problems or issues have been identified. The production environment assistant 100 also includes a user interface system 1000. The user interface system 1000 provides a variety of different ways that a client can interact with the production environment assistant 100 to obtain data or to cause various actions to occur. The user interface system could utilize speech recognition techniques in order to interact with a client using natural speech or pre-defined speech- based commands. The user interface system 1000 could also interact with various client users in more traditional ways, including graphical user interfaces presented over a computer system.

[0030] Each of the above discussed elements of the production

environment assistant 100 are discussed in more detail below. In addition, Figures 1 1 -17 illustrate the steps of various methods that would be performed by the elements of the production environment assistant 100 to monitor a client's production environment, determine when issues or problems have arisen, report on those problems or issues, as well as take remedial action.

[0031] Figure 2 illustrates various elements of a data collection and transformation unit 200 which can be part of a production environment assistant 100. The data collection unit 200 includes a passive collection unit 202, which receives data reported from the various systems of a client's production environment. The data reported to the passive collection unit 202 may be reported via various APIs that are installed in the client's production environment. Alternatively, or in addition, a dedicated agent could be installed on client servers or networking equipment. Such an agent could utilize the one or more separate API collection methods. The APIs are configured to periodically or continuously report various items of information regarding operations on the client's production environment.

[0032] The passive collection unit 202 can include an API configuration unit 204, which can be used to help configure the various APIs that are installed on a client's production environment. In particular, the API configuration unit 204 can be used to provide one or more client-specific encryption codes, tokens or keys to the APIs installed within a client's production environment. The APIs then include this encryption code, token or key with the data they report to the passive collection unit 204.

[0033] The passive collection unit 202 also includes a data receiving unit 206, which actually receives the data reported from the APIs installed on a client's production environment. The data receiving unit 206 checks the received data to ensure that it includes an appropriate client-specific encryption key, token or code. If so, the data receiving unit 206 accepts the received data. If the received data does not include an appropriate encryption code, token or key, then the data receiving unit ignores the received data. This make it very difficult for a malicious third party to spoof artificial and/or incorrect data. The client- specific encryption code, token or key may also act to identify received data as originating from a particular client.

[0034] The data collection unit 200 can also include an active collection unit 208. The active collection unit 208 actively seeks out and obtains particular items of information from a client's production environment by sending requests for such data to the APIs installed within a client's production environment. The active collection unit 208 can include an API configuration unit 210 which is used to help configure the APIs installed within a client's production environment so that they will respond to such requests. This can include providing the APIs within a client's production environment with various encryption keys or codes which must be used by the active collection unit 208 in order to obtain

information about a client's production environment from those APIs. In other words, the active collection unit 208 may need to provide an encryption key or code to the APIs within a client's production environment in order to obtain data from those APIs. The API configuration unit 210 helps to establish the encryption key or codes which will be used by the active collection unit 208 to obtain information from the APIs within a client's production environment.

[0035] The active collection unit 208 can also include an active collection rules unit 212. The active collection rules unit allows a system administrator or a client to set up pre-defined rules which will determine when and how the active collection unit 208 seeks out information from a client's production environment. Once such rules have been established, the active collection unit 208 acts to follow the rules.

[0036] The active collection unit 208 can further include a client

communication monitoring unit 214. The client communication monitoring unit 214 can include a communication collection unit 216 which monitors

communications which are generated by or received by various individuals employed by or associated with a particular client. This can include collecting copies of email messages, text messages, instant messages, other forms of written communications, as well as copies of audio communications passing between certain individuals. A communication analysis unit 218 then analyzes the client communications collected by the communication collection unit 216 to help determine whether certain activity is occurring within a client's system or production environment.

[0037] The goal of collecting and analyzing client communications is to determine if a problem or issue has arisen within a client's production

environment. To that end, the communications analysis unit 218 can search client communications for certain key words that are associated with a particular issue or problem. If one or more key words that relate to a specific type of problem or issue is found in the client communications, the communications analysis unit 218 is able to send that information to the evaluation unit 500 for deep correlation with other signals received by the system. It may send a notification about the potential issue or problem to a system administrator, or possibly to other elements of the production environment assistant so that a more detailed check could be performed, or so that remedial action can be taken.

[0038] The communications analysis unit 218 could compare key words in client communications to information technology words that have known applicability in certain contexts. The goal of the analysis is to determine a client's intent and acts with respect to specific types of issues or problems. A dictionary of information technology or computer words could be consulted for this purpose. Moreover, the communications analysis unit 218 may build up such a dictionary or database of key words over time, where certain key words become associated with certain types of problems. Such a dictionary or database could be specific to a particular client, or it could have broader applicability to multiple clients. This type of historical knowledge can be highly valuable in identifying when a problem has reoccurred.

[0039] The communications analysis unit 218 may use Natural Language Processing (NLP) algorithms to first build a corpus of IT systems intents and IT systems assets. For example, an intent is an action that can be taken

automatically or manually on a system. "Restart", "Increase", "Reboot",

"Shutdown", "Delete", "Add", "Scale", "Tune" are all examples for intents or actions that can be taken on an IT system. "CPU", "Memory", "Subnet", "Network Interface", "Garbage Collection", "I/O", "Disk" are all IT terms. Numbers and percentages, as well as nouns, are the bounding pieces creating the overall sentence semantics. For example, when a human is reporting via a computer messaging system: "Due to High CPU usage, I needed to restart server name: abc123" the communications analysis unit 218 analyzing the sentence would identify the key words such as "Due", "High", "CPU", "Restart", "abc123".

Identifying those key words and sending them to the evaluation unit 500, helps building causality and remediation connections between generic IT components which can be adapted for a specific environment or which can be used

transitively in a broader IT systems environments.

[0040] As mentioned above, the types of data that can be collected by the data collection unit 200 can include various data points about individual computer systems or networks which exist within a client's production environment. The data points can also relate to the operations of individual software applications which are running within a client's production environment. Moreover, the data acquired by the data collection unit 200 can include information about how the business is running, such as financial information, sales data, traffic within an online retailing system, traffic within a communication system, as well as virtually any other type of data relating to the operations of a client's production

environment.

[0041] Many clients will have already installed various monitoring systems or monitoring software applications to monitor the operations of the client's production environment. The data collection unit 200 can obtain information reported by those separate monitoring systems, often through APIs provided with those monitoring systems or monitoring software applications. Examples of such monitoring systems or monitoring software applications include Graphite, New Relic, Appdynamics, Datadog, Ruxit (by Dynatrace), Takipi, Rollbar, Sensu, Nagios, Zabbix, ELK Stack, as well as virtually any other production environment monitoring tool.

[0042] The data collection and transformation unit 300 of the production environment assistant 100 includes a data queue 302. Data and information obtained by the data collection unit 200 is first loaded into the data queue

302. The data queue 302 could include a data points queue 304 and an events queue 306. The data queue 302 is configured to hold a substantial amount of data which has been received from various clients' production environments. For example, the data queue 302 could be configured to hold up to one week's worth of data reported from a plurality of different client production environments. By placing the data immediately into the data queue 302, one can ensure that received data is never lost.

[0043] A storage optimization unit 314 then analyzes the data in the data queue 302 and stores all or various portions of the received data into a short- term repository 308, a medium-term repository 310, and a long-term repository 312. The storage optimization unit 314 can act to store the data in a highly efficient manner to minimize data storage costs. In addition, the storage optimization unit 314 may be responsible for breaking received data into component parts, and storing the received data in pre-defined formats which make it easier to analyze that data a later point in time.

[0044] The storage optimization unit 314, implements a configuration template that supports extending the different storage types and periods. For example, the template may include categories which first utilize extremely short time repository by memory only storage. This might be implemented as a tmpfs file system on each node, or by any other in-memory type technology such as caching layer (Redis, Memcache , RabbitMQ, ActiveMQ or any other related technology). The template might also include the short term, medium term and long term storage layers accordingly. The configuration template also might include each storage layer priority, fallback policy determination (in case of a write or read failure) and object type to be stored.

[0045] By checking first with the configuration template, the storage optimization unit 314 computes in real-time for each storage object, what is the optimal storage layer to use, and then implements a tiered-storage mechanism based on the policy. Once an object needs to be retrieved, since the object type and time is already known, it's possible to skip the search action and point directly to the relevant tier. This provides a great advantage with storage cost as well as performance.

[0046] The storage optimization algorithm can also split the actual data between different tiers and split it into separate files. For example, if a data stream contains 1 month of data points, the optimization storage unit 314 reads the policy template and based on time, priorities, cost or any other attribute, that the 1 -month of data points can be split into smaller sections, and also be split across the different storage types. On read request, each specific piece is retrieved and aggregated in memory before being sent back as the full result.

[0047] A metrics unit 400, which is part of the production environment assistant 100, is responsible for calculating various metrics based upon the data which has been received or obtained from a client's production environment. The metrics unit includes a metrics configuration unit 404 which allows a system administrator and/or a client to determine what type of metrics are to be calculated from the client data. A metrics calculation unit 406 then actually performs the metric calculations based on the configurations established by the metrics configuration unit 404.

[0048] Examples of metrics that can be calculated from data points received from a client's production environment include an average value, a mean, a variance, a covariance, as well as virtually any other type of metric. Such metrics can be calculated using multiple outlier detection algorithms, such as DBSCAN, Hampel Filter, HoltWinters. These metric values could be calculated for a certain period of time, or based on some other type of grouping. The metrics calculation unit 406 can utilize data pulled directly from the data queue 302 of the data collection and transformation unit 300, or data pulled from the short-term repository 308, medium-term repository 310 and long-term repository 312, or data from combinations of those sources. Calculated metrics are stored in a metrics repository 407.

[0049] The metrics unit 400 includes a metrics query interface 408 which allows system administrators, users, and other elements of the production environment assistant 100 to perform queries and obtain information from the calculated metrics information in the metrics repository 407. The metrics query interface makes it possible to obtain calculated metrics for a single client's production environment, or metrics which have been calculated for multiple different client production environments. As a result, one can compare the metrics from one production environment to the metrics in a different production environment to help identify trends, issues and problems.

[0050] The metrics calculation unit 406 may also calculate metrics of metrics. In other words, an average value of a production environment variable which has been calculated for multiple different similar production environments could be calculated by the metrics calculation unit 406 to create a global average for that variable. This global average value would then be stored in the metrics repository 407. The global average value could then be used as a baseline against which a particular client's average value is judged. The particular client's average metric value for that variable would be compared to the calculated global average value for that variable to see how the particular client's production environment compares to the global average.

[0051] The ability to compare an individual production environment metric to a global average is something that many individual companies are unable to perform. Typically, a company will only have access to their own metrics. Thus, the ability to compare metrics from one client's production environment to average values for the same metrics can be a powerful tool in helping to identify issues and problems within individual production environments. In addition, because the metric unit 400 can store not only raw data points, but also events, an aggregation of multiple attributes and combinations of events and data points are possible. This powerful combination, allows the administrator to query for calculated data points and examine correlated events at the same time. That mechanism could also be used automatically to identify potential correlations between events, system/server and time.

[0052] Event correlations are the methods and means for detecting the occurrence of exceptional events in a complex system and for identifying which particular event occurred and where it occurred. The set of events which occur can be detected in the system over a period of time as event streams.

[0053] The evaluation unit 500 of the environment assistant 100 utilizes received client data as well as calculated metrics to perform various analyses that are designed to determine if issues or problems are occurring within a client's production environment, as well as how they are related to each other. Often, events are related based on the timeline and dependencies, as event correlation can take place in both the "space" and time dimensions.

[0054] The evaluation unit 500 includes an evaluation rules unit 502 which is used to set up individual rules which are custom tailored to each individual client. The evaluation rules unit 502 includes a rules set up unit 504 that allows system administrators and clients to set up various rules which determine what types of evaluations are to be performed for a client's production environment. The rules could also establish how frequently and/or under what circumstances a particular type of evaluation should be performed. The rules could also establish various other aspects of how a particular analysis is to be performed.

[0055] The evaluation rules unit 502 also includes a customer interface 506 which makes it possible for an individual customer to access the evaluation rules unit to monitor the types of evaluations which are occurring, and to also alter the evaluation rules which have been set up for the client. The evaluation rules unit 502 also includes a rules database 508 where the evaluation rules are actually stored.

[0056] An analysis unit 512 of the evaluation unit 500 conducts various analyses using the rules stored in the rules database 508. The analysis unit 512 can perform traditional analyses, as well as artificial intelligence-based analyses. For example, the analysis unit 512 could utilize a DROOLS based engine for analyzing data based on a rule base which contains expert knowledge in the form of "if-then" or "condition-action" rules. The condition part of each rule determines whether the rule can be applied based on the current state of the working memory. The action part of a rule contains a conclusion which can be drawn from the rule when the condition is satisfied. The working memory is constantly scanned for facts which can be used to satisfy the condition part of each rule. When a condition is found, the rule is executed. Executing a rule means that the working memory is updated based on the conclusion contained in the rule.

[0057] Alternatively, the analysis unit 512 could utilize various types of rules based artificial intelligence engines such as the CLIPS system, which is an open source system developed by NASA. Various other types of artificial intelligence techniques and evaluation engines could also be used by the analysis unit 512 to analyze client data and metrics, and to apply correlation and noise reduction in order to determine if a problem or issue is occurring within a client's production environment. The analysis unit 512 could also determine the root-cause of an issue based on reasoning. [0058] The Al approach used by the analysis unit 512 utilizes knowledge obtained through the various events from the different IT monitoring

solutions/sensors/agents, as well as from the end-user feedback. Reasoning is accomplished by applying rules to detect the semantics of the event, as well as generic models which rely on generic algorithms, rather than expert knowledge, to correlate events based on an abstraction of the system architecture and its components.

[0059] As an example, if events A and B are detected, and it is known that event A could have been caused by problems n1 , n2, or n3, and event B could have been caused by problems n2, n4, or n6, then the diagnosis is that problem n2 has occurred, because it represents the intersection of the possible sources of events A and B. Planning is accomplished by analyzing the entire system state and conditions before applying an action or recommendation. Learning is accomplished by applying multiple machine learning algorithms in the family of supervised and unsupervised learning.

[0060] Another learning approach which could be taken is the Version Space algorithm. Given a hypothesis space H, and training data D, the version space is the complete subset of H that is consistent with D. The version space can be naively generated for any finite H by enumerating all hypotheses and eliminating the inconsistent ones. In another learning case, one would first scan a database to find frequent items, e.g. {a, b, c, d ...}. For each pair of such items, try to create a rule with only two items, e.g. {a}=>{b}. Then, find larger rules by recursively scanning the database for adding a single item at a time to the left or right part of each rule (left and right expansions), e.g. {a,c} =>{b} , then {a,c,d} →{b}, etc.

[0061] Each rule created is tested to see if it is valid. This provides an automated and constant learning approach to rules generation and adaptation. It also provides the ability to transfer rules and reasoning between different customers. Since IT production environments can be identified with exact or similar technologies, there are specific technology signatures that might be used. For example, customer A could set rules related to its environment that is deployed inside container technology such as Docker. Since the container technology itself is well recognized, it has a set of sensors and parameters that are always relevant in any deployment. Once the base signature is detected with Customer B, the system might inject the same generic rules and recommend the user to make the relevant adaptation to his own needs.

[0062] Last, natural language processing (communication), perception and the ability to act is also implemented as part of the remediation engine. Some of the Preventive monitoring approaches include statistical analysis (mostly

Bayesian networks), neural networks and fuzzy logic.

[0063] The evaluation unit 500 can also include a data acquisition unit, which is used by the analysis unit 512 to obtain the data needed to perform a particular type of analysis. The data acquisition unit 510 can obtain data from the metrics repository 407, and also from any of the data sources provided by the data collection and transformation unit 300. In some instances, the data acquisition unit 510 may engage the services of the active collection unit 208 to obtain certain data needed to perform an analysis.

[0064] If the analysis unit 512 ultimately concludes that a problem or issue is occurring or may be occurring within a client's production environment, the analysis unit indicates that an "incident" has occurred. The term "incident" is a broad term which is intended to apply to any type of activity, trend, occurrence or event which could be viewed as an issue or problem for a client's production environment. Incidents can be raised once a specific condition has been confirmed by the evaluation unit 500. A condition can be an Anomaly detected, a specific metric calculation or data point that is above or below a threshold, an event (such as a new code deployment, a new scaling activity detected or a configuration change detected), a complicated computation such as rate of change, or even a combination between all of the above. Incidents can be analyzed as well and take into account for the next evaluation cycle.

[0065] When incidents are determined to have occurred, the incidents are reported to the incident unit 600. The incident unit 600 includes an instant database 602 where such incidents are recorded. The incident unit 600 also includes an incident query unit 604 which can be used to query information in the incident database 602. Queries could be performed for a single client's production environment. Alternatively, the incident query unit 604 could allow a user to perform a query for the same or similar incidents that have occurred across multiple different client production environments.

[0066] For example, if a new specific type of incident has occurred for the first time for a first customer's production environment, one could then query the incident database 602 to determine if the same or a similar incident has occurred in other client production environments. If so, one could then look to those other client production environments to determine what sort of remedial action cured or mitigated the incident. Thus, the ability to query for incidents across all client production environments provides a valuable tool which can help to quickly determine how to solve or mitigate issues.

[0067] This ability to monitor and learn from multiple client production environments dramatically increases the knowledge base compared to a system that is dedicated to only one production environment. Also, the ability to review data generated from multiple client production environments helps with reasoning and causation inference. The ability to index in a shared fast data store that includes a knowledge base of incidents across clients, environments, events and data points allows for similarities algorithms based on time, semantics, key-terms and dependencies between systems.

[0068] For example, if the same event name occurred after a specific sequence, the system assigns that sequence, and for each step a number, as a representation. Applying sequence matching, similarities algorithms such as Hamming Distance, BM25, DFR, DFI, IB similarities, LM Dirichlet, LM Jelinek Mercer similarity as well as a priory algorithms can determine best potential match and score each relevancy. Here again, if a client only had his one past incidents to rely upon, this ability would not exist.

[0069] The notification unit 700 is responsible for notifying a client when problems or issues have occurred. The notification unit includes a notification rules setup unit 702 which is utilized by system administrators and clients to determine when and/or how incidents are to be reported to a client. The rules established by the notification rules setup unit 702 are then stored in the notification rules database 704. A notification analysis unit 706 utilizes the rules in the notification rules database to determine whether or when incidents identified by the evaluation unit 500 should be reported to a client. As is explained in greater detail below, the notification analysis unit 706 could determine that it is necessary to perform a secondary analysis or investigation once an incident is determined to have occurred before the incident is actually report to the client.

[0070] The notification unit 700 includes a notification transmittal unit 708 which is responsible for reporting incidents and other information to a client. The notification transmittal unit 708 can utilize various different communication channels to send such notifications to a client. For example, the notifications could be sent via email, text messaging, instant messaging, via telephone calls, via pagers, or via virtually any other communication channel which can connect to a client. Likewise, the notification transmittal unit 708 could be configured to send notifications both to a client and to a system administrator of the production environment assistant 100. Typically, the rules in the notification rules database 704 will indicate who should receive such a notification, and how the notification is to be transmitted.

[0071] The production environment assistant 100 also includes an active inspector system 800. The active inspector system 800 includes an active inspector configuration unit 802 which would be used to configure individual active inspectors for a particular client. In other words, a particular client could have multiple active inspectors, all which are simultaneously operational. Each of the individual active inspectors would be configured to look for or analyze for a particular type of problem or issue.

[0072] The active inspector system 800 includes a data acquisition and analysis unit 804. The data acquisition and analysis unit 804 could obtain information from the data queue 302 of the data collection and transformation unit 300, from the short-term repository 308, the medium-term repository 310 and/or the long-term repository unit 312. The data acquisition and analysis unit 804 can also seek information which has been calculated by the metrics unit 400 and stored in the metrics repository 407. Moreover, the data acquisition and analysis unit 804 could utilize the services of the active collection unit 208 of the data collection unit 200 to actively obtain the various items of information directly from a client's production environment through APIs that have been configured on that client's production environment.

[0073] If necessary, the data acquisition and analysis unit 804 could utilize the services of the metrics unit 400 to calculate metrics from obtained data. The data acquisition and analysis unit 804 could also utilize the services of the evaluation unit 500 to evaluate acquired information and metrics. Ultimately, the data acquisition and analysis unit 804 determines whether or not the issue, event, problem or incident that it has been configured to monitor for has occurred. If so, a reporting unit 806 of the active inspector system 800 would then report about the occurrence of that issue, problem, event or incident. The reporting unit 806 could utilize the services of the notification unit 700 to accomplish the reporting.

[0074] The production environment assistant 100 also includes a remediation unit 900. The remediation unit 900 is configured to take active steps in an attempt to correct or mitigate any problems or issues which may have occurred within a client's production environment. The remediation unit 900 includes a notification analysis interface 902. The notification analysis interface 902 receives notifications about incidents which have occurred, those

notifications having been sent via the notification unit 700. A keyword analysis unit 904 then analyzes the notification to determine whether certain keywords exist within the notification. A problem identification unit 906 utilizes output from the keyword analysis unit 904 to determine if the reported incident is indicative of a pre-defined type of problem.

[0075] If the notification analysis interface 902 ultimately determines that a pre-defined type of problem or issue has occurred, the remediation

recommendation unit 908 reviews various items of information to determine if there is an established protocol for correcting, mitigating or otherwise dealing with the identified issue or problem. The remediation recommendation unit 908 can look in a remediation action database 910 for pre-defined ways of helping to alleviate a problem or issue. The remediation recommendation unit 908 can also include a user portal 912 which allows various users to contribute to the remediation action database 910.

[0076] In one particular implementation, the remediation action database 910 can utilize Ansible Playbooks. A remote execution model over secure shell (SSH) is used to execute the procedure on each host, or by executing a set of API instructions on the infrastructure, such as Amazon Web Services Public Cloud provider, Google Cloud, Microsoft Azure Cloud or any other public or private cloud service (such as Cloud Foundry, OpenStack and others) as long as they support Application Protocol Interface (API). By providing a single

repository and exposing it based on remediation key words, systems and actions, anyone can search for a specific use case and find a relevant playbook or remediation script. A contributor can share from his own experience by writing a remediation script according to a pre-defined template, and uploading it to the shared repository. It is then possible for the system to index each key word and action term from the pre-defined template, and make it available for execution by anyone. Sharing the system and remediation knowledge increases remediation reliability and decreases execution errors.

[0077] In some instances, the remediation recommendation unit 908 may find that there are multiple remediation actions in the remediation action database 910 that could be used to address an identified issue or problem.

When that occurs, the query unit 914 could be used to obtain input from a system administrator or a client about which of the remediation actions to take in an attempt to mitigate or solve the identified issue or problem. In addition to allowing a system administrator or client to select one remediation action, the system administrator or client might also identify multiple remediation actions that are to be taken in a particular order until the identified problem is cured or mitigated. [0078] Once a remediation action or group of remediation actions is identified, a remediation action unit 916 then interacts with a client's production environment to carry out the remediation action(s) in an attempt to mitigate or solve the problem or issue.

[0079] A user interface system is illustrated in Figure 10. The user interface system 1000 is customizable and can adapt to various different user environments. A user customization unit 1002 determines how best to interact with a customer and his computing devices, and stores that user customization information in a user profile database 1004. The user customization information can include information about the specific devices and display screens which a user typically uses to interact with the production environment assistant 100. The user customization information can also include information about whether the user interacts via text, voice and/or video. Further, the user customization information can include information that allows the user interface system 1000 to adapt to specific user characteristics or traits, such as knowledge about a user's accent that must be taken into account when processing the user's voice commands. The information stored in the user profile database 1002 allows the user interface system 1000 to format information so that it can be effectively displayed on specific user computing devices, such as specific display screens, specific smartphones, tablets, and other mobile devices.

[0080] The user interface system 1000 also is capable of performing various different forms of user interaction. If the user choose to interact via text, a text interface 1006 performs the user interaction. The text interface could utilize one or more ChatBot components or services to communicate with a user. A ChatBot is basically a computer program designed to simulate conversation with human users, especially over the Internet. A ChatBot is typically powered by rules and artificial intelligence so that the user perceives that he is interacting with another human. The text interface 1006 could include one or more of its own ChatBot components or services, or the text interface 1006 could utilize ChatBot components or services provided by other service providers. For example, the text interface could utilize a ChatBot that is provided by Facebook Messenger, Slack, HipChat, Telegram, and other online providers.

[0081] In a typical text-based interaction, a user would ask a question or issue a command via text, and the text interface 1006 would interpret the text and cause appropriate action to occur. For example, a user could issue a text based question, and the text interface 1006 would interpret the question, cause an answer to be obtained, and then provide the answer to the user via a text- based response. The text interface 1006 may utilize Natural Language

Processing algorithms to interpret a user's text question or command.

[0082] In addition to the text interaction, the user interface system 1000 supports other means of user interaction, such as via audio and video. A voice interface 1008 could receive user input in the form of voice questions or commands. The voice interface 1008 then interprets the user's spoken audio input and causes appropriate actions to occur. For example, the user could issue a spoken audio question, and the voice interface would then interpret the question, obtain an answer to the question, and provide that answer to the user. The answer could be provided as an audio answer, as a text based answer, as a graphical response provided on a user display screen, or as combinations of those response formats.

[0083] A user's spoken audio input could be captured by any sort of user interface that includes a microphone. Such devices could include a computer, a smartphone, or a dedicated voice interface such as the Amazon Echo and the associated Alexa Skills SDK. Alternatively, the user could interact with the voice interface 1008 of the user interface system 1000 via the Apple SiRi interface, and the associated SiRi SDK.

[0084] When a user is making use of a separate voice interface, such as the Amazon Echo and Alexa voice service, the user interaction provided to the user interface system 1000 of the production environment assistant 100 could actually be provided in the form of text which is interpreted by the text interface 1006. For example, a user's voice command could be captured by the Echo device, and the Echo device or an associated Alexa skill could convert the spoken input into text. The text is then provided to the text interface 1006, which interprets the user's spoken input and takes appropriate action. The text interface 1006 could then provide a text-based response which is provided to the Echo device, and the Echo device convert the text response to audio voice which is played to the user by the Echo device. In this instance, the voice-to-text conversion and the text-to-voice conversion is not performed by the user interface system 1000, but rather by a separate entity.

[0085] If a user has a video camera, the user might also interact with the user interface system 1000 using video input. A video interface 1010 would receive the video from the user and interpret the video input. This could include interpreting different body movements and gestures depicted in the user- provided video. For example, a user is asked a yes or no question, the user could gesture with a Thumbs Up or Thumbs Down to provide a response to the question. The video interface could interpret the user's response and provide the answer to the portion of the production environment assistant 100 that posed the question.

[0086] If a user has a video camera, the video interface 1010 might also user-provided video to help accomplish user authentication. In this case, instead of having a user input a traditional user name and password, the user could simply look directly at the video camera, and the user's image is captured and used for user authentication purposes. Once the user has been identified, the user's profile could be accessed to determine the user's preferences for the subsequent user interactions.

[0087] The video interface 1010 could also be used to cause a "character" or "persona" to be displayed on a user display screen. The character or persona might have an abstract human-like face, body or other depiction, and the character or persona would represent the production environment assistant 100 in user interactions. A system character or persona that interacts with a user could be customized to have a particular name or appearance. The user may then use the character or persona's name when asking a question or issuing a command. For example, a user could issue a request for information by saying "Sam, please identify all servers with over 50% CPU usage in my production system and report back after you have restarted them one after another." Such a command contains the user's intentions (Identify, Report, Restart), nouns, metrics and specifics (production system).

[0088] An interactive feedback system may be implemented through the user interface system 1000. For each event presented either by voice, video or via the traditional graphical user interface, the user has the ability to provide feedback. This feedback is critical part of the system, as it forms one of the learning inputs to the systems. The system is capable of handling several feedback types. For example, a user could indicate that an event or incident is a false-positive. A user could also indicate that a recommendation is useful or not. The user may also provide input regarding what steps the user took in order to fix a particular problem. It may also be possible for a user to upload files to the system for indexing and future reference. Such user feedback is then used to improve the performance of the production environment assistant 100.

[0089] Figure 1 1 illustrates steps of a method which is performed to obtain data from a client's production environment and to store that data into one or more data queues. The method 1 100 begins and proceeds to step S1 102 where data reported by APIs installed on a client's production environment is received by the passive collection unit 202 of the data collection unit 200. The received data can include data points and events. Those data points and events can relate to individual elements of computer equipment, networking equipment, and also software applications which are running on the client's production

environment. As noted above, the received data could also include business- related data such as financial data or traffic data.

[0090] The method 1 100 also includes an optional step S1 104, where an active collection unit 208 of the data collection unit 200 actively obtains certain data from a client's production environment via APIs installed on the client's production environment. In step S1 106 the received data point information is loaded tin a data point queue. The method also includes step S1 108, received event information is loaded into an event queue. The method then ends. [0091] Figure 12 illustrates steps of a method that would be performed by the data collection and transformation unit 300 to store data. The method 1200 begins and proceeds to step S1202 where a storage optimization unit 314 of the data collection and transformation unit 300 obtains client data which has been stored in a data point queue 304 or an events queue 306. In step S1004 the storage optimization unit 314 manipulates the received data in various fashions to prepare the data for storage. This can include de-serializing received data, and reformatting the received data into pre-defined formats which make later analysis of the data easier to perform. The method then proceeds to step S1206 where the storage optimization unit 314 stores some items of data into a short- term repository 308. In step S1208 the storage optimization unit 314 stores certain items of data in medium term repository 310. In step S1210, the storage optimization unit 314 stores certain items of data into a long term repository. The method then ends.

[0092] Figure 13 illustrates steps of a method which would be performed by a metrics unit 400 of the production environment assistant 100. The method 1300 begins and proceeds to step S1302 where data relating to a client's production environment is obtained from a data point queue 304 and/or from an events queue 306 and/or from a data storage repository, such as the short-term storage repository 308, the medium term storage repository 310 and the long- term storage repository 312. In step S1304 the data is validated to ensure that it has been received from a particular client's APIs. This can include examining the data for the existence of a client-specific encryption key, token or code which has been provided along with the data.

[0093] The method then proceeds to step S1306 where the data is parsed. In step S1308 the data is arranged into predetermined data

formats. The parsing and arrangement steps S1306, 1308 are optional data steps that may or may not be performed depending upon the particular type of data which is being used and the metrics which are to be calculated. [0094] In step S1310, a metrics calculation unit 406 then calculates various metrics using the obtained data. In step S1312, the calculated metrics are then stored in a metrics repository 407. The method then ends.

[0095] Figure 14 illustrates steps of a method which would be performed by the evaluation unit 500 to determine if a particular incident has occurred. The method 1400 begins and proceeds to step S1402 where a data acquisition unit 510 of the evaluation unit 500 obtains data relating to a particular client's production environment. In step S1404 the obtained data is analyzed by the analysis unit 512 of the evaluation unit 500. In step S1406, the analysis unit 512 determines whether a pre-defined incident has occurred based on the analysis performed in step S1404. If a pre-defined incident is determined to have occurred, in step S1408 the incident is reported to an incident unit 600 and/or to a notification unit 700. The method then ends.

[0096] Figure 15 illustrates various steps of a method which would be performed by a notification unit 700 of the production environment assistant 100. The method 1500 begins and proceeds to step S1502 where the

notification unit 700 receives a report indicating that a pre-defined incident has occurred for a particular client's production environment. The method then proceeds to step S1504 were a notification analysis unit 706 checks a notification rules database 704 to determine if a rule for handling such an incident exists within the notification rules database 704. If no rule for the incident exists, the method proceeds to step S1506 where the incident is reported to a client and/or a system administrator according to a default reporting procedure.

[0097] If a rule for handling the incident exists, the notification transmittal unit reports the incident according to that rule. In some instances, the rule will simply indicate that the occurrence of the incident is to be reported to a client or system administrator through one or more communications channels. If that is the case, the notification transmittal unit 708 carries out the notification according to the rule.

[0098] In other instances, the rule for reporting an incident will indicate that some additional investigation or analysis is to be performed before the incident is reported to a client or system administrator. In that instance, the method proceeds to step S1508, where a secondary analysis is performed by a notification analysis unit 706 of the notification unit 700. The secondary analysis could include obtaining additional information or waiting for a predetermined period of time to determine if the incident persists. The method then proceeds to step S1510 where the incident is only reported if the secondary analysis performed in step S1508 indicates that the incident should be reported. The method then ends.

[0099] Figure 16 illustrates steps of a method which would performed by an active inspector which has been configured by the active inspector system 800. As mentioned above, an active inspector would actively check for data or events within a client's production environment to monitor for the occurrence of a particular problem or issue.

[00100] The method 1600 begins and proceeds to step S1602 where a data acquisition and analysis unit 804 of the active inspector actively collects data from a client's production environment using APIs that are installed within the client's production environment. The method then proceeds to step S1604 were various metrics are calculated utilizing the obtained data. Step S1604 could be performed utilizing the services of the metrics unit 400.

[00101] The method then proceeds to step S1606 where the obtained data and/or the calculated metrics are analyzed to determine if a pre-defined incident has occurred. This analysis could be performed with the services of the evaluation unit 500, as described above. The method then proceeds to step S1608, where the occurrence of the incident is reported, if it is determined to have occurred. Here again, the reporting on the incident could be performed with the services of the notification unit 700, as described above.

[00102] Figure 17 illustrates steps of a method that would be performed by the remediation unit 900 to attempt to correct or mitigate a problem or issue which has occurred within a client's production environment. The method 1700 begins and proceeds to step S1702 were a notification relating to a client's system is received by the remediation unit 900. The method then proceeds to step S1704 were a notification analysis interface 902 of the remediation unit 900 analyzes the received notification to determine if it relates to an issue or problem which could be corrected or mitigated by one or more types of remedial action. This analysis can also be performed with the services of the remediation recommendation unit 908 of the remediation unit 900.

[00103] The method then proceeds to step S1706 were a check is performed to determine if there are multiple different types of remedial actions which could be performed in order to correct or mitigate the identified problem. If multiple types of remedial action have been identified, the method proceeds to step S1708 were input is obtained about what type of remedial action(s) should be performed. This could include a query unit 914 of the remediation

recommendation unit 908 sending a query to a system administrator or client. The input received or obtained in step S1708 is then used to determine what type of remedial action(s) is to be performed, and in step S1701 that remedial action(s) is taken by the remediation action unit 916.

[00104] If the check performed a step S1706 indicates that no remedial action was identified, or that only a single type of remedial action is identified, the method proceeds to set S1712. In step S1712 a check is performed to

determine if only a single type of remedial action was identified. If so, the method proceeds to step S1714, where the remediation action unit 916 takes the remedial action. If the check performed in step S1712 indicates that no remedial actions were identified, the method simply proceeds to the end.

[00105] Although the methods and systems have been described relative to specific embodiments thereof, they are not so limited. As such, many

modifications and variations may become apparent in light of the above teachings. Many additional changes in the details, materials, and arrangement of parts, herein described and illustrated, can be made by those skilled in the art. Accordingly, it will be understood that the methods, devices, and systems provided herein are not to be limited to the embodiments disclosed herein, can include practices otherwise than specifically described, and are to be interpreted as broadly as allowed under the law. [00106] Implementations of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this

specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[00107] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[00108] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross- platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[00109] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language resource), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules,

sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[00110] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[00111] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[00112] To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending resources to and receiving resources from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[00113] Implementations of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this

specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[00114] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client- server relationship to each other. In some implementations, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

[00115] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[00116] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular implementations of particular inventions. Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[00117] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous.

Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.