System for automated capture and analysis of business information for security and client-facing infrastructure reliability

Abstract

A system for fully integrated collection of business impacting data, analysis of that data and generation of both analysis driven business decisions and analysis driven simulations of alternate candidate business actions has been devised and reduced to practice. This business operating system may be used to monitor and predictively warn of events that impact the security of business infrastructure and may also be employed to monitor client-facing services supported by both software and hardware to alert in case of reduction or failure and also predict deficiency, service reduction or failure based on current event data.

What is claimed is: 
     
         1 . A system for fully integrated collection of business impacting data, analysis of that data and generation of both analysis-driven business decisions and analysis-driven simulations of alternate candidate business decision comprising:
 a business data retrieval engine stored in a memory of and operating on a processor of a computing device;   a business data analysis engine stored in a memory of and operating on a processor of a computing device; and   a business decision and business action path simulation engine stored in a memory of and operating on a processor of one of more computing devices;   wherein, the business information retrieval engine:
 (a) retrieves a plurality of business related data from a plurality of sources; 
 (b) accept a plurality of analysis parameters and control commands directly from human interface devices or from one or more command and control storage devices; 
 (b) stores accumulated retrieved information for processing by data analysis engine or predetermined data timeout; 
   wherein the business information analysis engine:
 (c) retrieves a plurality of data types from the business information retrieval engine; 
 (d) performs a plurality of analytical functions and transformations on retrieved data based upon the specific goals and needs set forth in a current campaign by business process analysis authors; 
   wherein the business decision and business action path simulation engine:
 e) employs results of data analyses and transformations performed by the business information analysis engine, together with available supplemental data from a plurality of sources as well as any current campaign specific machine learning, commands and parameters from business process analysis authors to formulate current business operations and risk status reports; and 
 (f) employs results of data analyses and transformations performed by the business information analysis engine, together with available supplemental data from a plurality of sources, any current campaign specific commands and parameters from business process analysis authors, as well as input gleaned from machine learned algorithms to deliver business action pathway simulations and business decision support to a first end user. 
   
     
     
         2 . The system of  claim 1 , wherein the business information retrieval engine a stored in the memory of and operating on a processor of a computing device, employs a portal for human interface device input at least a portion of which are business related data and at least another portion of which are commands and parameters related to the conduct of a current business analysis campaign. 
     
     
         3 . The system of  claim 2 , wherein the business information retrieval engine employs a high volume deep web scraper stored in the memory of an operating on a processor of a computing device, which receives at least some scrape control and spider configuration parameters from the highly customizable cloud based interface, coordinates one or more world wide web searches (scrapes) using both general search control parameters and individual web search agent (spider) specific configuration data, receives scrape progress feedback information which may lead to issuance of further web search control parameters, controls and monitors the spiders on distributed scrape servers, receives the raw scrape campaign data from scrape servers, aggregates at least portions of scrape campaign data from each web site or web page traversed as per the parameters of the scrape campaign. 
     
     
         4 . The system of  claim 3 , wherein the archetype spiders are provided by a program library and individual spiders are created using configuration files. 
     
     
         5 . The system of  claim 3 , wherein scrape campaign requests are persistently stored and can be reused or used as the basis for similar scrape campaigns. 
     
     
         6 . The system of  claim 2 , wherein the business information retrieval engine employs a multidimensional time series data store stored in a memory of and operating on a processor of a computing device to receive a plurality of data from a plurality of sensors of heterogeneous types, some of which may have heterogeneous reporting and data payload transmission profiles, aggregates the sensor data over a predetermined amount of time, a predetermined quantity of data or a predetermined number of events, retrieves a specific quantity of aggregated sensor data per each access connection predetermined to allow reliable receipt and inclusion of the data, transparently retrieves quantities of aggregated sensor data too large to be reliably transferred by one access connection using a further plurality access connections to allow capture of all aggregated sensor data under conditions of heavy sensor data influx and stores aggregated sensor data in a simple key-value pair with very little or no data transformation from how the aggregated sensor data is received. 
     
     
         7 . The system of  claim 1 , wherein the business data analysis engine employs a directed computational graph stored in the memory of an operating on a processor of a computing device which, retrieves streams of input from one or more of a plurality of data sources, filters data to remove data records from the stream for a plurality of reasons drawn from, but not limited to a set comprising absence of all information, damage to data in the record, and presence of in-congruent information or missing information which invalidates the data record, splits filtered data stream into two or more identical parts, formats data within one data stream based upon a set of predetermined parameters so as to prepare for meaningful storage in a data store, sends identical data stream further analysis and either linear transformation or branching transformation using resources of the system. 
     
     
         8 . The system of  claim 1 , wherein the business data analysis engine employs a graph stack service module stored in a memory of and operating on a processor of one of more computing devices which organizes data retrieved from the multidimensional time series database module into graph formats where the objects are represented as vertices and the relationships between them as edges of the graph. 
     
     
         9 . A method for fully integrated collection of business impacting data, analysis of that data and generation of both analysis driven business decisions and analysis driven business decision simulations the method comprising the steps of:
 (a) retrieving business related data and analysis campaign command and control information using a business information retrieval engine stored in the memory of an operating on a processor of a computing device;   (b) analyzing and transforming retrieved business related data using a business information analysis engine stored in the memory of an operating on a processor of a computing device in conjunction with previously designed analysis campaign command and control information; and   (c) presenting business decision critical information as well as business pathway simulation information using a business decision and business path simulation engine based upon the results of analysis of previously retrieved business related data and previously entered analysis campaign command and control information.   
     
     
         10 . The method of  claim 9 , wherein the business information retrieval engine employs, a portal for human interface device input at least a portion of which are business related data and at least another portion of which are commands and parameters related to the conduct of a current business analysis campaign. 
     
     
         11 . The method of  claim 9 , wherein the business information retrieval engine employs a high volume deep web scraper stored in the memory of an operating on a processor of a computing device, which receives at least some scrape control and spider configuration parameters from the highly customizable cloud based interface, coordinates one or more world wide web searches (scrapes) using both general search control parameters and individual web search agent (spider) specific configuration data, receives scrape progress feedback information which may lead to issuance of further web search control parameters, controls and monitors the spiders on distributed scrape servers, and receives the raw scrape campaign data from scrape servers, aggregates at least portions of scrape campaign data from each web site or web page traversed as per the parameters of the scrape campaign. 
     
     
         12 . The method of  claim 10 , wherein the archetype spiders are provided by a program library and individual spiders are created using configuration files. 
     
     
         13 . The method of claim,  10  wherein scrape campaign requests are persistently stored and can be reused or used as the basis for similar scrape campaigns. 
     
     
         14 . The method of  claim 9 , wherein the business information retrieval engine employs a multidimensional time series data store stored in a memory of and operating on a processor of a computing device to receive a plurality of data from a plurality of sensors of heterogeneous types, some of which may have heterogeneous reporting and data payload transmission profiles, aggregates the sensor data over a predetermined amount of time, a predetermined quantity of data or a predetermined number of events, retrieves a specific quantity of aggregated sensor data per each access connection predetermined to allow reliable receipt and inclusion of the data, transparently retrieves quantities of aggregated sensor data too large to be reliably transferred by one access connection using a further plurality access connections to allow capture of all aggregated sensor data under conditions of heavy sensor data influx and stores aggregated sensor data in a simple key-value pair with very little or no data transformation from how the aggregated sensor data is received. 
     
     
         15 . The system of  claim 8 , wherein the business data analysis engine employs a directed computational graph, stored in the memory of an operating on a processor of a computing device which, retrieves streams of input from one or more of a plurality of data sources, filters data to remove data records from the stream for a plurality of reasons drawn from, but not limited to a set comprising absence of all information, damage to data in the record, and presence of in-congruent information or missing information which invalidates the data record, splits filtered data stream into two or more identical parts, formats data within one data stream based upon a set of predetermined parameters so as to prepare for meaningful storage in a data store, sends identical data stream further analysis and either linear transformation or branching transformation using resources of the system. 
     
     
         16 . The method of  claim 9 , wherein the business data analysis engine employs a graph stack service module stored in a memory of and operating on a processor of one of more computing devices which organizes data retrieved from the multidimensional time series database module into graph formats where the objects are represented as vertices and the relationships between then as edges of the graph. 
     
     
         17 . A method for the detection of Kerberos based security exploits using a system for fully integrated capture, and analysis of business information the method comprising the steps of:
 (a) retrieving ticket granting ticket request information, service session key request information, user sign on attempt data from a Kerberos domain controller using a multidimensional time series database module stored in a memory of and operating on a processor of a computing device.   (b) applying any pre-programmed multiple dimensional time series event-condition-action rules that are present and apply to Kerberos protocol events using the multidimensional time series database module.   (c) performing conversion of data into into graphs where objects are vertices and their relationships edges between vertices using a graph stack service stored in a memory of and operating on a processor of a computing device.   (d) performing any analytical transformations using a directed computational graph module.   
     
     
         18 . A method to monitor the function of business critical IT infrastructure and business software performance using a system for fully integrated capture, and analysis of business information resulting in improved client-facing IT infrastructure reliability the method comprising the steps of:
 (a) monitoring IT equipment and application status statistics as well as failure messages using a multidimensional time series database module stored in a memory of and operating on a processor of a computing device;   (b) processing the data retrieved from multidimensional time series database module using a graph stack service stored in a memory of and operating on a processor of a computing device with infrastructure items and software forming vertices of a relational graph and relationships between them forming edges of the graph;   (c) transforming data acquired by the multidimensional time series database module using directed computational graph to formulate more complex diagnostic queries based upon the existing data using pre-programmed logic and machine learning and then process the results of those complex queries as predetermined by authors of the monitoring effort; and   (d) presenting the results in format best suited to the downstream use of the processed data.   
     
     
         19 . The method of  claim 18 , wherein at least one set of results are displayed as a graphical simulation using an observation and state estimation service module stored in a memory of and operating on a processor of a computing device.