Hierarchical Service Oriented Application Topology Generation for a Network

Abstract

The technology disclosed relates to understanding traffic patterns in a network with a multitude of processes running on numerous hosts. In particular, it relates to using at least one of rule based classifiers and machine learning based classifiers for clustering processes running on numerous hosts into local services and clustering the local services running on multiple hosts into service clusters, using the service clusters to aggregate communications among the processes running on the hosts and generating a graphic of communication patterns among the service clusters with available drill-down into details of communication links. It also relates to using predetermined command parameters to create service rules and machine learning based classifiers that identify host-specific services. In one implementation, user feedback is used to create new service rules or classifiers and/or modify existing service rules or classifiers so as to improve accuracy of the identification of the host-specific services.

What is claimed is: 
     
         1 . A system for generating hierarchical service oriented application topology of a network with a multitude of processes running on numerous hosts, the system comprising:
 receiving, from a network-connected machine learning-based classifier, service profiles classifying hosts that run similar processes, the machine learning-based classifier being trained to cluster the hosts into service profiles by:   evaluating command parameters of respective processes running on the hosts by applying logistic regression to string vectors of the command parameters to calculate a probability of classifying a host into a particular service profile, and   based on the evaluation, classifying hosts that run similar processes as having a same service profile;   receiving, from a network-connected graphic generator, a graphic of the topography of the network, the graphic generator generating the graphic of the topology of the network based on the service profiles produced by the machine learning-based classifier; and   a user interface output device providing, for display, the graphic generated.   
     
     
         2 . The system of  claim 1 , wherein the service profiles cluster hosts that share common functionality. 
     
     
         3 . The system of  claim 1 , wherein the machine learning-based classifier evaluates string vectors of the command parameters against process-specific rules stored in a rule database. 
     
     
         4 . The system of  claim 1 , wherein the string vectors are chosen based on their respective term frequency-inverse document frequencies (TF-IDFs) prior to being evaluated by the machine learning-based classifier. 
     
     
         5 . The system of  claim 1 , wherein the machine learning-based classifier applies a logistic regression algorithm that uses cross-validation to classify hosts into service profiles. 
     
     
         6 . The system of  claim 1 , wherein the machine learning-based classifier is trained using manually labelled training data. 
     
     
         7 . The system of  claim 1 , wherein the machine learning-based classifier includes a neural network. 
     
     
         8 . The system of  claim 1 , wherein the machine learning-based classifier applies a naïve bayes algorithm to command parameters. 
     
     
         9 . A method of generating hierarchical service oriented application topology of a network with a multitude of processes running on numerous hosts, the method including:
 obtaining service profiles of hosts clustered using a trained machine learning-based classifier that:
 evaluates command parameters of respective processes running on the hosts by applying logistic regression to string vectors of the command parameters to calculate a probability of classifying a host into a particular service profile, and 
 based on the evaluation, classifies hosts that run similar processes as having a same service profile; and 
   obtaining a graphic of the topology of the network generated based on the service profiles produced by the trained machine learning-based classifier; and   providing the graphic of the topology of the network as output to a user interface output device for presentation.   
     
     
         10 . The method of  claim 9 , wherein the service profiles cluster hosts that share common functionality. 
     
     
         11 . The method of  claim 9 , wherein the trained machine learning-based classifier evaluates string vectors of the command parameters against process-specific rules stored in a rule database. 
     
     
         12 . The method of  claim 9 , wherein the string vectors are chosen based on their respective term frequency-inverse document frequencies (TF-IDFs) prior to being evaluated by the trained machine learning-based classifier. 
     
     
         13 . The method of  claim 9 , wherein the trained machine learning-based classifier applies a logistic regression algorithm that uses cross-validation to classify hosts into service profiles. 
     
     
         14 . The method of  claim 9 , wherein the trained machine learning-based classifier is trained using manually labelled training data. 
     
     
         15 . The method of  claim 9 , wherein the trained machine learning-based classifier includes a neural network. 
     
     
         16 . The method of  claim 9 , wherein the trained machine learning-based classifier applies a naïve bayes algorithm to command parameters. 
     
     
         17 . One or more non-transitory computer readable media having instructions stored thereon for performing a method of generating hierarchical service oriented application topology of a network with a multitude of processes running on numerous hosts, the method including:
 obtaining service profiles of hosts clustered using a trained machine learning-based classifier to cluster the hosts into service profiles by:
 evaluating command parameters of respective processes running on the hosts by applying logistic regression to string vectors of the command parameters to calculate a probability of classifying a host into a particular service profile, and 
 based on the evaluation, classifying hosts that run similar processes as having a same service profile; and 
   obtaining a graphic of the topology of the network generated based on the service profiles produced by the trained machine learning-based classifier.   
     
     
         18 . The one or more non-transitory computer readable media of  claim 17 , wherein the trained machine learning-based classifier includes a neural network. 
     
     
         19 . The one or more non-transitory computer readable media of  claim 17 , wherein the trained machine learning-based classifier evaluates string vectors of the command parameters against process-specific rules stored in a rule database. 
     
     
         20 . The one or more non-transitory computer readable media of  claim 17 , wherein the trained machine learning-based classifier applies a logistic regression algorithm that uses cross-validation to classify hosts into service profiles.