Dark web monitoring, analysis and alert system and method

Abstract

A dark web monitoring, analysis and alert system comprising a data receiving module configured to receive data collected from the dark web and structured; a Structured Data Database (SDD) connected with the data receiving module, the SDD configured to store the structured data; a Text Search and Analytic Engine (TSAE) connected with the SDD, the TSAE configured to enable advanced search and basic analysis in the structured data; a Knowledge Deduction Service (KDS) connected with the TSAE, the KDS configured to deeply analyze the collected data; the deep analysis comprises extracting insights regarding dark web surfers behavioral patterns and interactions; a Structured Knowledge Database (SKD) connected with the KDS, the SKD configured to store the deep analysis results; and an Alert Service connected with the TSAE and the SKD, the Alert Service configured to provide prioritized alerts based on the deep analysis.

What is claimed is: 
     
         1 . A method of providing searchable database and prioritized search user interface for exploring dark web content and surfer activity, comprising:
 obtaining data comprising at least one web page scanned and collected from the dark web using a repository;   extracting from the data and storing in a structural database at least one structural parameter of a content of the at least one web page;   analyzing data in the structural database to determine and store in a search engine at least one statistic of the content of the at least one web page and to determine by employing machine learning and store in a knowledge database at least one profile characterization of at least one of a behavioral pattern of a surfer engaged with the content of the at least one web page and an interaction of the surfer with at least one other surfer;   for each of one or more search results of a search query to the structural database by the search engine, calculating a score according to a set of defined criteria using data stored in the search engine and the knowledge database;   determining prioritization of the one or more search results using the score calculated for each; and   providing over a communication network to at least one computing device an output of the one or more search results prioritized according to the prioritization.   
     
     
         2 . The method of  claim 1 , wherein the set of defined criteria comprising at least one of: source scoring; recency; user reputation; record type scoring; search result relevance scoring; and content analysis scoring. 
     
     
         3 . The method of  claim 1 , wherein the at least one statistic comprising at least one of: date in which most of comments were written, number of posts a surfer wrote for a specific search query, distribution of categories in a site, time line trending for a specific search query and top sites for a specific query. 
     
     
         4 . The method of  claim 1 , wherein determination of the at least one profile characterization comprising at least one of: analyzing sentiment of comments on posts to calculate reputation evaluation; classifying posts into categories and summing posts in each category to determine fields of interest; monitoring a number of interactions between surfers and identifying groups having a number of interactions above a predetermined threshold; and analyzing activity times. 
     
     
         5 . The method of  claim 4 , wherein analyzing activity times comprising: calculating a temporal data distribution within a time frame; storing time frame which includes most data; and storing an average and a standard deviation of the temporal data distribution. 
     
     
         6 . The method of  claim 1 , wherein the analyzing further comprising identifying usage of different aliases by the at least one surfer using identity matching. 
     
     
         7 . The method of  claim 6 , wherein the identity matching comprising at least one of: 
 locating communication information used by more than one surfer;   looking for similar aliases excluding common names;   locating surfers with similar activity pattern using activity times analysis;   locating surfers with similar fields of interest;   locating surfers who are active for a certain period and continue being active in other places or by other aliases;   locating surfers who post a same content at a same time in different locations;   counting most frequent words used by a surfer; and   analyzing surfers’ text.   
     
     
         8 . The method of  claim 1 , further comprising sending prioritized alerts according to rules defined and stored in an alert rules database responsive to using the search query within monitoring scheduled with relation to at least one alert. 
     
     
         9 . The method of  claim 8 , wherein the alert rules database comprising at least one rule selected from the group consisting of: define wake up intervals for scheduling of monitoring with relation to the at least one alert, enable search by a key word, enable search by an activity related to a certain surfer, enable search by an activity of a certain group, enable search by a change in trend of a certain key word and enable search by a new phrase or a word that appears more than a predetermined number of times. 
     
     
         10 . The method of  claim 1 , further comprising providing case management interface configured to enable a user to create a case file in order to manage a research or an investigation. 
     
     
         11 . The method of  claim 10 , further comprising providing recommendation on adding relevant surfers and/or posts to the case file. 
     
     
         12 . The method of  claim 11 , wherein the recommendation being provided according to at least one of: building a connection map of existing surfers in the case file, analyzing connections in the connection map and recommending adding surfers that have a strong connection with existing surfers in the case file; identity matching based similar surfers; surfers that published posts collected in the case file; surfers that are mentioned in existing posts’ content; surfers having similar fields of interest; and posts that have a strong contextual matching comprising at least one of same classification, same time in a time range of posts in the case file and posts having a words-matching up to a certain threshold. 
     
     
         13 . The method of  claim 1 , further comprising providing analytical dashboard to enable view of data analysis comprising at least one of categories, number of posts by dates, search results, an option to create an alert from a search, a total number of search results, surfer details, surfer activity analysis, surfer number of posts by dates, surfer categories and surfer connection map. 
     
     
         14 . The method of  claim 1 , wherein obtaining the data from the dark web comprising using at least one crawler configured to manage an Internet Protocol address thereof by at least one of hiding the Internet Protocol address and changing the Internet Protocol address, progress from one web page to another using extracted links found in each web page, classify web pages extracted and control operation timing and pace of data collection. 
     
     
         15 . The method of  claim 14 , wherein the at least one crawler being further configured to optimize scanning pace versus secrecy thereof. 
     
     
         16 . The method of  claim 1 , wherein the at least one profile characterization comprising reputation evaluation performed using formulas (I) and (II), wherein: 
       
         
           
             
               R 
               
                 p 
               
               = 
               
                 
                   
                       
                     
                       R 
                       0 
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                       when 
                       ​ 
                         
                       Q 
                       p 
                         
                         
                       = 
                         
                         
                       
                         ∅ 
                         ′ 
                       
                         
                     
                     
                       
                         
                           
                             
                               S 
                               
                                 q 
                               
                                 
                               | 
                                 
                                 
                               q 
                                 
                                 
                               i 
                               n 
                                 
                                 
                               Q 
                               p 
                             
                           
                         
                       
                         
                         
                       ∗ 
                         
                         
                       
                         
                             
                           | 
                             
                           Q 
                           p 
                             
                           | 
                         
                         4 
                       
                         
                         
                         
                         
                       when 
                         
                       Q 
                       p 
                         
                         
                       ≠ 
                         
                         
                       ∅ 
                     
                   
                 
               
             
           
         
       
       
         
           
             
               R 
               
                 u 
               
               = 
               
                 
                   
                       
                     
                       0 
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                         
                       when 
                         
                       P 
                       u 
                         
                         
                       = 
                         
                         
                       
                         ∅ 
                         ′ 
                       
                     
                     
                       
                         
                           
                             
                               R 
                               
                                 p 
                               
                                 
                               | 
                                 
                                 
                               p 
                                 
                                 
                               i 
                               n 
                                 
                                 
                               P 
                               u 
                             
                           
                         
                       
                         
                         
                       ∗ 
                         
                         
                       
                         
                             
                           | 
                             
                           P 
                           u 
                             
                           | 
                         
                         4 
                       
                         
                         
                         
                       when 
                         
                       P 
                       u 
                         
                         
                       ≠ 
                         
                         
                       ∅ 
                     
                   
                 
               
             
           
         
       
       and wherein:
 u = user; 
 p = post; 
 Pu = user post list; 
 Qp = post comments list, not including a user comments on its own post; 
 R(p) = post reputation; 
 R(u) = user reputation; 
 S(q) = sentiment of comment q, where -1≤S(q) ≤1; 
 R0 = reputation of a post with no comments; 
 (G) = average value of G; 
 |G| = number of members in G. 
 
     
     
         17 . A system for searchable database and prioritized search user interface for exploring dark web content and surfer activity, comprising:
 a repository configured to be used in obtaining data comprising at least one web page scanned and collected from the dark web;   a structured data extractor for extracting from the data and storing in a structural database at least one structural parameter of a content of the at least one web page;   an analytic engine for analyzing data in the structural database to determine and store in a search engine at least one statistic of the content of the at least one web page;   a knowledge deduction service for analyzing data in the structural database using machine learning to determine and store in a knowledge database at least one profile characterization of at least one of a behavioral pattern of a surfer engaged with the content of the at least one web page and an interaction of the surfer with at least one other surfer;   an alert service for: 
 calculating, for each of one or more search results of a search query to the structural database by the search engine, a score according to a set of defined criteria using data stored in the search engine and the knowledge database; 
 determining prioritization of the one or more search results using the score calculated for each; and 
 providing over a communication network to at least one computing device an output of the one or more search results prioritized according to the prioritization. 
   
     
     
         18 . The system of  claim 17 , further comprising at least one crawler and a hidden service locator connected with the dark web and the at least one crawler, the hidden service locator being configured to find at least one hidden uniform resource locator in the dark web; wherein the at least one crawler being configured to scan and collect from the dark web at least a portion of said data using the at least one hidden uniform resource locator. 
     
     
         19 . The system of  claim 18 , further comprising a configuration database for storing respective of the at least one crawler a configuration comprising at least one of: at least one initial uniform resource location for the at least one crawler to start from, at least one username and at least one password of the at least one crawler, and timing setting of the at least one crawler. 
     
     
         20 . The system of  claim 17 , wherein the structured data extractor comprising:
 a wrapper generator;   a wrapper database; and   an extractor;   wherein the wrapper generator being configured to analyze each of the at least one web page, find patterns, create a wrapper and save the wrapper in the wrapper database;   wherein the extractor being configured to receive a respective one of the at least one web page and a wrapper from the wrapper database and to extract data from the respective web page according the wrapper.