Measuring accuracy of semantic graphs with exogenous datasets

Abstract

Provided is a process including: obtaining a semantic similarity graph having nodes corresponding to documents in an analyzed corpus and edges indicating semantic similarity between pairs of the documents; for at least a plurality of nodes in the graph, evaluating accuracy of the edges based on neighboring nodes and an external corpus by performing operations including: identifying the neighboring nodes based on adjacency to the respective node in the graph; selecting documents from an external corpus based on references in the selected documents to entities mentioned in the documents of the neighboring nodes; and determining how semantically similar the respective node is to the selected documents.

What is claimed is: 
     
         1 . A method of evaluating relationships in a relationship graph that indicates relationships between bodies of unstructured text, the method comprising:
 accessing a relationship graph to obtain a given node and a connected node connected to the given node, the document relationship graph comprising:
 nodes each representing respective unstructured text, and 
 edges each connecting respective pairs of the nodes and representing relationships between unstructured text of the connected nodes; 
   determining, from unstructured text of the connected node, an item referenced by the unstructured text of the connected node;   accessing an external dataset to obtain data related to the item, the external dataset including at least some data external to the relationship graph; and   determining a relationship between the data related to the item obtained from the external dataset and unstructured text of the given node.   
     
     
         2 . The method of  claim 1 , wherein:
 the relationship graph is a semantic similarity graph;   each node represents a document having unstructured text;   edges represent semantic similarity of documents;   accessing the relationship graph comprises retrieving nodes adjacent the given node and choosing the connected node from among the nodes adjacent the given node;   determining the item comprises extracting an ngram, topic, or other entity discussed by the unstructured text of the connected node;   accessing the external dataset comprises identifying a document for which the item is a topic and obtaining features of the document; and   determining the relationship comprises determining an amount of similarity between features of unstructured text of the given node and the features of the document.   
     
     
         3 . The method of  claim 1 , comprising:
 determining a quality score of the relationship graph based on the relationship between the data related to the item and unstructured text of the given node.   
     
     
         4 . The method of  claim 1 , comprising:
 forming at least part of the external dataset before accessing the external dataset to obtain data related to the item by determining vector representations of unstructured text in an external corpus that is external to an internal corpus from which the relationship graph is formed.   
     
     
         5 . The method of  claim 4 , comprising:
 removing non-shared features between external corpus and the internal corpus.   
     
     
         6 . The method of  claim 1 , wherein:
 the relationship graph comprises a semantic similarity graph having weighted edges indicating an amount of semantic similarity between unstructured text of nodes connected by the respective weighted edges.   
     
     
         7 . The method of  claim 1 , wherein:
 determining the item comprises determining an entity mentioned in unstructured text of a document represented by the given node.   
     
     
         8 . The method of  claim 7 , wherein:
 accessing the external dataset comprises retrieving a document about the entity.   
     
     
         9 . The method of  claim 1 , comprising:
 steps for expediting matrix operations on a matrix having feature vectors based on the external dataset or a corpus upon which the relationship graph is based.   
     
     
         10 . The method of  claim 1 , wherein:
 accessing the relationship graph comprises:
 steps for feature extraction and compression of a corpus; 
 steps for creating a feature set matrix; 
 steps for measuring relationships between documents; and 
 steps for determining neighboring nodes; 
   determining the item comprises:
 steps for extracting entities from adjacent nodes; 
   accessing the external dataset comprises:
 steps for creating a feature set matrix for the external dataset; and 
   determining the relationship comprises:
 steps for comparing between a weighted-adjacency vector of the given node and selected feature vectors from the external dataset. 
   
     
     
         11 . A system, comprising:
 one or more processors;   memory storing instructions that when executed by at least some of the processors effectuate operations comprising:
 accessing a relationship graph to obtain a given node and a connected node connected to the given node, the document relationship graph comprising:
 nodes each representing respective unstructured text, and 
 edges each connecting respective pairs of the nodes and representing relationships between unstructured text of the connected nodes; 
 
 determining, from unstructured text of the connected node, an item referenced by the unstructured text of the connected node; 
 accessing an external dataset to obtain data related to the item, the external dataset including at least some data external to the relationship graph; and 
 determining a relationship between the data related to the item obtained from the external dataset and unstructured text of the given node. 
   
     
     
         12 . The system of  claim 11 , wherein:
 the relationship graph is a semantic similarity graph;   each node represents a document having unstructured text;   edges represent semantic similarity of documents;   accessing the relationship graph comprises retrieving nodes adjacent the given node and choosing the connected node from among the nodes adjacent the given node;   determining the item comprises extracting an ngram, topic, or other entity discussed by the unstructured text of the connected node;   accessing the external dataset comprises identifying a document for which the item is a topic and obtaining features of the document; and   determining the relationship comprises determining an amount of similarity between features of unstructured text of the given node and the features of the document.   
     
     
         13 . The system of  claim 11 , the operations comprising:
 determining a quality score of the relationship graph based on the relationship between the data related to the item and unstructured text of the given node.   
     
     
         14 . The system of  claim 11 , the operations comprising:
 forming at least part of the external dataset before accessing the external dataset to obtain data related to the item by determining vector representations of unstructured text in an external corpus that is external to an internal corpus from which the relationship graph is formed.   
     
     
         15 . The system of  claim 14 , the operations comprising:
 removing non-shared features between external corpus and the internal corpus.   
     
     
         16 . The system of  claim 11 , wherein:
 the relationship graph comprises a semantic similarity graph having weighted edges indicating an amount of semantic similarity between unstructured text of nodes connected by the respective weighted edges.   
     
     
         17 . The system of  claim 11 , wherein:
 determining the item comprises determining an entity mentioned in unstructured text of a document represented by the given node.   
     
     
         18 . The system of  claim 17 , wherein:
 accessing the external dataset comprises retrieving a document about the entity.   
     
     
         19 . The system of  claim 11 , the operations comprising:
 steps for expediting matrix operations on a matrix having feature vectors based on the external dataset or a corpus upon which the relationship graph is based.   
     
     
         20 . The system of  claim 11 , wherein:
 accessing the relationship graph comprises:
 steps for feature extraction and compression of a corpus; 
 steps for creating a feature set matrix; 
 steps for measuring relationships between documents; and 
 steps for determining neighboring nodes; 
   determining the item comprises:
 steps for extracting entities from adjacent nodes; 
   accessing the external dataset comprises:
 steps for creating a feature set matrix for the external dataset; and 
   determining the relationship comprises:
 steps for comparing between a weighted-adjacency vector of the given node and selected feature vectors from the external dataset.