Entity-relationship modeling with provenance linking for enhancing visual navigation of datasets

Abstract

A method of data analysis is enabled by receiving raw data records extracted from one or more data sources, and then generating from the data records an entity-relationship model. The entity-relationship model comprises more entity instances, and one or more relationships between those entity instances. Data analysis of the model is facilitated using one or more provenance links. A provenance link associates raw data records and one or more entity instances. Using a visual explorer that displays a set of entity instances and relationships from a selected entity-relationship model, a user can display details for an entity instance, and see relationships between and among entity instances. By virtue of the underlying linkage provided by the provenance links, the user can also display source records for an entity instance, and display entity instances for a source record. The technique facilitates Big Data analytics.

What is claimed is as follows: 
     
         1 . A method of data analysis, comprising
 receiving raw data source records extracted from one or more data sources;   generating from the received raw data source records at least one entity-relationship model, the entity-relationship model comprising one or more entity instances, and one or more relationships between those entity instances;   generating entity-to-source record index entries that relate entity instances with the source records that contribute to those entity instances;   combining the entity-to-source record index entries into an entity-to-source record index and co-locating the entity-to-source record index in a first data store that stores the source records;   generating source record-to-entity index entries that relate source records to the entity instances to which the source records contribute;   combining the source record-to-entity index entries into a source record-to-entity index and co-locating the source record-to-entity index in second data store that stores the entity instances and relationships, the second data store being distinct from the first data store;   identifying and displaying a set of one or more source records of interest during visual exploration of the entity-relationship model using the entity-to-source record index; and   identifying and displaying a set of one or more entities and relationships during visual exploration of source records using the source records-to-entity index.   
     
     
         2 - 6 . (canceled) 
     
     
         7 . The method of  claim 1  wherein identifying and receiving a set of one or more source records of interest includes receiving a query against the entity-relationship model and, in response, updating a visual display. 
     
     
         8 . The method of  claim 7  wherein updating the visual display traverses the entity-relationship model to include a relationship that connects a new entity instance with an entity instance already visible in the entity-relationship model prior to receipt of the query. 
     
     
         9 . The method of  claim 1  wherein the entity-to-source record index entries, and the source record-to-entity index entries, are generated as data is extracted from the raw data records. 
     
     
         10 . The method of  claim 1  wherein the entity-to-source record index entries, and the source record-to-entity index entries, are internal data structures that are not exposed to a user during the visual exploration of the entity-relationship model. 
     
     
         11 . The method of  claim 1  wherein the raw data source records comprise enterprise network security-related data sources, and the entities are cybersecurity actors and assets. 
     
     
         12 . The method of  claim 1  wherein the raw data source records comprise healthcare billing-related data sources, and the entities are healthcare actors and assets. 
     
     
         13 . The method of  claim 1  wherein the raw data source records comprise financial training-related data sources, and the entities are financial-related actors and assets. 
     
     
         14 . The method of  claim 1  wherein the raw data source records comprise intelligence analysis data sources, and the entities are actors and assets. 
     
     
         15 . The method of  claim 1  wherein the raw data source records comprise counter-party risk analysis data sources, and the entities are risk-related actors and assets. 
     
     
         16 . Apparatus for data analysis of an entity-relationship model comprising one or more entity instances, and one or more relationships between those entity instances, the entity-relationship model generated from raw data source records, comprising:
 one or more computing machines;   a network-accessible display interface executing on one of the computing machines;   a source records data store to receive and store raw data source records extracted from one or more data sources;   the source records data store further storing, as an entity-to-source record index co-located with the source records, entity-to-source record index entries that relate entity instances with the source records that contribute to those entity instances;   an entity-relationship data store to receive and store the entity-relationship data model generated from the raw data source records, the entity-relationship data store being distinct from the source records data store;   the entity-relationship data store further storing, as a source record-to-entity index co-located with the entity instances to which the source records contribute, source record-to-entity index entries that relate source records to the entity instances to which the source records contribute; and   the network-accessible display interface using the entity-to-source record index to display a set of one or more source records of interest during a visual exploration of the entity-relationship model, and using the source record-to-entity index to display a set of one or more entities and relationships during a visual exploration of source records.   
     
     
         17 . The apparatus as described in  claim 16  further including an analytics application to generate the entity-relationship model, wherein the network-accessible display interface and the analytics application operate via a software-as-a-service model. 
     
     
         18 . The apparatus as described in  claim 17  wherein the analytics application receives a query and, in response, generates the entity-relationship model. 
     
     
         19 . The apparatus as described in  claim 18  wherein the entity-relationship model is updated based on the visual exploration. 
     
     
         20 . (canceled) 
     
     
         21 . An apparatus, comprising:
 one or more hardware processors;   computer memory to store computer program instructions executed by the hardware processors, the computer program instructions comprising:
 a network-accessible analytics application providing a visual explorer; 
 a data organizing application comprising program code (i) to ingest and store raw data records extracted from one or more data sources, (ii) to generate from the received raw data records at least one entity-relationship model, the entity-relationship model comprising one or more entity instances, and one or more relationships between those entity instances, (iii) to generate and store an entity-to-source record index, and a source record-to-entity index, the entity-to-source record index including entries that relate entity instances with the source records that contribute to those entity instances, the source record-to-entity index including entries that relate source records to the entity instances to which the source records contribute, the entity-to-source record index co-located and stored in a first data store with the raw data records, the entity-to-source record index co-located and stored in a second data store with the entity-relationship model, the first data store being distinct from the second data store, and (iv) during a visual exploration of the entity-relationship model using the visual explorer, and using the entity-to-source record index and the source record-to-entity index, to display one of: source records for an entity instance, and entity instances for a source record. 
   
     
     
         22 . The apparatus of  claim 21  wherein the raw data source records comprise enterprise network security-related data sources, and the entities are cybersecurity actors and assets. 
     
     
         23 . The apparatus of  claim 21  wherein the raw data source records comprise healthcare billing-related data sources, and the entities are healthcare actors and assets. 
     
     
         24 . The apparatus of  claim 21  wherein the raw data source records comprise financial training-related data sources, and the entities are financial-related actors and assets. 
     
     
         25 . The apparatus of  claim 21  wherein the raw data source records comprise intelligence analysis data sources, and the entities are actors and assets. 
     
     
         26 . The apparatus of  claim 21  wherein the raw data source records comprise counter-party risk analysis data sources, and the entities are risk-related actors and assets. 
     
     
         27 . The method of  claim 1  further including:
 tracking a state of at least one entity instance over time, wherein the state comprises a collection of elements, the elements being one of: scalar features, aggregate features, and relationships to other entity instances; and 
 using the entity-to-source record index to find at least one source record directly contributing to a state of the at least one entity instance at a given time.