Generating a view of a dataset in a data lakehouse based on user intent

Abstract

Described are techniques for generating a query based view of appropriate datasets in a data lakehouse. A query for data contained in the data lakehouse is received. A similarity between the query and a description of each table in the lakehouse is computed. Any description of a table with a similarity to the query that is greater than a threshold value results in the table potentially being included in a view. As a result, such a table is selected. A subset of the selected tables is then identified and joined to create a new view of a dataset. After generating a description of the new view based on headers and values of the dataset, a similarity between the query and the description of the new view is computed. If the similarity measure exceeds a threshold value, then the newly constructed view is displayed to the user.

1 . A computer-implemented method for generating a query based view of appropriate datasets in a data lakehouse, the method comprising;
 receiving a query for data contained in the data lakehouse;   computing a similarity between the query and a description of each table in the data lakehouse;   selecting one or more tables in the data lakehouse in response to the similarity between the query and the description of the one or more tables exceeding a first threshold value;   identifying a subset of the selected one or more tables using a relational graph convolutional network for linking keywords in the query to columns in the selected one or more tables of the data lakehouse, wherein the relational graph convolutional network performs multiple rounds of convolution on a graph, wherein, in each round, the relational graph convolutional network uses a graph structure to aggregate information from neighboring nodes and edges which is used to update features of each node in the graph;   joining the subset of the selected one or more tables to create a new view of a dataset;   generating a description of the new view of the dataset;   computing a similarity between the query and the description of the new view of the dataset; and   displaying the new view of the dataset to a user in response to the similarity between the query and the description of the new view of the dataset exceeding a second threshold value thereby providing to the user an appropriate view of the dataset in the data lakehouse for a required purpose of the user.   
     
     
         2 . The method as recited in  claim 1  further comprising:
 creating an embedding from the query; 
 creating an embedding for a description of a table for each table in the data lakehouse; and 
 computing a similarity between the query and the description of the table for each table in the data lakehouse using the embedding from the query and the embedding for the description of the table for each table in the data lakehouse. 
 
     
     
         3 . The method as recited in  claim 1 , wherein the similarity between the query and the descriptions of the tables in the data lakehouse is computed using semantic annotations of the tables in the data lakehouse. 
     
     
         4 . The method as recited in  claim 1 , wherein the relational graph convolutional network comprises concept nodes which are mapped to particular columns of the tables of the data lakehouse, wherein the mapped columns form the subset of the selected one or more tables which includes a reduced number of columns in the tables that will be part of the new view of the dataset. 
     
     
         5 . The method as recited in  claim 1  further comprising:
 selecting a class of tables within the selected one or more tables in the data lakehouse with a similarity greater than a third threshold value. 
 
     
     
         6 . The method as recited in  claim 1  further comprising:
 identifying the subset of the selected one or more tables based on headers containing keywords or related keywords in the query. 
 
     
     
         7 . The method as recited in  claim 1  further comprising:
 generating the description of the new view of the dataset based on headers and values of the dataset. 
 
     
     
         8 . A computer program product for generating a query based view of appropriate datasets in a data lakehouse, the computer program product comprising one or more computer readable storage mediums having program code embodied therewith, the program code comprising programming instructions for:
 receiving a query for data contained in the data lakehouse;   computing a similarity between the query and a description of each table in the data lakehouse;   selecting one or more tables in the data lakehouse in response to the similarity between the query and the description of the one or more tables exceeding a first threshold value;   identifying a subset of the selected one or more tables using a relational graph convolutional network for linking keywords in the query to columns in the selected one or more tables of the data lakehouse, wherein the relational graph convolutional network performs multiple rounds of convolution on a graph, wherein, in each round, the relational graph convolutional network uses a graph structure to aggregate information from neighboring nodes and edges which is used to update features of each node in the graph;   joining the subset of the selected one or more tables to create a new view of a dataset;   generating a description of the new view of the dataset;   computing a similarity between the query and the description of the new view of the dataset; and   displaying the new view of the dataset to a user in response to the similarity between the query and the description of the new view of the dataset exceeding a second threshold value thereby providing to the user an appropriate view of the dataset in the data lakehouse for a required purpose of the user.   
     
     
         9 . The computer program product as recited in  claim 8 , wherein the program code further comprises the programming instructions for:
 creating an embedding from the query;   creating an embedding for a description of a table for each table in the data lakehouse; and   computing a similarity between the query and the description of the table for each table in the data lakehouse using the embedding from the query and the embedding for the description of the table for each table in the data lakehouse.   
     
     
         10 . The computer program product as recited in  claim 8 , wherein the similarity between the query and the descriptions of the tables in the data lakehouse is computed using semantic annotations of the tables in the data lakehouse. 
     
     
         11 . The computer program product as recited in  claim 8 , wherein the relational graph convolutional network comprises concept nodes which are mapped to particular columns of the tables of the data lakehouse, wherein the mapped columns form the subset of the selected one or more tables which includes a reduced number of columns in the tables that will be part of the new view of the dataset. 
     
     
         12 . The computer program product as recited in  claim 8 , wherein the program code further comprises the programming instructions for:
 selecting a class of tables within the selected one or more tables in the data lakehouse with a similarity greater than a third threshold value.   
     
     
         13 . The computer program product as recited in  claim 8 , wherein the program code further comprises the programming instructions for:
 identifying the subset of the selected one or more tables based on headers containing keywords or related keywords in the query.   
     
     
         14 . The computer program product as recited in  claim 8 , wherein the program code further comprises the programming instructions for:
 generating the description of the new view of the dataset based on headers and values of the dataset.   
     
     
         15 . A system, comprising:
 a memory for storing a computer program for generating a query based view of appropriate datasets in a data lakehouse; and   a processor connected to the memory, wherein the processor is configured to execute program instructions of the computer program comprising:
 receiving a query for data contained in the data lakehouse; 
 computing a similarity between the query and a description of each table in the data lakehouse; 
 selecting one or more tables in the data lakehouse in response to the similarity between the query and the description of the one or more tables exceeding a first threshold value; 
 identifying a subset of the selected one or more tables using a relational graph convolutional network for linking keywords in the query to columns in the selected one or more tables of the data lakehouse, wherein the relational graph convolutional network performs multiple rounds of convolution on a graph, wherein, in each round, the relational graph convolutional network uses a graph structure to aggregate information from neighboring nodes and edges which is used to update features of each node in the graph; 
 joining the subset of the selected one or more tables to create a new view of a dataset; 
 generating a description of the new view of the dataset; 
 computing a similarity between the query and the description of the new view of the dataset; and 
 displaying the new view of the dataset to a user in response to the similarity between the query and the description of the new view of the dataset exceeding a second threshold value thereby providing to the user an appropriate view of the dataset in the data lakehouse for a required purpose of the user. 
   
     
     
         16 . The system as recited in  claim 15 , wherein the program instructions of the computer program further comprise:
 creating an embedding from the query;   creating an embedding for a description of a table for each table in the data lakehouse; and   computing a similarity between the query and the description of the table for each table in the data lakehouse using the embedding from the query and the embedding for the description of the table for each table in the data lakehouse.   
     
     
         17 . The system as recited in  claim 15 , wherein the similarity between the query and the descriptions of the tables in the data lakehouse is computed using semantic annotations of the tables in the data lakehouse. 
     
     
         18 . The system as recited in  claim 15 , wherein the relational graph convolutional network comprises concept nodes which are mapped to particular columns of the tables of the data lakehouse, wherein the mapped columns form the subset of the selected one or more tables which includes a reduced number of columns in the tables that will be part of the new view of the dataset. 
     
     
         19 . The system as recited in  claim 15 , wherein the program instructions of the computer program further comprise:
 selecting a class of tables within the selected one or more tables in the data lakehouse with a similarity greater than a third threshold value.   
     
     
         20 . The system as recited in  claim 15 , wherein the program instructions of the computer program further comprise:
 identifying the subset of the selected one or more tables based on headers containing keywords or related keywords in the query.