Detecting relationships across data columns

Abstract

There is a need for more effective and efficient detection of cross-data-column relationships. This need can be addressed by, for example, techniques for detecting cross-data-column data relationships that utilize at least one of feature-based similarity models and deep-learning-based similarity models. The cross-data-column data relationships may be displayed to an end-user using a cross-column relationship detection user interface.

1 . A computer-implemented method comprising:
 generating, by one or more processors using a deep-learning-based similarity model, a plurality of augmented data columns based at least in part on a plurality of input data columns, wherein: (a) the plurality of augmented data columns comprises: (i) a plurality of tagged augmented data columns comprising, for each tagged data column in a selected subset of one or more tagged data columns, a plurality of tagged augmented data columns that are generated by augmenting the tagged data column, and (ii) for an untagged data column, a plurality of untagged augmented data columns that are generated by augmenting the untagged data column, and (b) augmenting an input data column comprises shuffling the input data column;   generating, by the one or more processors and one or more trained image processing models, a vector representation for each augmented data column of the plurality of augmented data columns based at least in part on an image representation for the augmented data column;   generating, by the one or more processors, an updated related subset of the selected subset of the one or more tagged data columns based at least in part on each vector representation for an augmented data column of the plurality of augmented data columns; and   providing, by the one or more processors, for display of the updated related subset using a cross-column relationship detection user interface.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein generating the plurality of augmented data columns comprises:
 performing missing value remediation and missing value insertion on the plurality of input data columns to generate a plurality of updated data columns;   for each updated data column of the plurality of updated data columns, shuffling a row-wise ordering of the updated data column to generate a predefined number of shuffled data columns for the updated data column; and   determining the plurality of augmented data columns based at least in part on each predefined number of shuffled data columns for an updated data column of the plurality of updated data columns.   
     
     
         3 . The computer-implemented method of  claim 2 , wherein each predefined number is determined based at least in part on a trained column augmentation weight value of the deep-learning-based similarity model. 
     
     
         4 . The computer-implemented method of  claim 1 , further comprising
 generating, using the one or more processors, the image representation for each augmented data column of the plurality of augmented data columns based at least in part by:
 identifying one or more character designations of the augmented data column, wherein the one or more character designations comprise an end-of-row character designation for each row value associated with the augmented data column; 
 for each character designation of the one or more character designations of the augmented data column, generating a per-character one-hot-encoding in accordance with a one-hot-encoding scheme; and 
 generating the image representation to depict each per-character one-hot-encoding for a character designation of the one or more character designations. 
   
     
     
         5 . The computer-implemented method of  claim 1 , wherein generating the vector representation for an augmented data column of the plurality of augmented data columns comprises:
 processing the image representation for the augmented data column using a one-dimensional convolutional neural network autoencoder model of the one or more trained image processing models in order to generate a first vector representation for the augmented data column, wherein the one-dimensional convolutional neural network autoencoder model has been trained to minimize a one-dimensional image reconstruction error of a one-dimensional convolutional-neural-network-based encoder-decoder architecture;   processing the image representation for the augmented data column using a two-dimensional convolutional neural network autoencoder model of the one or more trained image processing models in order to generate a second vector representation for the augmented data column, wherein the two-dimensional convolutional neural network autoencoder model has been trained to minimize a two-dimensional image reconstruction error of a two-dimensional convolutional-neural-network-based encoder-decoder architecture; and   combining the first vector representation and the second vector representation to generate the vector representation for the augmented data column.   
     
     
         6 . The computer-implemented method of  claim 1 , wherein generating the updated related subset for the untagged data column comprises:
 for each column pair of a plurality of column pairs comprising a tagged augmented data column of the plurality of tagged augmented data columns and an untagged augmented data column of the plurality of untagged augmented data columns, determining a measure of vector similarity of a tagged vector representation for the tagged augmented data column in the column pair and an untagged vector representation for the untagged augmented data column in the column pair;   identifying a predefined number of the plurality of column pairs having a highest measure of vector similarity; and   determining the updated related subset based at least in part on each tagged data column of the one or more tagged data columns that is associated with at least one of the predefined number of the plurality of column pairs.   
     
     
         7 . The computer-implemented method of  claim 1 , wherein:
 the deep-learning-based similarity model is further configured to generate, for each augmented data column of the plurality of augmented data columns that is in the updated related subset, a relatedness likelihood value; and   the cross-column relationship detection user interface is configured to display each relatedness likelihood value for an augmented data column of the plurality of augmented data columns that is in the updated related subset.   
     
     
         8 . The computer-implemented method of  claim 1 , wherein the selected subset comprises all of the selected subset. 
     
     
         9 . The computer-implemented method of  claim 1 , wherein:
 the selected subset comprises an initial related subset of the one or more tagged data columns,   the initial related subset is determined using a feature-based similarity model, and   the feature-based similarity model is configured to: (i) determine, for each tagged data column of the one or more tagged data columns, one or more feature-based similarity measures, (ii) combine each of the one or more feature-based similarity measures for a tagged data column of the one or more tagged data columns in accordance with one or more similarity measure weight values to determine one or more weighted similarity scores for the tagged data column, and (iii) determine the initial related subset based at least in part on each one or more weighted similarity scores for a tagged data column of the one or more tagged data columns.   
     
     
         10 . The computer-implemented method of  claim 9 , wherein generating the feature-based similarity model and the deep-learning-based similarity model comprises:
 performing one or more first model training iterations using the one or more tagged data columns to generate the feature-based similarity model, wherein each first model training iterations of the one or more first model training iterations is configured to update the one or more similarity measure weight values in order to optimize a first model measure of error between first model outputs generated by the feature-based similarity model and ground-truth column relationship data for the one or more tagged data columns; and   performing one or more second model training iterations using the one or more tagged data columns to generate the deep-learning-based similarity model, wherein each second model training iterations of the one or more second model training iterations is configured to update one or more image processing weight values of the one or more trained image processing models in order to optimize a second model measure of error between second model outputs generated by the deep-learning-based similarity model and the ground-truth column relationship data for the one or more tagged data columns.   
     
     
         11 . A computing apparatus comprising memory and one or more processors communicatively coupled to the memory, the one or more processors configured to:
 generate, using a deep-learning-based similarity model, a plurality of augmented data columns based at least in part on a plurality of input data columns, wherein: (a) the plurality of augmented data columns comprises: (i) a plurality of tagged augmented data columns comprising, for each tagged data column in a selected subset of one or more tagged data columns, a plurality of tagged augmented data columns that are generated by augmenting the tagged data column, and (ii) for an untagged data column, a plurality of untagged augmented data columns that are generated by augmenting the untagged data column, and (b) augmenting an input data column comprises shuffling the input data column;   generate, using one or more trained image processing models, a vector representation for each augmented data column of the plurality of augmented data columns based at least in part on an image representation for the augmented data column;   generate an updated related subset of the selected subset of the one or more tagged data columns based at least in part on each vector representation for an augmented data column of the plurality of augmented data columns; and   provide for display of the updated related subset using a cross-column relationship detection user interface.   
     
     
         12 . The computing apparatus of  claim 11 , wherein generating the plurality of augmented data columns comprises:
 performing missing value remediation and missing value insertion on the plurality of input data columns to generate a plurality of updated data columns;   for each updated data column of the plurality of updated data columns, shuffling a row-wise ordering of the updated data column to generate a predefined number of shuffled data columns for the updated data column; and   determining the plurality of augmented data columns based at least in part on each predefined number of shuffled data columns for an updated data column of the plurality of updated data columns.   
     
     
         13 . The computing apparatus of  claim 11 , wherein the one or more processors are further configured to:
 generate the image representation for each augmented data column of the plurality of augmented data columns based at least in part by:
 identifying one or more character designations of the augmented data column, wherein the one or more character designations comprise an end-of-row character designation for each row value associated with the augmented data column; 
 for each character designation of the one or more character designations of the augmented data column, generating a per-character one-hot-encoding in accordance with a one-hot-encoding scheme; and 
 generating the image representation to depict each per-character one-hot-encoding for a character designation of the one or more character designations. 
   
     
     
         14 . The computing apparatus of  claim 11 , wherein generating the vector representation for an augmented data column of the plurality of augmented data columns comprises:
 processing the image representation for the augmented data column using a one-dimensional convolutional neural network autoencoder model of the one or more trained image processing models in order to generate a first vector representation for the augmented data column, wherein the one-dimensional convolutional neural network autoencoder model has been trained to minimize a one-dimensional image reconstruction error of a one-dimensional convolutional-neural-network-based encoder-decoder architecture;   processing the image representation for the augmented data column using a two-dimensional convolutional neural network autoencoder model of the one or more trained image processing models in order to generate a second vector representation for the augmented data column, wherein the two-dimensional convolutional neural network autoencoder model has been trained to minimize a two-dimensional image reconstruction error of a two-dimensional convolutional-neural-network-based encoder-decoder architecture; and   combining the first vector representation and the second vector representation to generate the vector representation for the augmented data column.   
     
     
         15 . The computing apparatus of  claim 11 , wherein generating the updated related subset for the untagged data column comprises:
 for each column pair of a plurality of column pairs comprising a tagged augmented data column of the plurality of tagged augmented data columns and an untagged augmented data column of the plurality of untagged augmented data columns, determining a measure of vector similarity of a tagged vector representation for the tagged augmented data column in the column pair and an untagged vector representation for the untagged augmented data column in the column pair;   identifying a predefined number of the plurality of column pairs having a highest measure of vector similarity; and   determining the updated related subset based at least in part on each tagged data column of the one or more tagged data columns that is associated with at least one of the predefined number of the plurality of column pairs.   
     
     
         16 . The computing apparatus of  claim 11 , wherein:
 the deep-learning-based similarity model is further configured to generate, for each augmented data column of the plurality of augmented data columns that is in the updated related subset, a relatedness likelihood value; and   the cross-column relationship detection user interface is configured to display each relatedness likelihood value for an augmented data column of the plurality of augmented data columns that is in the updated related subset.   
     
     
         17 . The computing apparatus of  claim 11 , wherein the selected subset comprises all of the one or more tagged data columns. 
     
     
         18 . The computing apparatus of  claim 11 , wherein:
 the selected subset comprises an initial related subset of the one or more tagged data columns,   the initial related subset is determined using a feature-based similarity model, and   the feature-based similarity model is configured to: (i) determine, for each tagged data column of the one or more tagged data columns, one or more feature-based similarity measures, (ii) combine each of the one or more feature-based similarity measures for a tagged data column of the one or more tagged data columns in accordance with one or more similarity measure weight values to determine one or more weighted similarity scores for the tagged data column, and (iii) determine the initial related subset based at least in part on each one or more weighted similarity scores for a tagged data column of the one or more tagged data columns.   
     
     
         19 . The computing apparatus of  claim 18 , wherein generating the feature-based similarity model and the deep-learning-based similarity model comprises:
 performing one or more first model training iterations using the one or more tagged data columns to generate the feature-based similarity model, wherein each first model training iterations of the one or more first model training iterations is configured to update the one or more similarity measure weight values in order to optimize a first model measure of error between first model outputs generated by the feature-based similarity model and ground-truth column relationship data for the one or more tagged data columns; and   performing one or more second model training iterations using the one or more tagged data columns to generate the deep-learning-based similarity model, wherein each second model training iterations of the one or more second model training iterations is configured to update one or more image processing weight values of the one or more trained image processing models in order to optimize a second model measure of error between second model outputs generated by the deep-learning-based similarity model and the ground-truth column relationship data for the one or more tagged data columns.   
     
     
         20 . One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to:
 generate, using a deep-learning-based similarity model, a plurality of augmented data columns based at least in part on a plurality of input data columns, wherein: (a) the plurality of augmented data columns comprises: (i) a plurality of tagged augmented data columns comprising, for each tagged data column in a selected subset of one or more tagged data columns, a plurality of tagged augmented data columns that are generated by augmenting the tagged data column, and (ii) for an untagged data column, a plurality of untagged augmented data columns that are generated by augmenting the untagged data column, and (b) augmenting an input data column comprises shuffling the input data column;   generate, using one or more trained image processing models, a vector representation for each augmented data column of the plurality of augmented data columns based at least in part on an image representation for the augmented data column;   generate an updated related subset of the selected subset of the one or more tagged data columns based at least in part on each vector representation for an augmented data column of the plurality of augmented data columns; and   provide for display of the updated related subset using a cross-column relationship detection user interface.