Machine learning engine for determining data similarity

Abstract

A system and method for training and using a machine-learning similarity framework are provided. During training, the similarity framework generates an ensemble of tress. The trees have different properties at each node. The similarity framework uses the ensemble of trees to determine similarity between objects. The objects are propagated through nodes of each tree in the ensemble of trees until the objects reach leaf nodes. The objects are propagated by comparing the properties at each node of the tree to the features of the objects until the objects reach the leaf nodes. The similarity framework determines a similarity score for a pair of objects in each tree and adjusts the similarity score by tree importance. The object similarity score is determined by combining the similarity scores from multiple trees in the ensemble of trees. The similarity framework generates a similarity matrix that stores object similarity scores for multiple pairs of objects.

What is claimed is: 
     
         1 . A method for determining similarity, the method comprising:
 generating, using a machine learning similarity framework, an ensemble of trees using features corresponding to first securities and at least one objective; and   determining a similarity matrix, the similarity matrix storing security similarity scores indicating similarities between pairs of securities in second securities, wherein determining the similarity matrix comprises:
 for each tree in the ensemble of trees:
 propagating the second securities through the each tree until the second securities reach leaf nodes of the each tree, wherein the propagating compares at least one feature associated with each second security in the second securities to at least one property of at least one node associated with the each tree; and 
 determining a similarity score for each pair of securities by calculating a distance between the each pair of securities in the each tree; 
 
 for the each pair of securities, combining similarity scores from the each tree in the ensemble of trees into a security similarity score; and 
 storing the security similarity scores from the pairs of securities in the similarity matrix. 
   
     
     
         2 . The method of  claim 1 , further comprising:
 generating a strategy, the strategy including at least one low liquid security, wherein the low liquid security has liquidity below a low liquidity threshold;   determining, by comparing security similarity scores between the at least one low liquidity security and the second securities in the similarity matrix, at least one liquid security, wherein the security similarity scores between the at least one liquid security and the at least one low liquid security are above a similarity threshold; and   substituting the at least one low liquidity security with the at least one liquid security in the strategy.   
     
     
         3 . The method of  claim 1 , further comprising:
 generating, using the similarity matrix, a cluster of similar securities, wherein the similar securities in the cluster have security similarity scores above a similarity threshold.   
     
     
         4 . The method of  claim 3 , further comprising:
 determining liquid securities in the cluster; and   determining at least one cluster price for the cluster based on the liquid securities.   
     
     
         5 . The method of  claim 1 , further comprising:
 generating a portfolio of securities, the portfolio including at least one low liquid security, wherein the low liquid security has liquidity below a liquidity threshold;   identifying, using the similarity matrix, a second security in the second securities having a liquidity greater than the low liquid security, wherein the second security and the low liquid security have a security similarity score above a similarity threshold; and   substituting the low liquid security in the portfolio with the second security.   
     
     
         6 . The method of  claim 1 , wherein determining the similarity score for the each pair of securities further comprises:
 assigning the similarity score of one when the each pair of securities are associated with a same leaf node in the leaf nodes of the each tree; or   assigning the similarity score of zero when the each pair of securities is associated with different leaf nodes in the leaf nodes of the each tree.   
     
     
         7 . The method of  claim 1 , wherein determining the similarity score for the each pair of securities further comprises:
 determining a deepest common node between the each pair of securities in the each tree; and   determining a depth of the each tree, wherein determining the similarity score is further based on the deepest common node and the depth of the each tree.   
     
     
         8 . The method of  claim 1 , wherein determining the similarity score for the each pair of securities further comprises:
 determining a tree importance weight for the each tree in the ensemble of trees; and   adjusting the similarity score for the each pair of securities by the tree importance weight.   
     
     
         9 . The method of  claim 1 , further comprising:
 generating a tree to be included in the ensemble of trees;   adding the tree in the ensemble of trees; and   determining a tree importance weight of the tree in the ensemble of trees by:
 calculating an error of the ensemble of trees before and after adding the tree to the ensemble of trees, wherein determining the tree importance weight is further based on the error. 
   
     
     
         10 . The method of  claim 1 , wherein the machine learning similarity framework uses a base function, a loss function, and at least one hyperparameter to generate the ensemble of trees. 
     
     
         11 . The method of  claim 10 , further comprising:
 determining, using the machine learning similarity framework, a steepest gradient descent of the loss function; and   estimating, based at least in part on the steepest gradient descent, the at least one property of at least one node in the each tree.   
     
     
         12 . A system comprising:
 a memory configured to store instructions for a machine learning similarity framework;   a processor coupled to the memory and configured to read the instructions from the memory to cause the system to perform operations, the operations comprising:
 generating, using the machine learning similarity framework, an ensemble of trees using features corresponding to first securities, an objective, a base function, and a loss function; and 
 determining a similarity matrix, the similarity matrix storing security similarity scores indicating similarities between pairs of securities in second securities, wherein determining the similarity matrix comprises:
 for each tree in the ensemble of trees:
 propagating the second securities through the each tree until the second securities reach leaf nodes of the each tree, wherein the propagating compares at least one feature associated with each second security in the second securities to at least one property of at least one node associated with the each tree; and 
 determining a similarity score for each pair of securities by calculating a distance between at least one leaf node storing the each pair of securities in the each tree; 
 
 for the each pair of securities, combining the similarity score from the each tree in the ensemble of trees into a security similarity score; and 
 storing the security similarity scores for the pairs of securities in the similarity matrix. 
 
   
     
     
         13 . The system of  claim 12 , wherein the operations further comprise:
 generating a strategy, the strategy including at least one low liquidity security;   determining, by comparing the at least one low liquid security to liquid securities in the similarity matrix, at least one liquid security having a security similarity score with the at least one low liquid security above a predefined threshold; and   substituting the at least one low liquid security with the at least one second liquid security in the strategy.   
     
     
         14 . The system of  claim 12 , wherein the operations further comprise:
 generating, using the similarity matrix, a cluster of similar securities, securities in the cluster having security similarity scores above a similarity threshold;   determining liquid securities in the cluster, wherein the liquid securities have a liquidity above a liquidity threshold; and   determining at least one cluster price for the cluster based on the liquid securities.   
     
     
         15 . The system of  claim 12 , wherein the operations further comprise:
 generating a portfolio of securities, the portfolio including at least one low liquid security;   identifying, using the similarity matrix, a second security that has a liquidity greater than the low liquid security, wherein the second security and the low liquid security have a security similarity score above a threshold; and   substituting the low liquid security in the portfolio with the second security.   
     
     
         16 . The system of  claim 12 , wherein the operations further comprise:
 determining a deepest common node in the each tree between the each pair of securities; and   determining a depth of the each tree, wherein determining the similarity score is further based on the deepest common node and the depth of the tree.   
     
     
         17 . The system of  claim 12 , wherein the operations further comprise:
 determining a tree importance weight for the each tree in the ensemble of trees; and   adjusting the similarity score for the each pair of securities by the tree importance weight.   
     
     
         18 . The system of  claim 12 , wherein the operations further comprise:
 generating a tree to be included in the ensemble of trees;   adding the tree in the ensemble of trees; and   determining a tree importance weight of the tree in the ensemble of trees by:
 calculating an error of the ensemble of trees before and after adding the tree to the ensemble of trees; wherein determining the tree importance weight is further based on the error. 
   
     
     
         19 . A non-transitory computer-readable medium having instructions thereon, that when executed by a processor, cause the processor to perform operations for determining similarity, the operations comprising:
 generating, using a machine learning similarity framework, an ensemble of trees using features corresponding to first securities and a hyperparameter; and   determining a similarity matrix, the similarity matrix storing security similarity scores indicating similarities between pairs of securities in second securities, wherein determining the similarity matrix comprises:
 for each tree in the ensemble of trees:
 propagating the second securities through the each tree until the second securities reach leaf nodes of the each tree, wherein the propagating compares at least one feature associated with each second security in the second securities to at least one property of at least one node associated with the each tree; and 
 determining a similarity score for each pair of securities by calculating a distance between the each pair of securities in the each tree; 
 
 for the each pair the securities, combining the similarity score from the each tree in the ensemble of trees into a security similarity score; and 
 storing the security similarity scores for the pairs of securities in the similarity matrix. 
   
     
     
         20 . The non-transitory computer-readable medium of  claim 19 , wherein rows and columns of the similarity matrix identify the second securities, and entries in the similarity matrix identify the security similarity scores associated with the pairs of securities.