Techniques for improving the accuracy of automated predications

Abstract

Techniques are provided for forming clusters of individual prediction targets (IPTs). An initial prediction target is a target for which an automated prediction has been generated. IPTs may be, for example, borrowers to which a lending entity has extended loans based on predictions generated by a credit policy. Each cluster includes (a) a “core” of underperforming entities (UEs), and (b) a set of boundary performant entities (PEs). The UEs that belong to the UE core of a cluster are “similarly situated” relative to the values of their features. For example, in the context where the IPTs are borrowers, the UEs at the core of a cluster may correspond to defaulting borrowers that had similar bureau data, lending entity data, and borrower data. The boundary performant entities of the cluster may be borrowers that have not defaulted, but had similar credit qualifications as the UEs of the cluster. Having formed these clusters, the clusters may be used in a variety of ways, including but not limited to improving the accuracy of the credit model, identifying potentially problematic future borrowers, generating visualizations that illustrate the relative importance of clusters of defaulting borrowers, etc.

1 . A method for improving accuracy of predictions of an automated prediction mechanism by automatically determining one or more input parameters whose values best distinguish (a) individual prediction targets for which predictions were accurate from (b) individual prediction targets for which predictions were erroneous, comprising:
 using the automated prediction mechanism to generate initial predictions;   wherein the initial predictions include an initial prediction for each individual prediction target in a population of individual prediction targets;   wherein, for each individual prediction target in the population of individual prediction targets, the automated prediction mechanism generates the initial prediction based on information about the individual prediction target;   wherein the information used by the automated prediction mechanism to generate the initial prediction for each individual prediction target includes values for each of a plurality of input parameters;   for each individual prediction target in the population of individual prediction targets, obtaining current performance information;   based on the current performance information, determining a set of erroneous predictions from the initial predictions;   determining distances between pairs of individual prediction targets based on the information about the individual prediction targets that was used by the automated prediction mechanism to generate the initial predictions;   based on the distances and the set of erroneous predictions, forming one or more clusters of individual prediction targets from the population of individual prediction targets, wherein each cluster includes:
 a core set of erroneous-prediction entities that are individual prediction targets that are:
 associated with erroneous predictions in the initial predictions, and 
 have relatively shorter distances between each other than to erroneous-prediction entities that do not belong to the cluster; and 
 
 a set of boundary accurate-prediction entities that are individual prediction targets that are:
 associated with accurate predictions in the initial predictions; and 
 have distances to the individual prediction targets in the core set of the cluster that place them at boundaries of the core set of the cluster; 
 
   for at least one cluster of the one or more clusters, determining at least one input parameter, of the plurality of input parameters, whose values distinguish the boundary accurate-prediction entities of the cluster from the core set of erroneous-prediction entities of the cluster;   wherein the method is performed by one or more computing devices.   
     
     
         2 . (canceled) 
     
     
         3 . The method of  claim 1  wherein determining at least one input parameter, of the plurality of input parameters, whose values distinguish the boundary accurate-prediction entities of the cluster from the core set of erroneous-prediction entities of the cluster comprises: for at least a particular cluster of the plurality of clusters, determining a DIFF-SET of the particular cluster, wherein the DIFF-SET of the particular cluster is a set of features that best distinguish the core set of erroneous-prediction entities of the particular cluster from the set of boundary accurate-prediction entities of the particular cluster. 
     
     
         4 . The method of  claim 3  wherein:
 the method further comprises determining a critical range for each feature in the DIFF-SET of the particular cluster; 
 wherein, for each feature in the DIFF-SET of the particular cluster, all members of the core set of erroneous-prediction entities of the particular cluster have values for the feature that fall into the critical range for the feature, and all members of the set of boundary accurate-prediction entities of a cluster have values for the feature that fall outside the critical range for the feature. 
 
     
     
         5 . The method of  claim 4  further comprising updating the automated prediction mechanism based on the DIFF-SET of the particular cluster and the critical range of each feature in the DIFF-SET of the particular cluster. 
     
     
         6 . The method of  claim 1  further comprising generating a display that depicts each cluster of the one or more clusters in a chart that visually communicates relative importance of each cluster of the one or more clusters. 
     
     
         7 . The method of  claim 1  wherein:
 the individual prediction targets are borrowers; 
 the core set of erroneous-prediction entities of each cluster represents a set of borrowers that are delinquent on payments for their respective loans; and 
 the set of boundary accurate-prediction entities of each cluster represents a set of borrowers that are not delinquent on payments for their respective loans. 
 
     
     
         8 . The method of  claim 7  wherein determining distances between pairs of individual prediction targets comprises:
 determining distances between pairs of borrowers based on information about the borrowers that was used by the automated prediction mechanism to generate the initial predictions. 
 
     
     
         9 . The method of  claim 8  wherein:
 the information about the borrowers includes at least bureau data and lending entity data; 
 the method further comprises:
 convoluting the bureau data and the lending entity data for a first borrower to produce a first deep-credit feature; 
 convoluting the bureau data and the lending entity data for a second borrower to produce a second deep-credit feature; 
 determining a distance between the first borrower and the second borrower based on the first deep-credit feature and the second deep-credit feature. 
 
 
     
     
         10 - 13 . (canceled) 
     
     
         14 . The method of  claim 8  further comprising:
 obtaining information about a potential borrower; 
 based on the information about the potential borrower, determining distances between the potential borrower and borrowers that belong to a particular cluster of the one or more clusters; and 
 determining how to handle the potential borrower based, at least in part, on whether the potential borrower is closer to:
 the set of borrowers, within the particular cluster, that are delinquent on payments for their respective loans, or 
 the set of borrowers, within the particular cluster, that are not delinquent on payments for their respective loans. 
 
 
     
     
         15 . The method of  claim 8  further comprising tracking one or more borrowers over time to determine whether distance between (a) the one or more borrowers, and (b) the set of borrowers, within a particular cluster, that are delinquent on payments for their respective loans, is increasing or decreasing. 
     
     
         16 . A one or more non-transitory computer media storing instructions for improving accuracy of predictions of an automated prediction mechanism by automatically determining one or more input parameters whose values best distinguish (a) individual prediction targets for which predictions were accurate from (b) individual prediction targets for which predictions were erroneous; wherein the instructions, when executed by one or more computing devices, cause:
 the automated prediction mechanism to generate initial predictions;   wherein the initial predictions include an initial prediction for each individual prediction target in a population of individual prediction targets;   wherein, for each individual prediction target in the population of individual prediction targets, the automated prediction mechanism generates the initial prediction based on information about the individual prediction target;   wherein the information used by the automated prediction mechanism to generate the initial prediction for each individual prediction target includes values for each of a plurality of input parameters;   for each individual prediction target in the population of individual prediction targets, obtaining current performance information;   based on the current performance information, determining a set of erroneous predictions from the initial predictions;   determining distances between pairs of individual prediction targets based on the information about the individual prediction targets that was used by the automated prediction mechanism to generate the initial predictions;   based on the distances and the set of erroneous predictions, forming one or more clusters of individual prediction targets from the population of individual prediction targets, wherein each cluster includes:
 a core set of erroneous-prediction entities that are individual prediction targets that are:
 associated with erroneous predictions in the initial predictions, and 
 have relatively shorter distances between each other than to erroneous-prediction entities that do not belong to the cluster; and 
 
 a set of boundary accurate-prediction entities that are individual prediction targets that are:
 associated with accurate predictions in the initial predictions; and 
 have distances to the individual prediction targets in the core set of the cluster that place them at boundaries of the core set of the cluster; 
 
   for at least one cluster of the one or more clusters, determining at least one input parameter, of the plurality of input parameters, whose values distinguish the boundary accurate-prediction entities of the cluster from the core set of erroneous-prediction entities of the cluster.   
     
     
         17 . (canceled) 
     
     
         18 . The one or more non-transitory computer media of  claim 16  wherein determining at least one input parameter, of the plurality of input parameters, whose values distinguish the boundary accurate-prediction entities of the cluster from the core set of erroneous-prediction entities of the cluster comprises:
 for at least a particular cluster of the plurality of clusters, determining a DIFF-SET of the particular cluster, wherein the DIFF-SET of the particular cluster is a set of features that best distinguish the core set of erroneous-prediction entities of the particular cluster from the set of boundary accurate-prediction entities of the particular cluster. 
 
     
     
         19 . The one or more non-transitory computer media of  claim 18  wherein:
 the instructions include instructions for determining a critical range for each feature in the DIFF-SET of the particular cluster; 
 wherein, for each feature in the DIFF-SET of the particular cluster, all members of the core set of erroneous-prediction entities of the particular cluster have values for the feature that fall into the critical range for the feature, and all members of the set of boundary accurate-prediction entities of a cluster have values for the feature that fall outside the critical range for the feature. 
 
     
     
         20 . The one or more non-transitory computer media of  claim 19  further comprising instructions for updating the automated prediction mechanism based on the DIFF-SET of the particular cluster and the critical range of each feature in the DIFF-SET of the particular cluster. 
     
     
         21 . The one or more non-transitory computer media of  claim 16  wherein:
 the individual prediction targets are borrowers; 
 the core set of erroneous-prediction entities of each cluster represents a set of borrowers that are delinquent on payments for their respective loans; and 
 the set of boundary accurate-prediction entities of each cluster represents a set of borrowers that are not delinquent on payments for their respective loans.