System, method and computer program product for generating a training set for a classifier

Abstract

There are provided a system, computer software product and method of generating a training set for a classifier using a processor. The method comprises: receiving a training set comprising training defects each having assigned attribute values, the training defects externally classified into classes comprising first and second major classes and a minor class; training a classifier upon the training set; receiving results of automatic classification of the training defects; automatically identifying a first defect that was externally classified into the first major class and automatically classified into the second major class; automatically identifying by the processor a second defect from the multiplicity of training defects that was externally classified into the minor class and automatically classified to the first or second major classes; and correcting the training set to include the first defect into the second major class, or to include the second defect into the first or the second major class.

The invention claimed is: 
     
       1. An examination system comprising:
 a review tool configured to review at least part of potential defects of an examined object, and assign each of the at least part of the potential defects with a multiplicity of attribute values; and
 a computer-based classifier configured to classify, based on the attribute values as assigned, the at least part of potential defects into a set of classes, the set comprising at least a first major class, a second major class and a first minor class, the classifier further configured to operate in a training mode and be trained based on a training set, wherein during a training mode the classifier is configured to:
 receive the training set comprising a multiplicity of training defects with assigned attribute values, the training defects externally classified into the set of classes; 
 train the classifier upon the training set; 
 automatically classify the multiplicity of training defects into the set of classes based on the respectively assigned attribute values; 
 automatically identify at least one first defect from the multiplicity of training defects that was externally classified into the first major class and classified by the classifier into the second major class; 
 automatically identify a second defect from the multiplicity of training defects that was externally classified into the minor class and classified by the classifier to a class selected from the first major class and the second major class; and 
 correct the training set to include the at least one first defect into the second major class or to include the second defect into the first major class or the second major class. 
 
 
 
     
     
       2. The system of  claim 1 , wherein the classifier is comprised in the review tool. 
     
     
       3. The system of  claim 1 , wherein classification of the training defects is done automatically. 
     
     
       4. The system of  claim 1 , wherein the classifier is further configured to be retrained using the corrected training set. 
     
     
       5. The system of  claim 1 , wherein the classifier is further configured to provide indications to a user regarding the least one first defect or the second defect, and receive from the user classification of the least one first defect to the second major class or classification of the second defect to the first major class or to the second major class. 
     
     
       6. The system of  claim 5 , wherein the indications are provided along with an image of the least one first defect or the second defect. 
     
     
       7. A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to:
 receive a training set comprising a multiplicity of training defects each having assigned attribute values, the multiplicity of training defects externally classified into a set of classes comprising at least a first major class, a second major class and a first minor class; 
 train a classifier upon the training set; 
 receive results of automatic classification by the classifier of the multiplicity of training defects into the set of classes, the automatic classification based on the attribute values; 
 automatically identify at least one first defect from the multiplicity of training defects that was externally classified into the first major class and automatically classified into the second major class; 
 automatically identify a second defect from the multiplicity of training defects that was externally classified into the minor class and automatically classified to a class selected from the first major class and the second major class; and 
 correct the training set to include the at least one first defect into the second major class, or to include the second defect into the first major class or the second major class. 
 
     
     
       8. The computer software product of  claim 7 , wherein the instructions, when read by the computer, further cause the computer to retrain the classifier using the corrected training set. 
     
     
       9. A method of generating a training set for a classifier using a processor operatively connected to a memory, the method comprising:
 receiving by the processor a training set comprising a multiplicity of training defects each having assigned attribute values, the multiplicity of training defects externally classified into a set of classes comprising at least a first major class, a second major class and a first minor class; 
 training by the processor a classifier upon the training set; 
 receiving by the processor results of automatic classification by the classifier of the multiplicity of training defects into the set of classes, the automatic classification based on the attribute values; 
 automatically identifying by the processor at least one first defect from the multiplicity of training defects that was externally classified into the first major class and automatically classified into the second major class; 
 automatically identifying by the processor a second defect from the multiplicity of training defects that was externally classified into the minor class and automatically classified to a class selected from the first major class and the second major class; and 
 correcting by the processor the training set to include the at least one first defect into the second major class, or to include the second defect into the first major class or the second major class. 
 
     
     
       10. The method of  claim 9 , further comprising retraining by the processor the classifier using the corrected training set. 
     
     
       11. The method of  claim 9 , further comprising providing indications to a user regarding the least one first defect or the second defect, and receiving from the user classification of the least one first defect to the second major class or classification of the second defect to the first major class or to the second major class. 
     
     
       12. The method of  claim 11 , wherein the indications are provided along with a confidence level. 
     
     
       13. The method of  claim 12 , wherein the indications are provided along with an image of the least one first defect or the second defect. 
     
     
       14. The method of  claim 9 , wherein the at least one first defect is identified based on a global effect of classifying the defect to the first major class on the automatic classification. 
     
     
       15. The method of  claim 14  wherein the global effect is an effect of the defect on partitioning planes including a partitioning plane between the first class and the second class. 
     
     
       16. The method of  claim 9 , wherein the least one first defect is identified based on presence of a multiplicity of defects from the training set in the vicinity of the least one first defect within a space defined by the attributes, being initially classified to the second major class. 
     
     
       17. The method of  claim 9 , wherein the least one first defect is identified based on a combination of global effect of the least one first defect on the classifier, and presence of a multiplicity of defects in the vicinity of the least one first defect being manually classified to the second major class. 
     
     
       18. The method of  claim 9 , wherein the second defect is identified subject to being within a dense area of defects classified to the first major class or the second major class within an attribute space defined by at least part of the multiplicity of attributes. 
     
     
       19. The method of  claim 9 , wherein the least one first defect or the second defect is identified using a two dimensional confusion matrix having an entry in one dimension for each major class and each minor class, and an entry in another dimension for each major class. 
     
     
       20. The method of  claim 9 , further comprising classifying further defects using the classifier.