Method for verifying training data, training system, and computer program product

Abstract

The disclosure provides a method for verifying training data, a training system, and a computer program produce. The method includes: providing a plurality of raw data to a plurality of annotators; retrieving a plurality of labelled results, wherein the labelled results includes a plurality of labelled data, and the labelled data are generated by the annotators via labelling the raw data; determining a plurality of consistencies by comparing the labelled results, and accordingly determining whether the labelled results are valid for training an artificial intelligence machine; in response to determining that the labelled results are valid, determining at least a specific part of the labelled results are valid for training the artificial intelligence machine.

What is claimed is: 
     
         1 . A method for verifying training data, comprising:
 providing a plurality of raw data to a plurality of annotators;   retrieving a plurality of labelled results, wherein the labelled results comprises a plurality of labelled data, and the labelled data are generated by the annotators via labelling the raw data;   determining a plurality of consistencies by comparing the labelled results, and accordingly determining whether the labelled results are valid for training an artificial intelligence machine; and   in response to determining that the labelled results are valid, determining at least a specific part of the labelled results are valid for training the artificial intelligence machine.   
     
     
         2 . The method according to  claim 1 , wherein the labelled results comprises a first labelled result generated by labelling one of the raw data with at least one object category chosen by the annotators, and the step of determining the consistencies by comparing the labelled results, and accordingly determining whether the labelled results are valid for training an artificial intelligence machine comprises:
 generating a recommend result for the first labelled result by comparing the first labelled data in the first labelled result of the annotators;   determining a first consistency score of each annotator on the first labelled result by comparing each labelled data with the recommend result;   determining a second consistency score of the first labelled result based on the first consistency score of each annotator; and   in response to determining that the second consistency score of the first labelled result is higher than a consistency score threshold, determining that the first labelled result is valid for training the artificial intelligence machine.   
     
     
         3 . The method according to  claim 2 , wherein the step of generating the recommend result for the first labelled result by comparing the first labelled data in the first labelled result of the annotators comprises:
 determining a specific object category of the at least one object category as the recommend result, wherein the specific object category has a highest number in the first labelled data.   
     
     
         4 . The method according to  claim 2 , wherein the annotators comprises a first annotator, the first labelled result comprises a specific labelled data labelled by the first annotator, and the step of determining the first consistency score of each annotator on the first labelled result by comparing each labelled data with the recommend result comprises:
 in response to determining that the specific labelled data is identical to the recommend result, determining the first consistency score of the first annotator to be 1;   in response to determining that the specific labelled data is different from the recommend result, determining the first consistency score of the first annotator to be 0.   
     
     
         5 . The method according to  claim 2 , wherein the step of determining the second consistency score of the first labelled result based on the first consistency score of each annotator comprises:
 determining an average of the first consistency score of each annotator as the second consistency score.   
     
     
         6 . The method according to  claim 1 , wherein the annotators comprise a first annotator and each of the labelled results comprises a specific labelled data labelled by the first annotator, and the method further comprises:
 determining a first consistency score of the first annotator on each of the labelled results by comparing the specific labelled data and a recommend result of each labelled result;   determining a third consistency score of the first annotator based on the first consistency score of the first annotator on each of the labelled results;   in response to determining that the third consistency score of the first annotator on each of the labelled results is higher than an annotator score threshold, determining that the first annotator is reliable for labelling the raw data or the labelled data labelled by the first annotator in the labelled results is valid for training the artificial intelligence machine.   
     
     
         7 . The method according to  claim 6 , wherein the step of determining the third consistency score of the first annotator based on the first consistency score of the first annotator on each of the labelled results comprises:
 determining an average of the first consistency score of the first annotator on each of the labelled results as the third consistency score.   
     
     
         8 . The method according to  claim 1 , wherein the raw data comprises a first raw data and a second raw data identical to the first raw data, and a first intra-annotator consistency of the consistencies is proportional to a first consistency of the first annotator labelling the first raw data and the second raw data. 
     
     
         9 . The method according to  claim 1 , wherein the labelled data are generated by labelling at least one region of interest with at least one bounding region in the raw data; wherein for a first raw data of the raw data, the first raw data is labelled with at least one bounding region to generate a first labelled result of the labelled results. 
     
     
         10 . The method according to  claim 9 , wherein the first labelled result has at least one tag corresponding to at least one object category, and the step of determining the consistencies by comparing the labelled results, and accordingly determining whether the labelled results are valid for training the artificial intelligence machine comprises:
 for the first labelled result, identifying at least one target object of each object category, wherein each target object is labelled by at least two of the annotators;   determining a consistency score of the first labelled result based on the at least one target object of each object category and the at least one bounding region labelled by the annotators; and   in response to the consistency score of the first labelled result is higher than a score threshold, determining that the first labelled result is valid for training the artificial intelligence machine.   
     
     
         11 . The method according to  claim 10 , wherein the annotators comprise a first annotator and a second annotator, the first annotator labels the first raw data with at least one first bounding region corresponding to a first object category, the second annotator labels the first raw data with at least one second bounding region corresponding to the first object category, and the step of identifying the at least one target object of each object category comprises:
 determining a plurality of region pairs, wherein each region pair comprises one of the at least one first bounding region and one of the at least one second bounding region;   determining a plurality of correlation coefficients that respectively corresponds to the region pairs, wherein each correlation coefficients characterizes a similarity of one of the region pairs;   merging the at least one first bounding region and the at least one second bounding region into a plurality of groups based on the correlation coefficients that are higher than a correlation threshold, wherein each group at least comprises one of the at least one first bounding region and one of the at least one second bounding region;   for each group, generating a reference region for identifying one of the at least one first target object based on the at least one first bounding region and the at least one second bounding region in each group.   
     
     
         12 . The method according to  claim 9 , wherein the step of merging the at least one first bounding region and the at least one second bounding region into the second groups based on the correlation coefficients that are higher than the correlation threshold comprises:
 (a) retrieving a specific correlation coefficient, wherein the specific correlation coefficient is highest among the correlation coefficients that are higher than the correlation threshold;   (b) retrieving a specific region pair corresponding to the specific correlation coefficient from the region pairs, wherein the specific region pair comprises a first specific region and a second specific region;   (c) determining whether one of the first specific region and the second specific region belongs to an existing group;   (d) in response to determining that neither of the first specific region or the second specific region belongs to the existing group, creating a new group based on the first specific region and the second specific region;   (e) in response to determining that one of the first specific region and the second specific region belongs to the existing group, determining whether another of the first specific region and the second specific region corresponds to the same annotator with a member of the existing group;   (f) in response to determining that the another of the first specific region and the second specific region does not correspond to the same annotator with the member of the existing group, adding the another of the first specific region and the second specific region into the existing group;   (g) excluding the specific correlation coefficient from the correlation coefficients and excluding the specific region pair from the region pairs; and   (h) in response to determining that the region pairs are not empty, returning to step (a).   
     
     
         13 . The method according to  claim 9 , wherein the step of determining the consistency score of the first labelled result based on the at least one target object of each object category and the at least one bounding region labelled by the annotators comprises:
 for each annotator, calculating at least one first consistency score of the at least one object category based on the at least one target object of each object category and the at least one bounding region, and taking an average of the at least one first consistency score to obtain a second consistency score; and   taking an average of the second consistency score of each annotator to obtain the consistency score of the first labelled result.   
     
     
         14 . The method according to  claim 9 , further comprising:
 for a certain annotator, calculating a first consistency score of the at least one object category based on the at least one target object of each object category and the at least one bounding region;   determining whether the certain annotator is reliable for labelling or whether to exclude the labelled data labelled by the certain annotator from training the artificial intelligence machine based on the first consistency score;   in response to determining that the first consistency score is lower than a certain threshold, determining that the certain annotator is unreliable for labelling or excluding the labelled data labelled by the certain annotator from training the artificial intelligence machine; and   in response to determining that the first consistency score is not lower than the certain threshold, determining that the certain annotator is reliable for labelling or not excluding the labelled data labelled by the certain annotator from training the artificial intelligence machine.   
     
     
         15 . The method according to  claim 13 , wherein the step of calculating the at least one first consistency score of the at least one object category based on the at least one target object of each object category and the at least one bounding region comprises:
 for a first annotator, retrieving a first number, wherein the first number characterizes a number of the at least one bounding region of the first annotator that matches at least one identified target object of a first object category;   retrieving a second number, wherein the second number is a sum of the first number, a third number, and a fourth number, wherein the third number is a number of the at least one identified target object that does not match any of the at least one bounding region of the first annotator, and the fourth number is a number of the at least one bounding region of the first annotator that does not match any of the at least one identified target object; and   dividing the first number with the second number to obtain the first consistency score of the first annotator on the first object category.   
     
     
         16 . The method according to  claim 9 , wherein in response to a first annotator does not label any bounding region in the first raw data, generating at least one virtual bounding region outside of the first raw data, wherein each virtual bounding region corresponds to one of the object categories. 
     
     
         17 . The method according to  claim 16 , wherein in response to determining that no target object exists in the first raw data, determining a first consistency score of the first annotator on each object category to be 1. 
     
     
         18 . The method according to  claim 1 , wherein in response to determining that the labelled results are invalid for training the artificial intelligence machine, the method further comprises creating a notification related to the labelled results;
 wherein the annotators comprise a first annotator, a second annotator, and a third annotator, the consistencies comprise a first inter-annotator consistency between the first annotator and the second annotator, a second inter-annotator consistency between the first annotator and the third annotator, and a third inter-annotator consistency between the second annotator and the third annotator, and after the step of creating the notification related to the labelled results, further comprises:   in response to determining that the first inter-annotator consistency is higher than a first threshold, feeding a consistently labelled data between a first labelled data set and a second labelled data set to the artificial machine, wherein the first labelled data set comprises a plurality of first labelled data labelled in the labelled results by the first annotator, and the second labelled data set comprises a plurality of second labelled data labelled in the labelled results by the second annotator.   
     
     
         19 . The method according to  claim 1 , further comprising:
 training the artificial intelligence machine with the specific part of the labelled results to generate an artificial intelligence model.   
     
     
         20 . A training system, comprising:
 a storage circuit, storing a plurality of modules;   a processor, coupled to the storage circuit and accessing the modules to perform following steps:   providing a plurality of raw data to a plurality of annotators;   retrieving a plurality of labelled results, wherein the labelled results comprises a plurality of labelled data, and the labelled data are generated by the annotators via labelling the raw data;   determining a plurality of consistencies by comparing the labelled results, and accordingly determining whether the labelled results are valid for training an artificial intelligence machine;   in response to determining that the labelled results are valid, determining at least a specific part of the labelled results are valid for training the artificial intelligence machine.   
     
     
         21 . A computer program product for use in conjunction with a training system, the computer program product comprising a computer readable storage medium and an executable computer program mechanism embedded therein, the executable computer program mechanism comprising instructions for:
 providing a plurality of raw data to a plurality of annotators;   retrieving a plurality of labelled results, wherein the labelled results comprises a plurality of labelled data, and the labelled data are generated by the annotators via labelling the raw data;   determining a plurality of consistencies by comparing the labelled results, and accordingly determining whether the labelled results are valid for training an artificial intelligence machine;   in response to determining that the labelled results are valid, determining at least a specific part of the labelled results are valid for training the artificial intelligence machine.