Training corpus refinement and incremental updating

Abstract

Training corpus refinement and incremental updating includes obtaining a training corpus having training samples, refining the training corpus to produce a refined training corpus of data, by applying to the training corpus overlap and noise reduction treatments, maintaining an incremental intelligence database based on filtered user feedback and having candidate feedback training samples to augment the refined training corpus, controlling integration of the candidate feedback training samples with the refined training corpus, and augmenting the refined training corpus with at least some of the candidate feedback training samples to produce an augmented training corpus.

What is claimed is: 
     
       1. A computer-implemented method comprising:
 obtaining a training corpus of data, the training corpus comprising a collection of training samples; 
 refining the obtained training corpus to produce a refined training corpus of data, wherein refining the obtained training corpus comprises applying to the obtained training corpus an overlap treatment and a noise reduction treatment, the overlap treatment and noise reduction treatment filtering out one or more samples of the collection of training samples; 
 maintaining an incremental intelligence database based on filtered user feedback, the incremental intelligence database storing candidate feedback training samples based on the filtered user feedback to augment the refined training corpus; and 
 controlling integration of the candidate feedback training samples with the refined training corpus, the controlling being based at least in part on whether accuracy of classification performed based on training that includes the candidate feedback training samples as part of the refined training corpus is greater than accuracy of classification performed based on training that does not include the candidate feedback training samples as part of the refined training corpus, wherein the controlling integration selectively determines whether to include the candidate feedback training samples in the refined training corpus and comprises:
 comparing a first determined accuracy, the first determined accuracy being the accuracy of classification performed based on training that includes the candidate feedback training samples as part of the refined training corpus, to a second determined accuracy, the second determined accuracy being the accuracy of classification performed based on training that does not include the candidate feedback training samples as part of the refined training corpus; 
 determining whether the first determined accuracy is greater than the second determined accuracy; and 
 based on determining that the first determined accuracy is greater than the second determined accuracy, augmenting the refined training corpus with at least some of the candidate feedback training samples to produce an augmented training corpus. 
 
 
     
     
       2. The method of  claim 1 , wherein the refining comprises:
 establishing a plurality of corpus feature vectors representative of the collection of training samples; 
 building an entropy meter database, the building the entropy meter database comprising extracting and storing tokens from each corpus feature vector of the plurality of feature vectors based on a Term Frequency-Inverse Class Frequency (TFICF) weight; 
 performing the overlap treatment to identify one or more overlapping training samples; 
 performing the noise detection treatment to identify one or more noisy training samples; and 
 filtering out the one or more samples from the training corpus, the one or more samples filtered-out from the training corpus being at least one selected from the group consisting of the one or more overlapping samples and the one or more noisy samples, the filtering out being based on quality scoring and risk determination, wherein the filtering produces the refined training corpus. 
 
     
     
       3. The method of  claim 2 , wherein the establishing the plurality of corpus feature vectors comprises:
 assigning each sample of the collection of training samples to a respective class of a plurality of corpus classes; and 
 based on the assigning, building a corpus feature vector for each class of the plurality of corpus classes, each feature vector comprising weighted TFICF tokens, and the built corpus feature vector being one corpus feature vector of the plurality of corpus feature vectors. 
 
     
     
       4. The method of  claim 2 , wherein the performing the overlap treatment comprises:
 identifying overlapping corpus classes based on pair-wise comparisons of the plurality of feature vectors; 
 identifying overlapping tokens of the plurality of feature vectors; 
 obtaining a standard class-wise token database having class-specific dictionaries of standard keywords, phrases, and synonyms; 
 identifying the one or more overlapping training samples based on the overlapping corpus classes, overlapping tokens, and standard class-wise token database; and 
 storing the overlapping training samples for a recommendation engine to perform the filtering out. 
 
     
     
       5. The method of  claim 2 , wherein the performing the noise detection treatment comprises:
 comparing tokens of the entropy meter database against tokens of the standard class-wise token database to identify anomalous tokens; 
 identifying, based on the comparing, corpus classes having one or more noisy tokens; and 
 storing, for a recommendation engine, indications of the noisy tokens and the identified corpus classes having one or more noisy tokens, to perform the filtering out. 
 
     
     
       6. The method of  claim 1 , wherein the maintaining comprises:
 feeding the filtered user feedback into a reinforcement learning model; and 
 adding the candidate feedback training samples of the filtered feedback to the incremental intelligence database based on classifying feedback training samples of the filtered user feedback into feedback classes and on assessing intra-class and inter-class effects of the classified feedback training samples. 
 
     
     
       7. The method of  claim 6 , wherein the classifying the feedback training samples comprises assigning each sample of the feedback training samples to a respective class of the feedback classes, and, based on the assigning, building a feedback feature vector for each class of the feedback classes, and wherein the method further comprises:
 establishing an overlap intensity threshold, a noise intensity threshold, and a class entropy threshold; and 
 for each pair of a respective feedback feature vector and respective corpus feature vector:
 determining a respective entropy intersection value for the pair; and 
 performing processing based on whether the class of the feedback feature vector of the pair is a class represented in the training corpus. 
 
 
     
     
       8. The method of  claim 7 , wherein, based on the class of the feedback feature vector being a new class not represented in the training corpus, the method further comprises storing, in a new class/intent database, any sample of the feedback training samples classified to the feedback class for which the feedback feature vector is built, based on the entropy intersection parameter being less than the overlap intensity threshold and on the feedback feature vector passing a noise intensity check based on the noise intensity threshold. 
     
     
       9. The method of  claim 8 , wherein the method further comprises accumulating in the new class/intent database a plurality of feedback feature vectors having the new class, and, based on (i) accumulating a threshold number of samples classified in that new class and on (ii) identifying a pattern among those samples, forwarding the accumulated samples to the incremental intelligence storage database as at least some of the candidate feedback training samples. 
     
     
       10. The method of  claim 7 , wherein, based on the class of the feedback feature vector being represented in the training corpus, the method further comprises determining whether to add samples classified into the class of the feedback feature vector to the incremental intelligence storage database, the determining being based at least in part on the overlap intensity threshold, noise intensity threshold, and class entropy threshold, and whether the class of the corpus feature vector of the pair matches the feedback class of the feedback feature vector. 
     
     
       11. The method of  claim 1 , further comprising building a filtered feedback database for the filtered feedback, the building the filtered feedback database comprising determining class-wise accuracy based on numbers of hit cases, missed cases, false-positive cases, and rejected cases as reported in the user feedback, and storing to the filtered feedback database feedback training samples for failed classes for which class-wise accuracy is above a threshold class accuracy level. 
     
     
       12. The method of  claim 1 , wherein the maintaining comprises continually updating the incremental intelligence database based on additional filtered user feedback to produce updated candidate feedback training samples, and wherein the method further comprises repeating the controlling integration and the augmenting using the updated candidate feedback training samples. 
     
     
       13. The method of  claim 1 , further comprising training a classification model with the augmented training corpus. 
     
     
       14. A computer program product comprising:
 a computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising:
 obtaining a training corpus of data, the training corpus comprising a collection of training samples; 
 refining the obtained training corpus to produce a refined training corpus of data, wherein refining the obtained training corpus comprises applying to the obtained training corpus an overlap treatment and a noise reduction treatment, the overlap treatment and noise reduction treatment filtering out one or more samples of the collection of training samples; 
 maintaining an incremental intelligence database based on filtered user feedback, the incremental intelligence database storing candidate feedback training samples based on the filtered user feedback to augment the refined training corpus; and 
 controlling integration of the candidate feedback training samples with the refined training corpus, the controlling being based at least in part on whether accuracy of classification performed based on training that includes the candidate feedback training samples as part of the refined training corpus is greater than accuracy of classification performed based on training that does not include the candidate feedback training samples as part of the refined training corpus, wherein the controlling integration selectively determines whether to include the candidate feedback training samples in the refined training corpus and comprises: 
 comparing a first determined accuracy, the first determined accuracy being the accuracy of classification performed based on training that includes the candidate feedback training samples as part of the refined training corpus, to a second determined accuracy, the second determined accuracy being the accuracy of classification performed based on training that does not include the candidate feedback training samples as part of the refined training corpus; 
 determining whether the first determined accuracy is greater than the second determined accuracy; and 
 based on determining that the first determined accuracy is greater than the second determined accuracy, augmenting the refined training corpus with at least some of the candidate feedback training samples to produce an augmented training corpus. 
 
 
     
     
       15. The computer program product of  claim 14 , wherein the refining comprises:
 establishing a plurality of corpus feature vectors representative of the collection of training samples, wherein the establishing the plurality of corpus feature vectors comprises:
 assigning each sample of the collection of training samples to a respective class of a plurality of corpus classes; and 
 based on the assigning, building a corpus feature vector for each class of the plurality of corpus classes, each feature vector comprising weighted TFICF tokens, and the built corpus feature vector being one corpus feature vector of the plurality of corpus feature vectors; 
 
 building an entropy meter database, the building the entropy meter database comprising extracting and storing tokens from each corpus feature vector of the plurality of feature vectors based on a Term Frequency-Inverse Class Frequency (TFICF) weight; 
 performing the overlap treatment to identify one or more overlapping training samples, wherein the performing the overlap treatment comprises:
 identifying overlapping corpus classes based on pair-wise comparisons of the plurality of feature vectors; 
 identifying overlapping tokens of the plurality of feature vectors; 
 obtaining a standard class-wise token database having class-specific dictionaries of standard keywords, phrases, and synonyms; 
 identifying the one or more overlapping training samples based on the overlapping corpus classes, overlapping tokens, and standard class-wise token database; and 
 storing the overlapping training samples for a recommendation engine to perform the filtering out; 
 
 performing the noise detection treatment to identify one or more noisy training samples, wherein the performing the noise detection treatment comprises:
 comparing tokens of the entropy meter database against tokens of the standard class-wise token database to identify anomalous tokens; 
 identifying, based on the comparing, corpus classes having one or more noisy tokens; and 
 storing, for a recommendation engine, indications of the noisy tokens and the identified corpus classes having one or more noisy tokens, to perform the filtering out; and 
 
 filtering out the one or more samples from the training corpus, the one or more samples filtered-out from the training corpus being at least one selected from the group consisting of the one or more overlapping samples and the one or more noisy samples, the filtering out being based on quality scoring and risk determination, wherein the filtering produces the refined training corpus. 
 
     
     
       16. The computer program product of  claim 14 , wherein the maintaining comprises:
 feeding the filtered user feedback into a reinforcement learning model; and 
 adding the candidate feedback training samples of the filtered feedback to the incremental intelligence database based on classifying feedback training samples of the filtered user feedback into feedback classes and on assessing intra-class and inter-class effects of the classified feedback training samples, wherein the classifying the feedback training samples comprises assigning each sample of the feedback training samples to a respective class of the feedback classes, and, based on the assigning, building a feedback feature vector for each class of the feedback classes, and wherein the method further comprises:
 establishing an overlap intensity threshold, a noise intensity threshold, and a class entropy threshold; and 
 for each pair of a respective feedback feature vector and respective corpus feature vector:
 determining a respective entropy intersection value for the pair; and 
 performing processing based on whether the class of the feedback feature vector of the pair is a class represented in the training corpus, wherein:
 based on the class of the feedback feature vector being a new class not represented in the training corpus, the method further comprises: 
  storing, in a new class/intent database, any sample of the feedback training samples classified to the feedback class for which the feedback feature vector is built, based on the entropy intersection parameter being less than the overlap intensity threshold and on the feedback feature vector passing a noise intensity check based on the noise intensity threshold; 
  accumulating in the new class/intent database a plurality of feedback feature vectors having the new class; and 
  based on (i) accumulating a threshold number of samples classified in that new class and on (ii) identifying a pattern among those samples, forwarding the accumulated samples to the incremental intelligence storage database as at least some of the candidate feedback training samples; and 
 based on the class of the feedback feature vector being represented in the training corpus, the method further comprises: 
 
 determining whether to add samples classified into the class of the feedback feature vector to the incremental intelligence storage database, the determining being based at least in part on the overlap intensity threshold, noise intensity threshold, and class entropy threshold, and whether the class of the corpus feature vector of the pair matches the feedback class of the feedback feature vector. 
 
 
 
     
     
       17. The computer program product of  claim 14 , wherein the method further comprises building a filtered feedback database for the filtered feedback, the building the filtered feedback database comprising determining class-wise accuracy based on numbers of hit cases, missed cases, false-positive cases, and rejected cases as reported in the user feedback, and storing to the filtered feedback database feedback training samples for failed classes for which class-wise accuracy is above a threshold class accuracy level. 
     
     
       18. A computer system comprising:
 a memory; and 
 a processor in communication with the memory, wherein the computer system is configured to perform a method comprising:
 obtaining a training corpus of data, the training corpus comprising a collection of training samples; 
 refining the obtained training corpus to produce a refined training corpus of data, wherein refining the obtained training corpus comprises applying to the obtained training corpus an overlap treatment and a noise reduction treatment, the overlap treatment and noise reduction treatment filtering out one or more samples of the collection of training samples; 
 maintaining an incremental intelligence database based on filtered user feedback, the incremental intelligence database storing candidate feedback training samples based on the filtered user feedback to augment the refined training corpus; and 
 controlling integration of the candidate feedback training samples with the refined training corpus, the controlling being based at least in part on whether accuracy of classification performed based on training that includes the candidate feedback training samples as part of the refined training corpus is greater than accuracy of classification performed based on training that does not include the candidate feedback training samples as part of the refined training corpus, wherein the controlling integration selectively determines whether to include the candidate feedback training samples to the refined training corpus and comprises:
 comparing a first determined accuracy, the first determined accuracy being the accuracy of classification performed based on training that includes the candidate feedback training samples as part of the refined training corpus, to a second determined accuracy, the second determined accuracy being the accuracy of classification performed based on training that does not include the candidate feedback training samples as part of the refined training corpus; 
 determining whether the first determined accuracy is greater than the second determined accuracy; and 
 based on determining that the first determined accuracy is greater than the second determined accuracy, augmenting the refined training corpus with at least some of the candidate feedback training samples to produce an augmented training corpus. 
 
 
 
     
     
       19. The computer system of  claim 18 , wherein the refining comprises:
 establishing a plurality of corpus feature vectors representative of the collection of training samples, wherein the establishing the plurality of corpus feature vectors comprises:
 assigning each sample of the collection of training samples to a respective class of a plurality of corpus classes; and 
 based on the assigning, building a corpus feature vector for each class of the plurality of corpus classes, each feature vector comprising weighted TFICF tokens, and the built corpus feature vector being one corpus feature vector of the plurality of corpus feature vectors; 
 
 building an entropy meter database, the building the entropy meter database comprising extracting and storing tokens from each corpus feature vector of the plurality of feature vectors based on a Term Frequency-Inverse Class Frequency (TFICF) weight; 
 performing the overlap treatment to identify one or more overlapping training samples, wherein the performing the overlap treatment comprises:
 identifying overlapping corpus classes based on pair-wise comparisons of the plurality of feature vectors; 
 identifying overlapping tokens of the plurality of feature vectors; 
 obtaining a standard class-wise token database having class-specific dictionaries of standard keywords, phrases, and synonyms; 
 identifying the one or more overlapping training samples based on the overlapping corpus classes, overlapping tokens, and standard class-wise token database; and 
 storing the overlapping training samples for a recommendation engine to perform the filtering out; 
 
 performing the noise detection treatment to identify one or more noisy training samples, wherein the performing the noise detection treatment comprises:
 comparing tokens of the entropy meter database against tokens of the standard class-wise token database to identify anomalous tokens; 
 identifying, based on the comparing, corpus classes having one or more noisy tokens; and 
 storing, as the one or more noisy training samples, those training samples in which the identified noisy tokens appear, for a recommendation engine to perform the filtering out; and 
 filtering out the one or more samples from the training corpus, the one or more samples filtered-out from the training corpus being at least one selected from the group consisting of the one or more overlapping samples and the one or more noisy samples, the filtering out being based on quality scoring and risk determination, wherein the filtering produces the refined training corpus. 
 
 
     
     
       20. The computer system of  claim 18 , wherein the maintaining comprises:
 feeding the filtered user feedback into a reinforcement learning model; and 
 adding the candidate feedback training samples of the filtered feedback to the incremental intelligence database based on classifying feedback training samples of the filtered user feedback into feedback classes and on assessing intra-class and inter-class effects of the classified feedback training samples, wherein the classifying the feedback training samples comprises assigning each sample of the feedback training samples to a respective class of the feedback classes, and, based on the assigning, building a feedback feature vector for each class of the feedback classes, and wherein the method further comprises:
 establishing an overlap intensity threshold, a noise intensity threshold, and a class entropy threshold; and 
 for each pair of a respective feedback feature vector and respective corpus feature vector:
 determining a respective entropy intersection value for the pair; and 
 performing processing based on whether the class of the feedback feature vector of the pair is a class represented in the training corpus, wherein:
 based on the class of the feedback feature vector being a new class not represented in the training corpus, the method further comprises: 
  storing, in a new class/intent database, any sample of the feedback training samples classified to the feedback class for which the feedback feature vector is built, based on the entropy intersection parameter being less than the overlap intensity threshold and on the feedback feature vector passing a noise intensity check based on the noise intensity threshold; 
  accumulating in the new class/intent database a plurality of feedback feature vectors having the new class; and 
  based on (i) accumulating a threshold number of samples classified in that new class and on (ii) identifying a pattern among those samples, forwarding the accumulated samples to the incremental intelligence storage database as at least some of the candidate feedback training samples; and 
 
 based on the class of the feedback feature vector being represented in the training corpus, the method further comprises determining whether to add samples classified into the class of the feedback feature vector to the incremental intelligence storage database, the determining being based at least in part on the overlap intensity threshold, noise intensity threshold, and class entropy threshold, and whether the class of the corpus feature vector of the pair matches the feedback class of the feedback feature vector.