Machine-learnt field-specific standardization

Abstract

A system tokenizes raw values and corresponding standardized values into raw token sequences and corresponding standardized token sequences. A machine-learning model learns standardization from token insertions and token substitutions that modify the raw token sequences to match the corresponding standardized token sequences. The system tokenizes an input value into an input token sequence. The machine-learning model determines a probability of inserting an insertion token after an insertion markable token in the input token sequence. If the probability of inserting the insertion token satisfies a threshold, the system inserts the insertion token after the insertion markable token in the input token sequence. The machine-learning model determines a probability of substituting a substitution token for a substitutable token in the input token sequence. If the probability of substituting the substitution token satisfies another threshold, the system substitutes the substitution token for the substitutable token in the input token sequence.

The invention claimed is: 
     
       1. A system for machine-learnt field-specific standardization, the system comprising:
 one or more processors; and 
 a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the one or more processors to: 
 train a machine-learning model to tokenize raw values and corresponding standardized values into raw token sequences and corresponding standardized token sequences; 
 train the machine-learning model to learn standardization of each one of a plurality of specific data fields from token insertions and token substitutions that modify the raw token sequences to match the corresponding standardized token sequences; 
 tokenize an input value into an input token sequence; 
 determine, by the machine-learning model, a probability of inserting an insertion token after an insertion markable token in the input token sequence, based on learning a standardization of one of the plurality of specific data fields which comprises inserting the insertion token after a class of the insertion markable token in the raw token sequences; 
 determine whether the probability of inserting the insertion token exceeds a threshold; 
 insert the insertion token, which is punctuation inserted after the insertion markable token in the input token sequence, in response to a determination that the probability of inserting the insertion token exceeds the threshold; 
 determine, by the machine-learning model, a probability of substituting a substitution token, for a substitutable token that differs in location from the insertion markable token, in the input token sequence, based on learning a standardization of one of the plurality of specific data fields which comprises substituting the substitution token for a class of the substitutable token in the raw token sequences; 
 determine whether the probability of substituting the substitution token exceeds another threshold; and 
 substitute the substitution token, which is one of a nil character, a letter that is part of a word, or a digit that is part of a number, for the substitutable token in the input token sequence, in response to a determination that the probability of substituting the substitution token exceeds the other threshold. 
 
     
     
       2. The system of  claim 1 , wherein the plurality of instructions, when executed, will further cause the one or more processors to receive at least one input value. 
     
     
       3. The system of  claim 1 , wherein tokenizing the raw values and the corresponding standardized values into the raw token sequences and the corresponding standardized token sequences comprises aligning the raw token sequences with the corresponding standardized token sequences. 
     
     
       4. The system of  claim 1 , wherein determining the probability of inserting the insertion token after the insertion markable token in the input token sequence is based on a count of instances that the insertion token is inserted after the class of the insertion markable token in any raw token sequence and a count of instances that any raw token sequence includes the class of the insertion markable token. 
     
     
       5. The system of  claim 1 , wherein determining the probability of substituting the substitution token for the substitutable token in the input token sequence is based on a count of instances that the substitution token is substituted for a class of the substitutable token in any raw token sequence and a count of instances that any raw token sequence includes the class of the substitutable token. 
     
     
       6. The system of  claim 1 , wherein the plurality of instructions, when executed, will further cause the one or more processors to rearrange tokens in the input token sequence. 
     
     
       7. The system of  claim 1 , wherein the plurality of instructions, when executed, will further cause the one or more processors to join tokens in the input token sequence. 
     
     
       8. A computer program product comprising computer-readable program code to be executed by one or more processors when retrieved from a non-transitory computer-readable medium, the program code including instructions to:
 train a machine-learning model to tokenize raw values and corresponding standardized values into raw token sequences and corresponding standardized token sequences; 
 train the machine-learning model to learn standardization of each one of a plurality of specific data fields from token insertions and token substitutions that modify the raw token sequences to match the corresponding standardized token sequences; 
 tokenize an input value into an input token sequence; 
 determine, by the machine-learning model, a probability of inserting an insertion token after an insertion markable token in the input sequence, based on learning a standardization of one of the plurality of specific data fields which comprises inserting the insertion token after a class of the insertion markable token in the raw token sequences; 
 determine whether the probability of inserting the insertion token exceeds a threshold; 
 insert the insertion token, which is punctuation inserted after the insertion markable token in the input token sequence, in response to a determination that the probability of inserting the insertion token exceeds the threshold; 
 determine, by the machine-learning model, a probability of substituting a substitution token, for a substitutable token that differs in location from the insertion markable token, in the input token sequence, based on learning a standardization of one of the plurality of specific data fields which comprises substituting the substitution token for a class of the substitutable token in the raw token sequences; 
 determine whether the probability of substituting the substitution token exceeds another threshold; and 
 substitute the substitution token, which is one of a nil character, a letter that is part of a word, or a digit that is part of a number, for the substitutable token in the input token sequence, in response to a determination that the probability of substituting the substitution token exceeds the other threshold. 
 
     
     
       9. The computer program product of  claim 8 , wherein the program code further includes instructions to receive at least one input value. 
     
     
       10. The computer program product of  claim 8 , wherein tokenizing the raw values and the corresponding standardized values into the raw token sequences and the corresponding standardized token sequences comprises aligning the raw token sequences with the corresponding standardized token sequences. 
     
     
       11. The computer program product of  claim 8 , wherein determining the probability of inserting the insertion token after the insertion markable token in the input token sequence is based on a count of instances that the insertion token is inserted after a class of the insertion markable token in any raw token sequence and a count of instances that any raw token sequence includes the class of the insertion markable token. 
     
     
       12. The computer program product of  claim 8 , wherein determining the probability of substituting the substitution token for the substitutable token in the input token sequence is based on a count of instances that the substitution token is substituted for a class of the substitutable token in any raw token sequence and a count of instances that any raw token sequence includes the class of the substitutable token. 
     
     
       13. The computer program product of  claim 8 , wherein the program code further includes instructions to rearrange tokens in the input token sequence. 
     
     
       14. The computer program product of  claim 8 , wherein the program code further includes instructions to join tokens in the input token sequence. 
     
     
       15. A method for machine-learnt field-specific standardization, the method comprising:
 training a machine-learning model to tokenize raw values and corresponding standardized values into raw token sequences and corresponding standardized token sequences; 
 training the machine-learning model to learn standardization of each one of a plurality of specific data fields from token insertions and token substitutions that modify the raw token sequences to match the corresponding standardized token sequences; 
 tokenizing an input value into an input token sequence; 
 determining, by the machine-learning model, a probability of inserting an insertion token after an insertion markable token in the input token sequence, based on learning a standardization of one of the plurality of specific data fields which comprises inserting the insertion token after a class of the insertion markable token in the raw token sequences; 
 determining whether the probability of inserting the insertion token exceeds a threshold; 
 inserting the insertion token, which is punctuation inserted after the insertion markable token in the input token sequence, in response to a determination that the probability of inserting the insertion token exceeds the threshold; 
 determining, by the machine-learning model, a probability of substituting a substitution token, for a substitutable token that differs in location from the insertion markable token, in the input token sequence, based on learning a standardization of one of the plurality of specific data fields which comprises substituting the substitution token for a class of the substitutable token in the raw token sequences; 
 determining whether the probability of substituting the substitution token exceeds another threshold; and 
 substituting the substitution token, which is one of nil, a letter that is part of a word, or a digit that is part of a number, for the substitutable token in the input token sequence, in response to a determination that the probability of substituting the substitution token exceeds the other threshold. 
 
     
     
       16. The method of  claim 15 , wherein the method further comprises receiving at least one input value. 
     
     
       17. The method of  claim 15 , wherein tokenizing the raw values and the corresponding standardized values into the raw token sequences and the corresponding standardized token sequences comprises aligning the raw token sequences with the corresponding standardized token sequences. 
     
     
       18. The method of  claim 15 , wherein determining the probability of inserting the insertion token after the insertion markable token in the input token sequence is based on a count of instances that the insertion token is inserted after a class of the insertion markable token in any raw token sequence and a count of instances that any raw token sequence includes the class of the insertion markable token. 
     
     
       19. The method of  claim 15 , wherein determining the probability of substituting the substitution token for the substitutable token in the input token sequence is based on a count of instances that the substitution token is substituted for a class of the substitutable token in any raw token sequence and a count of instances that any raw token sequence includes the class of the substitutable token. 
     
     
       20. The method of  claim 15 , wherein the method further comprises
 rearranging tokens in the input token sequence; and 
 joining tokens in the input token sequence.