Mask pattern for protein language models

Abstract

The technology disclosed relates to accessing a multiple sequence alignment that aligns a query residue sequence to a plurality of non-query residue sequences, applying a set of periodically-spaced masks to a first set of residues at a first set of positions in the multiple sequence alignment, and cropping a portion of the multiple sequence alignment that includes the set of periodically-spaced masks at the first set of positions, and a second set of residues at a second set of positions in the multiple sequence alignment to which the set of periodically-spaced masks is not applied. The first set of residues includes a residue-of-interest at a position-of-interest in the query residue sequence.

What is claimed is: 
     
         1 . A computer-implemented method of variant pathogenicity prediction, including:
 accessing a multiple sequence alignment that aligns a query residue sequence to a plurality of non-query residue sequences;   applying a set of periodically-spaced masks to a first set of residues at a first set of positions in the multiple sequence alignment, wherein the first set of residues includes a residue-of-interest at a position-of-interest in the query residue sequence;   cropping a portion of the multiple sequence alignment that includes
 (i) the set of periodically-spaced masks at the first set of positions, and 
 (ii) a second set of residues at a second set of positions in the multiple sequence alignment to which the set of periodically-spaced masks is not applied; and 
   generating a pathogenicity prediction for a variant at the position-of-interest based on the portion of the multiple sequence alignment.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the multiple sequence alignment aligns the query residue sequence to the plurality of non-query residue sequences along a per-position dimension and along a per-sequence dimension. 
     
     
         3 . The computer-implemented method of  claim 2 , wherein the set of periodically-spaced masks is applied along the per-sequence dimension within a window of sequences in the multiple sequence alignment. 
     
     
         4 . The computer-implemented method of  claim 3 , wherein the set of periodically-spaced masks is applied along the per-position dimension within a window of positions across the window of sequences in the multiple sequence alignment. 
     
     
         5 . The computer-implemented method of  claim 1 , wherein the portion has a predetermined width and a predetermined height. 
     
     
         6 . The computer-implemented method of  claim 5 , wherein the portion is padded to compensate for multiple sequence alignments that have widths smaller the predetermined width of the portion. 
     
     
         7 . The computer-implemented method of  claim 2 , wherein the set of periodically-spaced masks is distributed along the per-sequence dimension into subsets of periodically-spaced masks. 
     
     
         8 . The computer-implemented method of  claim 7 , wherein the subsets of periodically-spaced masks correspond to sequences in a window of sequences. 
     
     
         9 . The computer-implemented method of  claim 1 , wherein the set of periodically-spaced masks has a pattern. 
     
     
         10 . The computer-implemented method of  claim 9 , further including right-shifting a cropping window used for the cropping to minimize padding of the portion. 
     
     
         11 . The computer-implemented method of  claim 10 , further including left-shifting the cropping window to minimize the padding of the portion. 
     
     
         12 . The computer-implemented method of  claim 1 , further including configuring a cropping window to position the position-of-interest in a center column of the portion. 
     
     
         13 . The computer-implemented method of  claim 12 , further including configuring the cropping window to position the position-of-interest adjacent to the center column. 
     
     
         14 . The computer-implemented method of  claim 1 , further including substituting, in the portion, the set of periodically-spaced masks at the first set of positions with learned mask embeddings, and substituting, in the portion, and the second set of residues at the second set of positions with learned residue embeddings. 
     
     
         15 . The computer-implemented method of  claim 14 , further including substituting, in the portion, the set of periodically-spaced masks at the first set of positions and the second set of residues at the second set of positions with learned position embeddings. 
     
     
         16 . The computer-implemented method of  claim 15 , further including chunking the portion with learned mask embeddings, the learned residue embeddings, and the learned position embeddings into a plurality of chunks. 
     
     
         17 . The computer-implemented method of  claim 16 , further including processing the plurality of chunks as an aggregate and generating an alternative representation of the portion. 
     
     
         18 . The computer-implemented method of  claim 1 , further including generating the pathogenicity prediction for the variant based on a difference between a log probability of the variant and a log probability of a corresponding reference amino acid less an entropy evaluated over amino acid-wise predictions. 
     
     
         19 . A system including one or more processors coupled to memory, the memory loaded with computer instructions to predict variant pathogenicity, the instructions, when executed on the one or more processors, implement actions comprising:
 accessing a multiple sequence alignment that aligns a query residue sequence to a plurality of non-query residue sequences;   applying a set of periodically-spaced masks to a first set of residues at a first set of positions in the multiple sequence alignment, wherein the first set of residues includes a residue-of-interest at a position-of-interest in the query residue sequence;   cropping a portion of the multiple sequence alignment that includes
 (i) the set of periodically-spaced masks at the first set of positions, and 
 (ii) a second set of residues at a second set of positions in the multiple sequence alignment to which the set of periodically-spaced masks is not applied; and 
   generating a pathogenicity prediction for a variant at the position-of-interest based on the portion of the multiple sequence alignment.   
     
     
         20 . A non-transitory computer readable storage medium impressed with computer program instructions to predict variant pathogenicity, the instructions, when executed on a processor, implement actions comprising:
 accessing a multiple sequence alignment that aligns a query residue sequence to a plurality of non-query residue sequences;   applying a set of periodically-spaced masks to a first set of residues at a first set of positions in the multiple sequence alignment, wherein the first set of residues includes a residue-of-interest at a position-of-interest in the query residue sequence;   cropping a portion of the multiple sequence alignment that includes
 (i) the set of periodically-spaced masks at the first set of positions, and 
 (ii) a second set of residues at a second set of positions in the multiple sequence alignment to which the set of periodically-spaced masks is not applied; and 
   generating a pathogenicity prediction for a variant at the position-of-interest based on the portion of the multiple sequence alignment.