Pathogenicity language model

Abstract

A system comprises chunking logic that chunks (or splits) a multiple sequence alignment (MSA) into chunks, first attention logic that attends to a representation of the chunks and produces a first attention output, first aggregation logic that produces a first aggregated output that contains those features in the first attention output that correspond to masked residues in the plurality of masked residues, mask revelation logic that produces an informed output based on the first aggregated output and a Boolean mask, second attention logic that attends to the informed output and produces a second attention output based on masked residues revealed by the Boolean mask, second aggregation logic that produces a second aggregated output that contains those features in the second attention output that correspond to masked residues concealed by the Boolean mask, and output logic that produces identifications of the masked residues based on the second aggregated output.

What is claimed is: 
     
         1 . A computer-implemented method, including:
 accessing a multiple sequence alignment (MSA), wherein the MSA has p rows and r columns, wherein the p rows correspond to p protein sequences, and wherein the r columns correspond to r residue positions;   accessing a mask grid, wherein the mask grid has m mask distributions, and wherein each of the m mask distributions has k periodically-spaced masks at k ordinal positions that begin at varying offsets from a first residue position in the mask grid;   applying the m mask distributions to m protein sequences in the p protein sequences to generate a partially-masked MSA that contains masked residues and unmasked residues, where p > m;   translating the masked residues and the unmasked residues into learned embeddings, concatenating the learned embeddings with residue position embeddings to generate an embedded representation of the partially-masked MSA;   chunking the embedded representation into a series of chunks, concatenating chunks in the series of chunks into a stack, and translating the stack into a compressed representation of the embedded representation, wherein the compressed representation has m rows and r columns;   iteratively applying axial-attention across the m rows and the r columns of the compressed representation, and interleaving the applied attention to generate an updated representation of the compressed representation, wherein the updated representation has m rows and r columns;   aggregating, from the updated representation, k updated representation tiles, wherein each of the k updated representation tiles contains those updated representation features of the updated representation that correspond to the masked residues, wherein each of the k updated representation tiles has m rows and k columns, wherein a given column in the k columns of a given updated representation tile contains a respective subset of the updated representation features, wherein the respective subset is located at a given ordinal position in the k ordinal positions, and wherein the given ordinal position is represented by the given column;   aggregating, from the embedded representation, k embedding tiles corresponding to the k updated representation tiles, wherein each of the k embedding tiles contains those embedding features in a first chunk of the series of chunks that are translations of the masked residues, wherein each of the k embedding tiles has m rows and k columns, wherein a given column in the k columns of a given embedding tile contains a respective subset of the embedding features, wherein the respective subset is located at a given ordinal position in the k ordinal positions, and wherein the given ordinal position is represented by the given column;   applying k Boolean tiles to the k embedding tiles to generate k Booleaned embedding tiles, wherein each of the k Boolean tiles has m rows and k columns, wherein each of the k Boolean tiles causes concealment of a corresponding one of the k columns in a corresponding one of the k embedding tiles, and causes revelation of other ones of the k columns in the corresponding one of the k embedding tiles, and wherein each of the k Booleaned embedding tiles has m rows and k columns;   concatenating the k Booleaned embedding tiles with the k updated representation tiles to generate k concatenated tiles, and translating the k concatenated tiles into k compressed tile representations of the k concatenated tiles, wherein each of the k compressed tile representations has m rows and k columns;   iteratively applying self-attention to the k compressed tile representations to generate interpretations of those compressed tile features in the k compressed tile representations that correspond to those embedding features in the k embedding tiles that are revealed by the k Boolean tiles;   aggregating those interpreted features from the interpretations that correspond to those embedding features in the k embedding tiles that are concealed by the k Boolean tiles to generate an aggregated representation of the interpretations, wherein the aggregated representation has m rows and k columns; and   translating the aggregated representation into identities of the masked residues.   
     
     
         2 . The computer-implemented of  claim 1 , further including using a one-hot encoding scheme to translate twenty naturally-occurring residues, a gap residue, and a mask into respective one-hot encoded vectors. 
     
     
         3 . The computer-implemented of  claim 2 , further including training a neural network to generate respective learned embeddings for the respective one-hot encoded vectors. 
     
     
         4 . The computer-implemented of  claim 3 , wherein the masked residues and the unmasked residues are translated into the learned embeddings based on a lookup table that maps the respective one-hot encoded vectors to the respective learned embeddings. 
     
     
         5 . The computer-implemented of  claim 1 , wherein the chunks are concatenated into the stack along a channel dimension. 
     
     
         6 . The computer-implemented of  claim 1 , wherein the stack is translated into the compressed representation by processing the stack through a linear projection. 
     
     
         7 . The computer-implemented of  claim 6 , wherein the linear projection uses a plurality of one-dimensional (1D) convolution filters. 
     
     
         8 . The computer-implemented of  claim 1 , wherein the aggregated representation is translated into the identities of the masked residues by processing the aggregated representation through a revelation output head. 
     
     
         9 . The computer-implemented of  claim 1 , wherein p = m. 
     
     
         10 . The computer-implemented of  claim 1 , wherein each of the k Boolean tiles causes concealment of the corresponding one of the k columns in the corresponding one of the k embedding tiles, and causes revelation of at least some of the other ones of the k columns in the corresponding one of the k embedding tiles. 
     
     
         11 . The computer-implemented of  claim 1 , wherein each of the k Boolean tiles causes concealment of a corresponding subset of the k columns in the corresponding one of the k embedding tiles, and causes revelation of at least some of the other ones of the k columns in the corresponding one of the k embedding tiles. 
     
     
         12 . A system, comprising:
 memory storing a multiple sequence alignment (MSA) with a plurality of masked residues; chunking logic configured to chunk the MSA into a series of chunks;   first attention logic configured to attend to a representation of the series of chunks and produce a first attention output;   first aggregation logic configured to produce a first aggregated output that contains those features in the first attention output that correspond to masked residues in the plurality of masked residues;   mask revelation logic configured to produce an informed output based on the first aggregated output and a Boolean mask that, on a subset-by-subset basis, alternates between concealing a given subset of the masked residues and revealing remaining subsets of the masked residues;   second attention logic configured to attend to the informed output and produce a second attention output based on masked residues revealed by the Boolean mask;   second aggregation logic configured to produce a second aggregated output that contains those features in the second attention output that correspond to masked residues concealed by the Boolean mask; and   output logic configured to produce identifications of the masked residues based on the second aggregated output.   
     
     
         13 . The system of  claim 12 , wherein the first attention logic uses axial-attention. 
     
     
         14 . The system of  claim 12 , wherein the second attention logic uses self-attention. 
     
     
         15 . A computer-implemented method, including:
 accessing a multiple sequence alignment (MSA), wherein the MSA has p rows and r columns, wherein the p rows correspond to p protein sequences, and wherein the r columns correspond to r residue positions;   accessing a mask grid, wherein the mask grid has m mask distributions, and wherein each of the m mask distributions has k periodically-spaced masks at k ordinal positions;   applying the m mask distributions to m protein sequences in the p protein sequences to generate a partially-masked MSA that contains masked residues and unmasked residues, where p > m;   translating the masked residues and the unmasked residues into learned embeddings, concatenating the learned embeddings with residue position embeddings to generate an embedded representation of the partially-masked MSA;   chunking the embedded representation into a series of chunks, concatenating chunks in the series of chunks into a stack, and translating the stack into a compressed representation of the embedded representation;   iteratively applying axial-attention across m rows and r columns of the compressed representation, and interleaving the applied attention to generate an updated representation of the compressed representation;   aggregating, from the updated representation, k updated representation tiles, wherein each of the k updated representation tiles contains those updated representation features of the updated representation that correspond to the masked residues;   aggregating, from the embedded representation, k embedding tiles corresponding to the k updated representation tiles, wherein each of the k embedding tiles contains those embedding features in a first chunk of the series of chunks that are translations of the masked residues;   applying k Boolean tiles to the k embedding tiles to generate k Booleaned embedding tiles, wherein each of the k Boolean tiles causes concealment of a corresponding one of k columns in a corresponding one of the k embedding tiles, and causes revelation of other ones of the k columns in the corresponding one of the k embedding tiles;   concatenating the k Booleaned embedding tiles with the k updated representation tiles to generate k concatenated tiles, and translating the k concatenated tiles into k compressed tile representations of the k concatenated tiles;   iteratively applying self-attention to the k compressed tile representations to generate interpretations of those compressed tile features in the k compressed tile representations that correspond to those embedding features in the k embedding tiles that are revealed by the k Boolean tiles;   aggregating those interpreted features from the interpretations that correspond to those embedding features in the k embedding tiles that are concealed by the k Boolean tiles to generate an aggregated representation of the interpretations; and   translating the aggregated representation into identities of the masked residues.   
     
     
         16 . The computer-implemented of  claim 15 , wherein the k periodically-spaced masks of at least some of the m mask distributions begin at varying offsets from a first residue position in mask grid. 
     
     
         17 . The computer-implemented of  claim 16 , wherein the k periodically-spaced masks of at least some of the m mask distributions begin at a same offset from the first residue position. 
     
     
         18 . The computer-implemented of  claim 15 , wherein the compressed representation has m rows and r columns. 
     
     
         19 . The computer-implemented of  claim 15 , wherein each of the k updated representation tiles has m rows and k columns, wherein a given column in the k columns of a given updated representation tile contains a respective subset of the updated representation features, wherein the respective subset is located at a given ordinal position in the k ordinal positions, and wherein the given ordinal position is represented by the given column. 
     
     
         20 . The computer-implemented of  claim 15 , wherein each of the k embedding tiles has m rows and k columns, wherein a given column in the k columns of a given embedding tile contains a respective subset of the embedding features, wherein the respective subset is located at a given ordinal position in the k ordinal positions, and wherein the given ordinal position is represented by the given column.