Systems and methods for generating and training convolutional neural networks using biological sequences and relevance scores derived from structural, biochemical, population and evolutionary data

Abstract

We describe systems and methods for generating and training convolutional neural networks using biological sequences and relevance scores derived from structural, biochemical, population and evolutionary data. The convolutional neural networks take as input biological sequences and additional information and output molecular phenotypes. Biological sequences may include DNA, RNA and protein sequences. Molecular phenotypes may include protein-DNA interactions, protein-RNA interactions, protein-protein interactions, splicing patterns, polyadenylation patterns, and microRNA-RNA interactions, which may be described using numerical, categorical or ordinal attributes. Intermediate layers of the convolutional neural networks are weighted using relevance score sequences, for example, conservation tracks. The resulting molecular phenotype convolutional neural networks may be used in genetic testing, to identify drug targets, to identify patients that respond similarly to a drug, to ascertain health risks, or to connect patients that have similar molecular phenotypes.

The invention claimed is: 
     
       1. A system for weighting convolutional layers in a molecular phenotype convolutional neural network (MPCNN), the system comprising:
 a. the MPCNN comprising at least three layers, each of the at least three layers configured to receive inputs and produce outputs, a first layer of the at least three layers configured to obtain a biological sequence comprising a plurality of positions, a last layer of the at least three layers representing a molecular phenotype, each layer of the at least three layers other than the first layer configured to receive inputs from the produced outputs of one or more prior layers of the at least three layers; 
 b. one or more of the at least three layers configured as convolutional layers, each of the convolutional layers comprising one or more convolutional filters linking the received inputs of the convolutional layer to produced outputs of the convolutional layer, the received inputs of the convolutional layer comprising a plurality of convolutional layer input positions, the produced outputs of the convolutional layer comprising a plurality of convolutional layer output positions; and 
 c. one or more weighting units, each of the one or more weighting units linked to at least one of the one or more convolutional filters of a convolutional layer, each of the one or more weighting units associated with a relevance score sequence, each of the relevance score sequences comprising a plurality of relevance score sequence positions, each of the plurality of relevance score sequence positions associated with a numerical value, wherein the numerical value quantifies a biological relevance of a corresponding position in the biological sequence with respect to the at least one of the one or more convolutional filters of a convolutional layer, each of the one or more weighting units configured to use the associated relevance score sequence to weight operations of the associated convolutional filter of the one or more convolutional filters. 
 
     
     
       2. The system of  claim 1 , wherein at least one of the one or more weighting units is configured to use the associated relevance score sequence to weight the produced outputs of the associated convolutional layer. 
     
     
       3. The system of  claim 1 , wherein at least one of the one or more weighting units is configured to use the associated relevance score sequence to weight the received inputs of the associated convolutional layer. 
     
     
       4. The system of  claim 1 , wherein one or more of the at least three layers are configured as pooling layers, each of the pooling layers comprising a pooling unit linking received inputs of the pooling layer to produced outputs of the pooling layer, the received inputs of the pooling layer comprising a plurality of pooling layer input positions, the produced outputs of the pooling layer comprising a plurality of pooling layer output positions, wherein the received inputs of the pooling layer are linked to the produced outputs of at least one of the one or more convolutional layers. 
     
     
       5. The system of  claim 1 , wherein one or more of the at least three layers other than the first layer are configured as fully connected layers, wherein the produced outputs of each of the one or more fully connected layers are obtained at least in part by multiplying the received inputs of the fully connected layer by corresponding parameters to produce a plurality of products, determining a sum of the plurality of products, and applying a linear or a nonlinear function to the sum. 
     
     
       6. The system of  claim 1 , wherein the one or more relevance score sequences are obtained from evolutionary conservation sequences, population allele frequency sequences, nucleosome positioning sequences, ribonucleic acid (RNA)-secondary structure sequences, protein secondary structure sequences, or retroviral insertion sequences. 
     
     
       7. The system of  claim 1 , further comprising an encoder configured to encode the biological sequence as a vector sequence. 
     
     
       8. The system of  claim 1 , further comprising an MPCNN training unit configured to train the MPCNN using a plurality of training cases, each of the plurality of training cases comprising a biological sequence and a molecular phenotype. 
     
     
       9. The system of  claim 1 , further comprising a relevance score neural network configured to generate the one or more relevance score sequences. 
     
     
       10. The system of  claim 9 , further comprising a relevance score neural network training unit configured to train the relevance score neural network using a plurality of training cases, each of the plurality of training cases comprising a biological sequence and a relevance score sequence. 
     
     
       11. A method for weighting layers in a molecular phenotype convolutional neural network (MPCNN), the method comprising:
 a. obtaining the MPCNN comprising at least three layers, each of the at least three layers receiving inputs and producing outputs, a first layer of the at least three layers obtaining a biological sequence comprising a plurality of positions, a last layer of the at least three layers representing a molecular phenotype, each layer of the at least three layers other than the first layer receiving inputs from the produced outputs of one or more prior layers of the at least three layers, wherein one or more of the at least three layers are convolutional layers, each of the convolutional layers comprising one or more convolutional filters linking the received inputs of the convolutional layer to produced outputs of the convolutional layer, the received inputs of the convolutional layer comprising a plurality of convolutional layer input positions, the produced outputs of the convolutional layer comprising a plurality of convolutional layer output positions; 
 b. obtaining one or more relevance score sequences, each of the one or more relevance score sequences comprising a plurality of relevance score sequence positions, each of the plurality of relevance score sequence positions associated with a numerical value, wherein the numerical value quantifies a biological relevance of a corresponding position in the biological sequence with resect to the at least one of the one or more convolutional filters of a convolutional layer; and 
 c. applying one or more weighting operations, wherein each weighting operation of the one or more weighting operations comprises using an associated relevance score sequence in the one or more relevance score sequences to weight operations of an associated convolutional filter of the one or more convolutional filters. 
 
     
     
       12. The method of  claim 11 , wherein applying at least one of the one or more weighting operations comprises using the associated relevance score sequence to weight the produced outputs of the associated convolutional layer. 
     
     
       13. The method of  claim 11 , wherein applying at least one of the one or more weighting operations comprises using the associated relevance score sequence to weight the received inputs of the associated convolutional layer. 
     
     
       14. The method of  claim 11 , wherein one or more of the at least three layers are configured as pooling layers, each of the pooling layers performing a pooling operation to link the received inputs of the pooling layer to produced outputs of the pooling layer, the received inputs of the pooling layer comprising a plurality of pooling layer input positions, the produced outputs of the pooling layer comprising a plurality of pooling layer output positions, wherein the received inputs of the pooling layer are linked to the produced outputs of at least one of the one or more convolutional layers. 
     
     
       15. The method of  claim 11 , wherein one or more of the at least three layers other than the first layer are configured as fully connected layers, wherein the produced outputs of each of the one or more fully connected layers are obtained at least in part by multiplying the received inputs of the fully connected layer by corresponding parameters to produce a plurality of products, determining a sum of the plurality of products, and applying a linear or a nonlinear function to the sum. 
     
     
       16. The method of  claim 11 , wherein the one or more relevance score sequences are obtained from evolutionary conservation sequences, population allele frequency sequences, nucleosome positioning sequences, RNA-secondary structure sequences, protein secondary structure sequences, or retroviral insertion sequences. 
     
     
       17. The method of  claim 11 , further comprising performing an encoding operation that encodes the biological sequence as a vector sequence. 
     
     
       18. The method of  claim 11 , further comprising training the MPCNN using a plurality of training cases, each of the plurality of training cases comprising a biological sequence and a molecular phenotype. 
     
     
       19. The method of  claim 11 , further comprising generating the one or more relevance score sequences using a relevance score neural network. 
     
     
       20. The method of  claim 19 , further comprising training the relevance score neural network using a plurality of training cases, each of the plurality of training cases comprising a biological sequence and a relevance score sequence. 
     
     
       21. The system of  claim 8 , wherein training the MPCNN comprises adjusting parameters of the MPCNN using gradients of the parameters. 
     
     
       22. The system of  claim 21 , wherein adjusting parameters of the MPCNN comprises one or more of: a batch gradient descent, a stochastic gradient descent, a dropout, and a conjugate gradient method. 
     
     
       23. The system of  claim 9 , wherein the relevance score neural network comprises a fully connected neural network, a convolutional neural network, a multi-task neural network, a recurrent neural network, a long short-term memory neural network, an autoencoder, or a combination thereof. 
     
     
       24. The system of  claim 10 , wherein training the relevance score neural network comprises adjusting parameters of the relevance score neural network using gradients of the relevance score neural network. 
     
     
       25. The system of  claim 24 , wherein adjusting parameters of the relevance score neural network comprises one or more of: a batch gradient descent, a stochastic gradient descent, a dropout, and a conjugate gradient method. 
     
     
       26. The method of  claim 18 , wherein training the MPCNN comprises adjusting parameters of the MPCNN using gradients of the parameters. 
     
     
       27. The method of  claim 26 , wherein adjusting parameters of the MPCNN comprises one or more of: a batch gradient descent, a stochastic gradient descent, a dropout, and a conjugate gradient method. 
     
     
       28. The method of  claim 19 , wherein the relevance score neural network comprises a fully connected neural network, a convolutional neural network, a multi-task neural network, a recurrent neural network, a long short-term memory neural network, an autoencoder, or a combination thereof. 
     
     
       29. The method of  claim 20 , wherein training the relevance score neural network comprises adjusting parameters of the relevance score neural network using gradients of the relevance score neural network. 
     
     
       30. The method of  claim 29 , wherein adjusting parameters of the relevance score neural network comprises one or more of: a batch gradient descent, a stochastic gradient descent, a dropout, and a conjugate gradient method. 
     
     
       31. The method of  claim 1 , wherein the MPCNN outputs molecular phenotypes comprising numerical values which quantify aspects of biological molecules of cells. 
     
     
       32. The method of  claim 11 , wherein the MPCNN outputs molecular phenotypes comprising numerical values which quantify aspects of biological molecules of cells.