Molecular structure transformers for property prediction

Abstract

Computer-implemented methods may include accessing a multi-dimensional embedding space that supports relating embeddings of molecules to predicted values of a given property of the molecules. The method may also include identifying one or more points of interest within the embedding space based on the predicted values. Each of the one or more points of interest may include a set of coordinate values within the multi-dimensional embedding space and may be associated with a corresponding predicted value of the given property. The method may further include generating, for each of the one or more points of interest, a structural representation of a molecule by transforming the set of coordinate values included in the point of interest using a decoder network. The method may include outputting a result that identifies, for each of the one or more points of interest, the structural representation of the molecule corresponding to the point of interest.

What is claimed is: 
     
       1. A computer-implemented method comprising:
 accessing a multi-dimensional embedding space that supports relating embeddings of molecules to predicted values of a given property of the molecules; 
 identifying one or more points of interest within the multi-dimensional embedding space based on the predicted values, wherein each of the one or more points of interest:
 includes a set of coordinate values within the multi-dimensional embedding space, 
 conveys spatial information of atoms or bonds in a molecule, and 
 is associated with a corresponding predicted value of the given property; 
 
 generating, for each of the one or more points of interest, a structural representation of the molecule by transforming the set of coordinate values included in a point of interest using a decoder network, wherein training of the decoder network included learning to transform positions within the embedding space to outputs representing molecular-structure characteristics, wherein the training of the decoder network was performed at least in part concurrently to training an encoder network to transform positions within the embedding space to predictions corresponding to values of the given property; and 
 outputting a result that identifies, for each of the one or more points of interest, the structural representation of the molecule corresponding to the point of interest. 
 
     
     
       2. The method of  claim 1 , wherein training of the encoder network included learning to transform partial or complete bond string and position (BSP) representations of molecules into positions within the embedding space, and wherein each BSP representation identifies relative positions of atoms connected by a bond in the represented molecule. 
     
     
       3. The method of  claim 2 , wherein each BSP representation of the molecules used to train the encoder network includes a set of coordinates for each of the atoms connected by the bond in the represented molecule and further identifies each of the atoms connected by the bond in the represented molecule. 
     
     
       4. The method of  claim 2 , wherein the BSP representation of the molecules used to train the encoder network to identify, for each of at least some bonds in a respective molecule, a bond type. 
     
     
       5. The method of  claim 2 , wherein a format of the structural representation identified in the result is different than the BSP representation. 
     
     
       6. The method of  claim 1 , wherein training of the encoder network included learning to transform partial or complete molecular graph representations of molecules into positions within the embedding space, and wherein each molecular graph representation identifies angles and distances of bonds in the represented molecule. 
     
     
       7. The method of  claim 1 , wherein the decoder network and the encoder network were trained by training a transformer model that uses self-attention, wherein the transformer model includes the decoder network and the encoder network. 
     
     
       8. The method of  claim 1 , wherein the decoder network and the encoder network were trained by training a transformer model that includes an attention head. 
     
     
       9. The method of  claim 1 , further comprising training a machine-learning model that includes the encoder network and the decoder network by:
 accessing a set of supplemental training elements, wherein each of the set of training elements includes a representation of a structure of a corresponding given molecule; 
 masking, for each supplemental training element in the set of supplemental training elements, at least part of the representation to obscure at least part of the structure of the corresponding given molecule; and 
 training the machine-learning model to predict the obscured at least part of the structure. 
 
     
     
       10. A system comprising:
 one or more data processors; and 
 a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform a method comprising: 
 accessing a multi-dimensional embedding space that supports relating embeddings of molecules to predicted values of a given property of the molecules; 
 identifying one or more points of interest within the multi-dimensional embedding space based on the predicted values, wherein each of the one or more points of interest:
 includes a set of coordinate values within the multi-dimensional embedding space, 
 conveys spatial information of atoms or bonds in a molecule, and 
 is associated with a corresponding predicted value of the given property; 
 
 generating, for each of the one or more points of interest, a structural representation of the molecule by transforming the set of coordinate values included in a point of interest using a decoder network, wherein training of the decoder network included learning to transform positions within the embedding space to outputs representing molecular-structure characteristics, wherein the training of the decoder network was performed at least in part concurrently to training an encoder network to transform positions within the embedding space to predictions corresponding to values of the given property; and 
 outputting a result that identifies, for each of the one or more points of interest, the structural representation of the molecule corresponding to the point of interest. 
 
     
     
       11. The system of  claim 10 , wherein training of the encoder network included learning to transform partial or complete bond string and position (BSP) representations of molecules into positions within the embedding space, and wherein each BSP representation identifies relative positions of atoms connected by a bond in the represented molecule. 
     
     
       12. The system of  claim 11 , wherein each BSP representation of the molecules used to train the encoder network includes a set of coordinates for each of the atoms connected by the bond in the represented molecule and further identifies each of the atoms connected by the bond in the represented molecule. 
     
     
       13. The system of  claim 11 , wherein the BSP representation of the molecules used to train the encoder network to identify, for each of at least some bonds in a respective molecule, a bond type. 
     
     
       14. The system of  claim 11 , wherein a format of the structural representation identified in the result is different than the BSP representation. 
     
     
       15. The system of  claim 10 , wherein training of the encoder network included learning to transform partial or complete molecular graph representations of molecules into positions within the embedding space, and wherein each molecular graph representation identifies angles and distances of bonds in the represented molecule. 
     
     
       16. The system of  claim 10 , wherein the decoder network and the encoder network were trained by training a transformer model that uses self-attention, wherein the transformer model includes the decoder network and the encoder network. 
     
     
       17. The system of  claim 10 , wherein the decoder network and the encoder network were trained by training a transformer model that includes an attention head. 
     
     
       18. The system of  claim 10 , the method further comprising training a machine-learning model that includes the encoder network and the decoder network by:
 accessing a set of supplemental training elements, wherein each of the set of training elements includes a representation of a structure of a corresponding given molecule; 
 masking, for each supplemental training element in the set of supplemental training elements, at least part of the representation to obscure at least part of the structure of the corresponding given molecule; and 
 training the machine-learning model to predict the obscured at least part of the structure. 
 
     
     
       19. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform a method comprising:
 accessing a multi-dimensional embedding space that supports relating embeddings of molecules to predicted values of a given property of the molecules; 
 identifying one or more points of interest within the multi-dimensional embedding space based on the predicted values, wherein each of the one or more points of interest:
 includes a set of coordinate values within the multi-dimensional embedding space, 
 conveys spatial information of atoms or bonds in a molecule, and 
 is associated with a corresponding predicted value of the given property; 
 
 generating, for each of the one or more points of interest, a structural representation of the molecule by transforming the set of coordinate values included in a point of interest using a decoder network, wherein training of the decoder network included learning to transform positions within the embedding space to outputs representing molecular-structure characteristics, wherein the training of the decoder network was performed at least in part concurrently to training an encoder network to transform positions within the embedding space to predictions corresponding to values of the given property; and 
 outputting a result that identifies, for each of the one or more points of interest, the structural representation of the molecule corresponding to the point of interest.