Neural implicit function for end-to-end reconstruction of dynamic cryo-em structures

Abstract

A computer-implemented method is provided. The method includes obtaining a plurality of images representing projections of an object placed in a plurality of poses and a plurality of translations; assigning a pose embedding vector, a flow embedding vector and a contrast transfer function (CTF) embedding vector to each image; encoding, by a computer device, a machine learning model comprising a pose network, a flow network, a density network and a CTF network; training the machine learning model using the plurality of images; and reconstructing a 3D structure of the object based on the trained machine learning module.

What is claimed is: 
     
         1 . A computer-implemented method comprising:
 obtaining a plurality of images representing projections of an object placed in a plurality of poses and a plurality of translations;   assigning a pose embedding vector, a flow embedding vector, and a Contrast Transfer Function (CTF) embedding vector to each of the plurality of images;   encoding, by a computer device, a machine learning model comprising a pose network, a flow network, a density network, and a CTF network, wherein the pose network is configured to map an image to a rotation and a translation via the pose embedding vector, the flow network is configured to concatenate a spatial coordinate with the flow embedding vector, the density network is configured to derive a density value in accordance with the spatial coordinate and to generate a projection image, and the CTF network is configured to modulate the projection image appended with the CTF embedding vector to generate a rendered image;   training the machine learning model using the plurality of images; and   reconstructing a 3D structure of the object based on a trained machine learning module.   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising:
 simulating an intensity value of a pixel in the projection image by estimating a continuous integral using a quadrature rule.   
     
     
         3 . The computer-implemented method of  claim 1 , further comprising:
 partitioning the projection image into a plurality of bins, and selecting a pixel from each of the plurality of bins; and   simulating an intensity value of a selected pixel in the projection image by estimating a continuous integral using a quadrature rule.   
     
     
         4 . The computer-implemented method of  claim 2 , further comprising:
 partitioning the projection image into a plurality of patches, and selecting a patch from the plurality of patches; and   training the machine learning model using a selected patch.   
     
     
         5 . The computer-implemented method of  claim 2 , further comprising:
 training the machine learning model by minimizing a mean-square-error (MSE) loss between rendered images with a ground truth.   
     
     
         6 . The computer-implemented method of  claim 1 , further comprising:
 prepending a positional encoding layer to map the spatial coordinate to a high-frequency representation.   
     
     
         7 . The computer-implemented method of  claim 1 , wherein the pose network is configured to output a quaternion representation of the rotation and the translation. 
     
     
         8 . The computer-implemented method of  claim 1 , further comprising:
 obtaining each of the pose embedding vector, the flow embedding vector, and the CTF embedding vector by indexing a dictionary.   
     
     
         9 . The computer-implemented method of  claim 1 , wherein each of the plurality of images is a cryogenic electron microscopy (cryo-EM) image. 
     
     
         10 . The computer-implemented method of  claim 2 , wherein the object is a particle dissolved in an amorphous ice, and each of the plurality of images is a micrograph. 
     
     
         11 . The computer-implemented method of  claim 1 , wherein each of the pose network, the flow network, and the density network is a multi-layer perceptron (MLP), and the CTF network is a convolutional neural network (CNN). 
     
     
         12 . The computer-implemented method of  claim 11 , wherein the MLP is an 8-layer skip-connected MLP of 256 hidden dimensions. 
     
     
         13 . The computer-implemented method of  claim 1 , further comprising:
 training the machine learning model by applying a penalty on the density value obtained during a current batch.   
     
     
         14 . The computer-implemented method of  claim 1 , further comprising:
 training the machine learning model by sampling pixels from the image in accordance with an inverse cumulative density function.   
     
     
         15 . The computer-implemented method of  claim 1 , further comprising:
 pre-training the CTF network by applying a plurality of CTF parameters to white noise patterns.