Real time holography using learned error feedback

Abstract

Techniques related to generating holographic images for a holographic heads up display are discussed. Such techniques include application of a machine learning model to the target image to generate data that is used to enable the determination of a phase pattern via an iterative propagation feedback model. The iterative propagation feedback model is used to generate a feedback strength value, which is then used to generate a phase diffraction pattern for presentation at a holographic plane of the heads up display.

What is claimed is: 
     
         1 . A head-mounted display (HMD) system comprising:
 a holographic plane; and   a system to present a holographic image at the holographic plane, the system comprising circuitry configured to:   apply a machine learning model to a target holographic image to generate a feedback strength value;   apply an iterative propagation feedback model based on the target holographic image and the feedback strength value to generate a phase diffraction pattern; and   output the phase diffraction pattern for presentation at the holographic plane for generation of a corresponding holographic image.   
     
     
         2 . The HMD system of  claim 1 , wherein to apply the iterative propagation feedback model based on the target holographic image and the feedback strength value, the circuitry is configured to:
 receive a current phase and amplitude image plane estimation; and   replace an amplitude plane of the current phase and amplitude image plane estimation with a next target amplitude image comprising:
 a sum of the target holographic image and a product of the feedback strength value and a correction term comprising a difference between the target holographic image and the current phase and amplitude image plane to generate a current amplitude adjusted image. 
   
     
     
         3 . The HMD system of  claim 2 , wherein to apply the iterative propagation feedback model based on the target holographic image and the feedback strength value, the circuitry is configured to:
 apply an inverse propagation model to a previous amplitude adjusted image to generate a phase and amplitude holographic plane estimation;   normalize an amplitude plane of the phase and amplitude holographic plane estimation to generate a phase only diffraction pattern image; and   apply a forward propagation model to the phase only diffraction pattern image to generate the current phase and amplitude image plane estimation.   
     
     
         4 . The HMD system of  claim 1 , wherein the machine learning model corresponds to a trained neural network and the machine learning model comprises at least one of fewer convolutional kernels with respect to the trained neural network or a reduced bit depth of weights of the machine learning model with respect to the trained neural network. 
     
     
         5 . The HMD system of  claim 4 , wherein the target holographic image is downsampled prior to application of the machine learning model. 
     
     
         6 . The HMD system of  claim 1 , wherein the target holographic image comprises first and second color planes, and the circuitry is configured to apply the machine learning model to the target holographic image to generate the feedback strength value for the first color plane and a second feedback strength value for the second color plane. 
     
     
         7 . The HMD system of  claim 6 , wherein the circuitry is further to:
 apply the iterative propagation feedback model based on the second color plane of the target holographic image; and   use the second feedback strength value to generate a second phase diffraction pattern for presentation at the holographic plane or a second holographic plane.   
     
     
         8 . The HMD system of  claim 1 , wherein the holographic image is a three dimensional holographic image. 
     
     
         9 . A method for presenting holographic images at a holographic plane of a head-mounted display, the method comprising:
 applying, via one or more processors, a machine learning model to a target holographic image to generate a feedback strength value;   applying an iterative propagation feedback model based on the target holographic image and the feedback strength value to generate a phase diffraction pattern; and   outputting the phase diffraction pattern for presentation at the holographic plane for generation of a corresponding holographic image.   
     
     
         10 . The method of  claim 9 , wherein applying the iterative propagation feedback model based on the target holographic image and the feedback strength value comprises:
 receiving a current phase and amplitude image plane estimation;   replacing an amplitude plane of the current phase and amplitude image plane estimation with a next target amplitude image comprising a sum of the target holographic image and a product of the feedback strength value and a correction term comprising a difference between the target holographic image and the current phase and amplitude image plane to generate a current amplitude adjusted image;   applying an inverse propagation model to a previous amplitude adjusted image to generate a phase and amplitude holographic plane estimation;   normalizing an amplitude plane of the phase and amplitude holographic plane estimation to generate a phase only diffraction pattern image; and   applying a forward propagation model to the phase only diffraction pattern image to generate the current phase and amplitude image plane estimation.   
     
     
         11 . The method of  claim 9 , wherein the machine learning model corresponds to a trained neural network and the machine learning model comprises at least one of fewer convolutional kernels with respect to the trained neural network or a reduced bit depth of weights of the machine learning model with respect to the trained neural network. 
     
     
         12 . The method of  claim 11 , further comprising downsampling the target holographic image prior to applying the machine learning model. 
     
     
         13 . The method of  claim 9 , wherein the target holographic image comprises first and second color planes and the method further comprises applying the machine learning model to the target holographic image to generate the feedback strength value for the first color plane and a second feedback strength value for the second color plane. 
     
     
         14 . The method of  claim 13 , further comprising:
 applying the iterative propagation feedback model based on the second color plane of the target holographic image; and   using the second feedback strength value to generate a second phase diffraction pattern for presentation at the holographic plane or a second holographic plane.   
     
     
         15 . The method of  claim 9 , wherein the holographic image is a three dimensional holographic image. 
     
     
         16 . A non-transitory machine readable medium comprising a plurality of instructions that, in response to being executed by a processor of a holographic head-mounted display system, causes the processor to perform operations comprising:
 applying a machine learning model to a target holographic image to generate a feedback strength value;   applying an iterative propagation feedback model based on the target holographic image and the feedback strength value to generate a phase diffraction pattern for presentation at a holographic plane of the holographic head-mounted display system; and   outputting the phase diffraction pattern for presentation at the holographic plane for generation of a corresponding holographic image.   
     
     
         17 . The non-transitory machine readable medium as in  claim 16 , wherein applying the iterative propagation feedback model based on the target holographic image and the feedback strength value includes operations comprising:
 receiving a current phase and amplitude image plane estimation;   replacing an amplitude plane of the current phase and amplitude image plane estimation with a next target amplitude image comprising a sum of the target holographic image and a product of the feedback strength value and a correction term comprising a difference between the target holographic image and the current phase and amplitude image plane to generate a current amplitude adjusted image;   applying an inverse propagation model to a previous amplitude adjusted image to generate a phase and amplitude holographic plane estimation;   normalizing an amplitude plane of the phase and amplitude holographic plane estimation to generate a phase only diffraction pattern image; and   applying a forward propagation model to the phase only diffraction pattern image to generate the current phase and amplitude image plane estimation.   
     
     
         18 . The non-transitory machine readable medium as in  claim 16 , wherein the machine learning model corresponds to a trained neural network, the machine learning model comprises at least one of fewer convolutional kernels with respect to the trained neural network or a reduced bit depth of weights of the machine learning model with respect to the trained neural network, and the operations further comprise downsampling the target holographic image prior to applying the machine learning model. 
     
     
         19 . The non-transitory machine readable medium as in  claim 16 , wherein the target holographic image comprises first and second color planes and the operations further comprise applying the machine learning model to the target holographic image to generate the feedback strength value for the first color plane and a second feedback strength value for the second color plane. 
     
     
         20 . The non-transitory machine readable medium as in  claim 19 , wherein the holographic image is a three dimensional holographic image and the operations further comprise:
 applying the iterative propagation feedback model based on the second color plane of the target holographic image; and   using the second feedback strength value to generate a second phase diffraction pattern for presentation at the holographic plane or a second holographic plane.