Hand calibration using single depth camera

Abstract

A system generates a user hand shape model from a single depth camera. The system includes the single depth camera and a hand tracking unit. The single depth camera generates single depth image data of a user's hand. The hand tracking unit applies the single depth image data to a neural network model to generate heat maps indicating locations of hand features. The locations of hand features are used to generate a user hand shape model customized to the size and shape of the user's hand. The user hand shape model is defined by a set of principle component hand shapes defining a hand shape variation space. The limited number of principle component hand shape models reduces determination of user hand shape to a smaller number of variables, and thus provides for a fast calibration of the user hand shape model.

What is claimed is: 
     
       1. A system comprising:
 a single depth camera configured to generate single depth image data of a user's hand; 
 a circuitry configured to:
 receive the single depth image data of the user's hand from the single depth camera; 
 determine, based on applying the single depth image data to a neural network model, heat maps indicating locations of hand features; 
 determine a hand pose based on the locations of the hand features; 
 determine an input hand shape model based on the hand pose; 
 determine a set of principal component hand shape models, the principal component hand shape models defining orthogonal dimensions of hand size variance with respect to each other such that the set of principal component hand shape models defines a hand size variation space; 
 determine a user hand shape model based on fitting the input hand shape model to the set of principal component hand shape models; and 
 render an image of the user's hand based on the user hand shape model; and 
 
 a display device configured to present the rendered image to the user. 
 
     
     
       2. The system of  claim 1 , wherein the user hand shape model includes a combination of the set of principal component hand shape models. 
     
     
       3. The system of  claim 2 , wherein the circuitry configured to determine the user hand shape model based on fitting the input hand shape model to the set of principal component hand shape models includes the circuitry being configured to:
 determine principal hand meshes of the set principal component hand shape models; 
 map vertices of the input hand shape model to vertices of the principal hand meshes; 
 determine error values between the vertices of the input hand shape model and the vertices of the principal hand meshes; and 
 determine the user hand shape model as a combination of the principal component hand shape models that minimizes the error values between the vertices of the input hand shape model to the vertices of the principal hand meshes. 
 
     
     
       4. The system of  claim 3 , wherein the circuitry configured to map the vertices of the input hand shape model to the vertices of the principal hand meshes includes the circuitry being configured to apply a procrustes transformation. 
     
     
       5. The system of  claim 1 , wherein the circuitry is further configured to:
 receive second single depth image data of the user's hand from the single depth camera; 
 determine, based on applying the second single depth image data to the neural network model, second heat maps indicating second locations of the hand features; 
 determine a second pose of the user's hand based on fitting the second locations of the hand features to the user hand shape model; and 
 determine a user input based on the second pose of the user's hand. 
 
     
     
       6. The system of  claim 1 ,
 wherein the display device is located in a head-mounted display (HMD). 
 
     
     
       7. The system of  claim 6 , wherein the single depth camera is attached to the HMD. 
     
     
       8. The system of  claim 1 , wherein the circuitry configured to determine a hand pose based on the location of the hand features includes the circuitry being configured to:
 determine degrees of freedom of the hand features; and 
 determine values for the degrees of freedom that fit the locations of the hand features. 
 
     
     
       9. The system of  claim 1 , wherein the circuitry configured to determine the input hand shape model based on the hand pose includes the circuitry being configured to:
 determine a rest skeletal structure based on the hand pose; and 
 determine vertices of the input hand shape model based on the rest skeletal structure. 
 
     
     
       10. The system of  claim 1 , wherein the heat maps indicate the predicated locations of the hand features in 3-dimensional (3D) space. 
     
     
       11. The system of  claim 1 , wherein the circuitry is further configured to:
 determine a plurality of input hand shape models from the single depth image data, the single depth image data including a plurality of poses of the user's hand; and 
 determine the user hand shape model based on fitting the plurality of input hand shape models to the set of principal component hand shape models. 
 
     
     
       12. A device, comprising:
 a circuitry configured to:
 receive single depth image data of a user's hand from a single depth camera, 
 determine, based on applying the single depth image data to a neural network model, heat maps indicating locations of hand features; 
 determine a hand pose based on the locations of the hand features; 
 determine an input hand shape model based on the hand pose; 
 determine a set of principal component hand shape models, the principal component hand shape models defining orthogonal dimensions of hand size variance with respect to each other such that the set of principal component hand shape models defines a hand size variation space; 
 determine a user hand shape model based on fitting the input hand shape model to the set of principal component hand shape models; and 
 render an image of the user's hand based on the user hand shape model; and 
 
 a display configured to present the rendered image to the user. 
 
     
     
       13. The device of  claim 12 , wherein the user hand shape model includes a combination of the set of principal component hand shape models. 
     
     
       14. The device of  claim 12 , wherein the circuitry configured to determine the user hand shape model based on fitting the input hand shape model to the set of principal component hand shape models includes the circuitry being configured to:
 determine principal hand meshes of the set principal component hand shape models; 
 map vertices of the input hand shape model to vertices of the principal hand meshes; 
 determine error values between the vertices of the input hand shape model and the vertices of the principal hand meshes; and 
 determine the user hand shape model as a combination of the principal component hand shape models that minimizes the error values between the vertices of the input hand shape model to the vertices of the principal hand meshes. 
 
     
     
       15. The device of  claim 14 , wherein the circuitry configured to map the vertices of the input hand shape model to the vertices of the principal hand meshes includes the circuitry being configured to apply a procrustes transformation. 
     
     
       16. The device of  claim 14 , wherein the circuitry is further configured to:
 receive second single depth image data of the user's hand from the single depth camera; 
 determine, based on applying the second single depth image data to the neural network model, second heat maps indicating second locations of the hand features; 
 determine a second pose of the user's hand based on fitting the second locations of the hand features to the user hand shape model; and 
 determine a user input based on the second pose of the user's hand. 
 
     
     
       17. The device of  claim 12 , wherein the circuitry configured to determine a hand pose based on the location of the hand features includes the circuitry being configured to:
 determine degrees of freedom of the hand features; and 
 determine values for the degrees of freedom that fit the locations of the hand features. 
 
     
     
       18. The device of  claim 12 , wherein the circuitry configured to determine the input hand shape model based on the hand pose includes the circuitry being configured to:
 determine a rest skeletal structure based on the hand pose; and 
 determine the vertices of the input hand shape model based on the rest skeletal structure. 
 
     
     
       19. The device of  claim 12 , wherein the heat maps indicate the predicated locations of the hand features in 3-dimensional (3D) space. 
     
     
       20. The device of  claim 12 , wherein the circuitry is further configured to:
 determine a plurality of input hand shape models from the single depth image data, the single depth image data including a plurality of poses of the user's hand; and 
 determine the user hand shape model based on fitting the plurality of input hand shape models to the set of principal component hand shape models. 
 
     
     
       21. A head-mounted display (HMD), comprising:
 a circuitry configured to:
 receive single depth image data of a user's hand from a single depth camera; 
 determine, based on applying the single depth image data to a neural network model, heat maps indicating locations of hand features; 
 determine a hand pose based on the locations of the hand features; 
 determine an input hand shape model based on the hand pose; 
 determine a set of principal component hand shape models, the principal component hand shape models defining orthogonal dimensions of hand size variance with respect to each other such that the set of principal component hand shape models defines a hand size variation space; 
 determine a user hand shape model based on fitting the input hand shape model to the set of principal component hand shape models; and 
 render an image of the user's hand based on the user hand shape model; and 
 
 a display device configured to present the rendered image to the user.