Hand gesture input for wearable system

Abstract

Techniques are disclosed for allowing a user's hands to interact with virtual objects. An image of at least one hand may be received from an image capture devices. A plurality of keypoints associated with at least one hand may be detected. In response to determining that a hand is making or is transitioning into making a particular gesture, a subset of the plurality of keypoints may be selected. An interaction point may be registered to a particular location relative to the subset of the plurality of keypoints based on the particular gesture. A proximal point may be registered to a location along the user's body. A ray may be cast from the proximal point through the interaction point. A multi-DOF controller for interacting with the virtual object may be formed based on the ray.

What is claimed is: 
     
       1. A method of interacting with a virtual object, the method comprising:
 receiving an image of a user's hand from one or more image capture devices of a wearable system; 
 analyzing the image to detect a plurality of keypoints associated with the user's hand; 
 determining, based on analyzing the image, whether the user's hand is making or is transitioning into making a particular gesture from a plurality of gestures; 
 in response to determining that the user's hand is making or is transitioning into making the particular gesture:
 selecting a subset of the plurality of keypoints that correspond to the particular gesture; 
 determining a particular location relative to the subset of the plurality of keypoints, wherein the particular location is determined based on the subset of the plurality of keypoints and the particular gesture; 
 registering an interaction point to the particular location; 
 registering a proximal point to a location along the user's body; 
 casting a ray from the proximal point through the interaction point; and 
 forming a multi-DOF controller for interacting with the virtual object based on the ray; 
 
 determining an angle associated with at least three keypoints from the plurality of keypoints, wherein the angle is between (i) a first line connecting a tip of a thumb and a midpoint between two keypoints on the thumb and an index finger and (ii) a second line connecting a tip of the index finger and the midpoint between the two keypoints on the thumb and the index finger; 
 detecting an action event associated with the multi-DOF controller based on the angle transitioning from being outside a particular range to being within the particular range; 
 determining that the angle continues to be within the particular range while the ray is moving; and 
 based on determining that the angle continues to be within the particular range while the ray is moving, causing the virtual object to move in accordance with a movement of the ray. 
 
     
     
       2. The method of  claim 1 , wherein the plurality of gestures includes at least one of a grasping gesture, a pointing gesture, or a pinching gesture. 
     
     
       3. The method of  claim 1 , wherein the subset of the plurality of keypoints is selected from a plurality of subsets of the plurality of keypoints, wherein each of the plurality of subsets of the plurality of keypoints corresponds to a different gesture from the plurality of gestures. 
     
     
       4. The method of  claim 1 , further comprising:
 displaying a graphical representation of the multi-DOF controller. 
 
     
     
       5. The method of  claim 1 , wherein the location to which the proximal point is registered is at an estimated location of the user's shoulder, an estimated location of the user's elbow, or between the estimated location of the user's shoulder and the estimated location of the user's elbow. 
     
     
       6. The method of  claim 1 , further comprising:
 capturing, by an image capture device of the one or more image capture devices, the image of the user's hand. 
 
     
     
       7. The method of  claim 6 , wherein the image capture device is mounted to a headset of a wearable system. 
     
     
       8. A system comprising:
 one or more processors; and 
 a machine-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
 receiving an image of a user's hand from one or more image capture devices of a wearable system; 
 analyzing the image to detect a plurality of keypoints associated with the user's hand; 
 determining, based on analyzing the image, whether the user's hand is making or is transitioning into making a particular gesture from a plurality of gestures; 
 in response to determining that the user's hand is making or is transitioning into making the particular gesture:
 selecting a subset of the plurality of keypoints that correspond to the particular gesture; 
 determining a particular location relative to the subset of the plurality of keypoints, wherein the particular location is determined based on the subset of the plurality of keypoints and the particular gesture; 
 registering an interaction point to the particular location; 
 registering a proximal point to a location along the user's body; 
 casting a ray from the proximal point through the interaction point; and 
 
 forming a multi-DOF controller for interacting with a virtual object based on the ray; 
 determining an angle associated with at least three keypoints from the plurality of keypoints, wherein the angle is between (i) a first line connecting a tip of a thumb and a midpoint between two keypoints on the thumb and an index finger and (ii) a second line connecting a tip of the index finger and the midpoint between the two keypoints on the thumb and the index finger; 
 detecting an action event associated with the multi-DOF controller based on the angle transitioning from being outside a particular range to being within the particular range; 
 
 determining that the angle continues to be within the particular range while the ray is moving; and
 based on determining that the angle continues to be within the particular range while the ray is moving, causing the virtual object to move in accordance with a movement of the ray. 
 
 
     
     
       9. The system of  claim 8 , wherein the plurality of gestures includes at least one of a grasping gesture, a pointing gesture, or a pinching gesture. 
     
     
       10. The system of  claim 8 , wherein the subset of the plurality of keypoints is selected from a plurality of subsets of the plurality of keypoints, wherein each of the plurality of subsets of the plurality of keypoints corresponds to a different gesture from the plurality of gestures. 
     
     
       11. The system of  claim 8 , wherein the operations further comprise:
 displaying a graphical representation of the multi-DOF controller. 
 
     
     
       12. The system of  claim 8 , wherein the location to which the proximal point is registered is at an estimated location of the user's shoulder, an estimated location of the user's elbow, or between the estimated location of the user's shoulder and the estimated location of the user's elbow. 
     
     
       13. The system of  claim 8 , wherein the operations further comprise:
 capturing, by an image capture device of the one or more image capture devices, the image of the user's hand. 
 
     
     
       14. The system of  claim 13 , wherein the image capture device is mounted to a headset of a wearable system. 
     
     
       15. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
 receiving an image of a user's hand from one or more image capture devices of a wearable system; 
 analyzing the image to detect a plurality of keypoints associated with the user's hand; 
 determining, based on analyzing the image, whether the user's hand is making or is transitioning into making a particular gesture from a plurality of gestures; and 
 in response to determining that the user's hand is making or is transitioning into making the particular gesture:
 selecting a subset of the plurality of keypoints that correspond to the particular gesture; 
 determining a particular location relative to the subset of the plurality of keypoints, wherein the particular location is determined based on the subset of the plurality of keypoints and the particular gesture; 
 registering an interaction point to the particular location; 
 registering a proximal point to a location along the user's body; 
 casting a ray from the proximal point through the interaction point; and 
 forming a multi-DOF controller for interacting with a virtual object based on the ray; 
 
 determining an angle associated with at least three keypoints from the plurality of keypoints, wherein the angle is between (i) a first line connecting a tip of a thumb and a midpoint between two keypoints on the thumb and an index finger and (ii) a second line connecting a tip of the index finger and the midpoint between the two keypoints on the thumb and the index finger; 
 detecting an action event associated with the multi-DOF controller based on the angle transitioning from being outside a particular range to being within the particular range; 
 determining that the angle continues to be within the particular range while the ray is moving; and 
 based on determining that the angle continues to be within the particular range while the ray is moving, causing the virtual object to move in accordance with a movement of the ray. 
 
     
     
       16. The non-transitory machine-readable medium of  claim 15 , wherein the plurality of gestures includes at least one of a grasping gesture, a pointing gesture, or a pinching gesture. 
     
     
       17. The non-transitory machine-readable medium of  claim 15 , wherein the subset of the plurality of keypoints is selected from a plurality of subsets of the plurality of keypoints, wherein each of the plurality of subsets of the plurality of keypoints corresponds to a different gesture from the plurality of gestures. 
     
     
       18. The non-transitory machine-readable medium of  claim 15 , wherein the operations further comprise:
 displaying a graphical representation of the multi-DOF controller. 
 
     
     
       19. The non-transitory machine-readable medium of  claim 15 , wherein the location to which the proximal point is registered is at an estimated location of the user's shoulder, an estimated location of the user's elbow, or between the estimated location of the user's shoulder and the estimated location of the user's elbow. 
     
     
       20. The non-transitory machine-readable medium of  claim 15 , wherein the operations further comprise:
 capturing, by an image capture device of the one or more image capture devices, the image of the user's hand.