Wearable electronic devices and extended reality systems including neuromuscular sensors

Abstract

Computerized systems, methods, kits, and computer-readable media storing code for implementing the methods are provided for interacting with a physical object in an augmented reality (AR) environment generated by an AR system. One such system includes: a plurality of neuromuscular sensors able to sense a plurality of neuromuscular signals from a user, and at least one computer processor. The neuromuscular sensors are arranged on one or more wearable devices worn by the user to sense the neuromuscular signals. The at least one computer processor is or are programmed to: determine, based at least in part, on the neuromuscular signals sensed by the neuromuscular sensors, information about an interaction of the user with the physical object in the AR environment generated by the AR system; and instruct the AR system to provide feedback based, at least in part, on the information about the interaction of the user with the physical object.

1 . A wearable electronic device, comprising:
 a wearable structure configured to be worn on a wrist of a user, the wearable structure including an inner surface and an outer surface, wherein the inner surface contacts the user's wrist when the wearable structure is donned by the user;   a plurality of neuromuscular sensors, arranged on the inner surface of the wearable structure, configured to record, at the user's wrist, neuromuscular signals generated by the user; and   one or more processors configured to:
 detect, in real-time, one or more gestures of the user based on the neuromuscular signals generated by the user; 
 map the one or more gestures of the user to a control function; 
 execute the control function; and 
 provide visual feedback to the user on a display, wherein the visual feedback is related to execution of the control function. 
   
     
     
         2 . The wearable electronic device of  claim 1 , wherein the plurality of neuromuscular sensors includes at least two distinct pairs of neuromuscular sensors, wherein:
 a first pair of neuromuscular sensors from the at least two distinct pairs of neuromuscular sensors is configured to record neuromuscular signals generated by the user at a first wrist location when the wearable structure is donned by the user; and   a second pair of neuromuscular sensors from the at least two distinct pairs of neuromuscular sensors is configured to record neuromuscular signals generated by the user at a second wrist location when the wearable structure is donned by the user.   
     
     
         3 . The wearable electronic device of  claim 2 , wherein the at least two distinct pairs of neuromuscular sensors comprise more than two distinct pairs of neuromuscular sensors arranged circumferentially around the inner surface of the wearable structure. 
     
     
         4 . The wearable electronic device of  claim 1 , wherein the control function comprises a control function for controlling at least one of:
 a robot;   a vehicle;   scrolling through text;   an extended reality system; or   a virtual avatar.   
     
     
         5 . The wearable electronic device of  claim 1 , wherein the control function comprises at least one virtual control associated with a physical object grasped by the user. 
     
     
         6 . The wearable electronic device of  claim 1 , further comprising an inertial measurement unit mounted on or in the wearable structure, wherein the inertial measurement unit is configured to sense movement of the wearable structure. 
     
     
         7 . The wearable electronic device of  claim 6 , wherein the inertial measurement unit comprises at least one of:
 an accelerometer;   a gyroscope; or   a magnetometer.   
     
     
         8 . The wearable electronic device of  claim 1 , wherein each of the neuromuscular sensors of the plurality of neuromuscular sensors comprises an electromyography sensor. 
     
     
         9 . The wearable electronic device of  claim 1 , wherein the display comprises a display of an extended reality system. 
     
     
         10 . The wearable electronic device of  claim 9 , wherein the display comprises a head-mounted display. 
     
     
         11 . The wearable electronic device of  claim 1 , wherein the one or more processors is further configured to provide at least one of audio feedback, electrical stimulation feedback, or haptic feedback related to execution of the control function. 
     
     
         12 . An extended reality system, comprising:
 a wearable electronic device configured to be worn on a wrist of a user, the wearable electronic device comprising a plurality of neuromuscular sensors arranged on an inner surface of the wearable electronic device and configured to sense, at the user's wrist, neuromuscular signals generated by the user;   one or more processors configured to detect one or more gestures of the user based on data from the plurality of neuromuscular sensors and to execute a control action based on the one or more gestures; and   a head-mounted display device configured to be worn on a head of the user and configured to present to the user a visual indication of the control action.   
     
     
         13 . The extended reality system of  claim 12 , further comprising a camera mounted on the head-mounted display device, wherein the camera is positioned and oriented to capture at least one image of at least a portion of the user's hand. 
     
     
         14 . The extended reality system of  claim 13 , wherein the one or more processors is further configured to identify, at least in part based on the at least one image, a physical object grasped by the user's hand. 
     
     
         15 . The extended reality system of  claim 13 , wherein the one or more processors is further configured to generate a representation of the user's hand on the head-mounted display device based on the at least one image and the sensed neuromuscular signals. 
     
     
         16 . The extended reality system of  claim 12 , wherein the one or more gestures comprises at least one of:
 a static gesture;   a dynamic gesture;   a covert gesture;   a muscle activation state; or   a sub-muscular activation state.   
     
     
         17 . The extended reality system of  claim 12 , wherein the control action is configured to be used to control a function of the head-mounted display device. 
     
     
         18 . The extended reality system of  claim 12 , wherein the one or more processors is further configured to provide feedback to the user based on the detected one or more gestures. 
     
     
         19 . The extended reality system of  claim 18 , wherein the feedback comprises at least one of:
 visual feedback;   audio feedback;   electrical stimulation feedback; or   haptic feedback.   
     
     
         20 . A wearable electronic system, comprising:
 a wearable device;   a first pair of neuromuscular sensors arranged on an inner surface of the wearable device, the first pair of neuromuscular sensors being configured to record neuromuscular signals generated by a user at a first wrist location when the wearable device is donned by the user;   a second pair of neuromuscular sensors arranged on the inner surface of the wearable device, the second pair of neuromuscular sensors being configured to record neuromuscular signals generated by the user at a second wrist location when the wearable device is donned by the user; and   one or more processors configured to:
 detect one or more gestures based on data from the first pair of neuromuscular sensor and the second pair of neuromuscular sensors; 
 execute a control action based on the one or more gestures; and 
 provide feedback to the user that is related to execution of the control action.