Systems, methods, apparatuses and devices for detecting facial expression and for tracking movement and location in at least one of a virtual and augmented reality system

Abstract

Systems, methods, apparatuses and devices for detecting facial expressions according to EMG signals for a virtual and/or augmented reality (VR/AR) environment, in combination with a system for simultaneous location and mapping (SLAM), are presented herein.

What is claimed is: 
     
         1 . A virtual environment system for a plurality of users, comprising:
 a first virtual environment head mounted display for being worn on a head of a first user;   a first apparatus comprising a plurality of EMG (electromyography) electrodes configured for contact with a face of said first user; and   a first computational device configured with instructions operating thereon to cause the computational device to:
 process a plurality of EMG signals received from said EMG electrodes to form processed EMG signals; and 
 classify a facial expression according to said processed EMG signals using a classifier; 
   a second virtual environment head mounted display for being worn on a head of a second user;   a computer network for communication between said first and second computational devices;   wherein a classified facial expression of said first user is transmitted from said first computational device to said second computational device; wherein said second computational device causes said second virtual environment head mounted display to display facial expression information according to said classified facial expression of said first user.   
     
     
         2 . The system of  claim 1 , further comprising: a first virtual environment head mounted display for being worn on a head of said first user; a second apparatus comprising a plurality of EMG (electromyography) electrodes configured for contact with a face of said second user; wherein said second computational device is configured with instructions operating thereon to cause the computational device to:
 process a plurality of EMG signals received from said EMG electrodes to form processed EMG signals; and   classify a facial expression according to said processed EMG using a classifier;   wherein a classified facial expression of said second user is transmitted from said second computational device to said first computational device; wherein said first computational device causes said first virtual environment head mounted display to display facial expression information according to said classified facial expression of said second user.   
     
     
         3 . The system of  claim 2 , further comprising multiple apparatuses comprising EMG electrodes, multiple computational devices and multiple virtual environment head mounted displays for a multi-user virtual environment; wherein said multiple computational devices are in communication with each other through said computer network. 
     
     
         4 . The system of  claim 1 , wherein classifying comprises determining whether the facial expression corresponds to a neutral expression or a non-neutral expression. 
     
     
         5 . The system of  claim 4 , wherein upon determining a non-neutral expression, classifying includes determining said non-neutral expression. 
     
     
         6 . The system of  claim 1 , wherein said predefined window occurs within 100 ms. 
     
     
         7 . The system of  claim 1 , wherein said classifier classifies said processed EMG signals of the user using at least one of (1) a discriminant analysis classifier; (2) a Riemannian geometry classifier; (3) Naïve Bayes classifier, (4) a k-nearest neighbor classifier, (5) a RBF (radial basis function) classifier, (6) a Bagging classifier, (7) a SVM (support vector machine) classifier, (8) a node classifier (NC), (9) NCS (neural classifier system), (10) SCRLDA (Shrunken Centroid Regularized Linear Discriminate and Analysis), or (11) a Random Forest classifier. 
     
     
         8 . The system of  claim 7 , wherein said discriminant analysis classifier is one of (1) LDA (linear discriminant analysis), (2) QDA (quadratic discriminant analysis), or (3) sQDA. 
     
     
         9 . The system of  claim 7 , wherein said classifier is one of (1) Riemannian geometry, (2) QDA and (3) sQDA. 
     
     
         10 . The system of  claim 1 , wherein: said processing comprises determining a roughness of said EMG signals according to a predefined window, and said classifier classifies the facial expression according to said roughness. 
     
     
         11 . The system of  claim 1 , wherein the instructions are additionally configured to cause the computational device to receive data associated with at least one facial expression of the user before classifying the facial expression as a neutral expression or a non-neutral expression. 
     
     
         12 . The system of  claim 11 , wherein said at least one facial expression is a neutral expression. 
     
     
         13 . The system of  claim 11 , wherein said at least one facial expression is a non-neutral expression. 
     
     
         14 . The system of  claim 1 , wherein said electrodes comprise unipolar electrodes. 
     
     
         15 . The system of  claim 14 , wherein processing said EMG signals comprises removing common mode interference of said unipolar electrodes. 
     
     
         16 . The system of  claim 1 , wherein said apparatus further comprises a local board in electrical communication with said EMG electrodes, the local board configured for converting said EMG signals from analog signals to digital signals, and a main board for receiving said digital signals. 
     
     
         17 . The system of  claim 1 , wherein said EMG electrodes comprise eight unipolar EMG electrodes and one reference electrode, the system further comprising:
 an electrode interface in electrical communication with said EMG electrodes and with said computational device, and configured for providing said EMG signals from said EMG electrodes to said computational device; and   a mask configured to contact an upper portion of the face of the user and including an electrode plate; wherein said EMG electrodes are configured to attach to said electrode plate of said mask, such that said EMG electrodes contact said upper portion of the face of the user.   
     
     
         18 . The system of  claim 1 , wherein the instructions are further configured to cause the computational device to determine a level of said facial expression according to a standard deviation of said roughness. 
     
     
         19 . The system of  claim 1 , wherein said processing comprises removing electrical power line interference (PLI). 
     
     
         20 . The system of  claim 19 , wherein said removing said PLI comprising filtering said EMG signals with two series of Butterworth notch filters of order 1, a first series of filter at 50 Hz and all its harmonics up to the Nyquist frequency, and a second series of filter with cutoff frequency at 60 Hz and all its harmonics up to the Nyquist frequency. 
     
     
         21 . The system of  claim 1 , wherein said determining said roughness further comprises calculating an EMG-dipole; and determining said roughness for said EMG-dipole. 
     
     
         22 . The system of  claim 21 , wherein said determining said roughness further comprises a movement of said signals according to said EMG-dipole. 
     
     
         23 . The system of  claim 1 , wherein said classifier determines said facial expression at least partially according to a plurality of features, wherein said features comprise one or more of roughness, roughness of EMG-dipole, a direction of movement of said EMG signals of said EMG-dipole and a level of facial expression. 
     
     
         24 . A virtual environment system for a plurality of users, comprising:
 a first virtual environment head mounted display for being worn on a head of a first user;   a first apparatus comprising a plurality of EMG (electromyography) electrodes configured for contact with a face of said first user; and   a first computational device configured with instructions operating thereon to cause the computational device to:
 process a plurality of EMG signals received from said EMG electrodes to form processed EMG signals; and 
 classify a facial expression according to said processed EMG signals using a classifier; 
   wherein facial expression information of said first user is displayed to said first user through said virtual environment head mounted display according to said classified facial expression of said first user.   
     
     
         25 . A virtual environment system for a user, comprising:
 a virtual environment head mounted display for being worn on a head of a user;   an apparatus comprising a plurality of EMG (electromyography) electrodes configured for contact with a face of said user; and   a computational device configured with instructions operating thereon to cause the computational device to:
 process a plurality of EMG signals received from said EMG electrodes to form processed EMG signals; and 
 classify a facial expression according to said processed EMG signals using a classifier; 
   wherein facial expression information of said user is displayed to said user through said virtual environment head mounted display according to said classified facial expression of said user.   
     
     
         26 . The system of  claim 25 , adapted for training of said user. 
     
     
         27 . The system of  claim 25 , adapted for therapy of said user. 
     
     
         28 . The system of  claim 27 , further comprising a processor for executing therapeutic instructions and a memory for storing said therapeutic instructions; wherein said processor executes said therapeutic instructions for therapy of said user. 
     
     
         29 . The system of  claim 28 , wherein said therapeutic instructions are executed for a therapeutic game. 
     
     
         30 . The system of  claim 29 , wherein said therapeutic instructions comprise instructions for a gesture analysis module. 
     
     
         31 . The system of  claim 30 , wherein said memory further comprises instructions for calibration of a movement of said user, and wherein said therapeutic instructions further comprise instructions for determining a range of motion of said movement. 
     
     
         32 . The system of  claim 28 , wherein said therapeutic instructions are executed for a mirror avatar. 
     
     
         33 . The system of  claim 32 , wherein said mirror avatar is displayed for at least one of facial emotion therapy and phantom limb therapy. 
     
     
         34 . The system of  claim 32 , wherein said mirror avatar is displayed, and communication between said user and said mirror avatar is evaluated in said virtual environment. 
     
     
         35 . The system of  claim 28 , wherein said therapeutic instructions are executed for providing a virtual object to said virtual environment head mounted display of said user and for assessing an interaction of said user with said virtual object in said virtual environment. 
     
     
         36 . The system of  claim 25 , adapted for diagnosing said user. 
     
     
         37 . The system of  claim 25 , further comprising multiple virtual environment head mounted displays for a multi-user virtual environment; a computer network; and multiple additional computational devices in communication with each other, and in communication with said computational device in communication with said apparatus comprising said EMG electrodes; wherein each additional computational device is in communication with at least one of said multiple virtual environment head mounted displays; wherein said multiple computational devices are in communication with each other through said computer network; wherein said classified facial expression information is sent from said at least one computational device in communication with said at least one apparatus to said multiple computational devices for display in said multiple virtual environment head mounted displays.