Head-mounted display with facial expression detecting capability

Abstract

Embodiments relate to detecting a user's facial expressions in real-time using a head-mounted display unit that includes a 2D camera (e.g., infrared camera) that capture a user's eye region, and a depth camera or another 2D camera that captures the user's lower facial features including lips, chin and cheek. The images captured by the first and second camera are processed to extract parameters associated with facial expressions. The parameters can be sent or processed so that the user's digital representation including the facial expression can be obtained

1 . A head-mounted display unit, comprising:
 a first capturing device configured to capture an upper portion of a user's face including an eye region;   a second capturing device configured to capture a lower portion of the user's face;   a display device configured to display images to the user;   a body configured to mount the first and second capturing devices and the display device; and   a member under the body and configured to mount the second capturing device at a location below the first capturing device.   
     
     
         2 . The head-mounted display unit of  claim 1 , wherein the member is slidable relative to the body. 
     
     
         3 . The head-mounted display unit of  claim 1 , wherein the body is connected to a strap configured to secure the head-mounted display to the user's head. 
     
     
         4 . The head-mounted display unit of  claim 1 , wherein the display device includes a first display and a second display, wherein the first display is configured to display left side images to a left eye of the user and the second display is configured to display right side images to a right eye of the user. 
     
     
         5 . The head-mounted display unit of  claim 1 , wherein the first capturing device comprises a pair of cameras, each of the cameras installed at an opposite side of the body. 
     
     
         6 . The head-mounted display unit of  claim 1 , wherein the first capturing device comprises a camera installed in a middle of the body. 
     
     
         7 . The head-mounted display unit of  claim 1 , wherein the second capturing device is mounted directly onto the body. 
     
     
         8 . The head-mounted display unit of  claim 1 , wherein the body has a bulging upper portion to enclose the eye region. 
     
     
         9 . The head-mounted display unit of  claim 1 , wherein the display device comprises a pair of distinct display units and the first capturing device comprises two cameras between the pair of display units. 
     
     
         10 . A method for detecting facial expressions, the method comprising:
 capturing, by a first image capturing device on a main body of a head-mounted display, first images of an upper portion of a user's face including the user's eye region, the main body comprising one or more displays;   capturing, by a second image capturing device on an extension member extending downwards from the main body, second images of the user including a lower portion of the user's face; and   processing the first images and the second images to extract facial expression parameters representing a facial expression of the user.   
     
     
         11 . The method of  claim 10 , wherein the extension member is movable relative to the main body. 
     
     
         12 . The method of  claim 10 , wherein the main body is connected to a strap configured to secure the head-mounted display to the user's head. 
     
     
         13 . The method of  claim 10 , wherein the first image capturing device comprises a pair of infrared cameras, and the second image capturing device comprises one of a depth camera, a color camera, an infrared camera or two stereoscopic cameras. 
     
     
         14 . The method of  claim 10 , processing the first images and the second images comprises:
 detecting landmark locations associated with the user's eye and the user's eyebrow from the first images; and   detecting landmark locations associated with a lower portion of the user's face from the second images.   
     
     
         15 . The method of  claim 10 , further comprising performing calibration by:
 generating a personalized neutral face mesh based on the calibration images, and   building a personalized tracking model by applying a deformation transfer technique to the personalized neutral face mesh, and   wherein the processing of the first images and the second images comprises fitting at least a blendshape model to landmark locations in the first images and the second images based on the personalized tracking model to obtain the facial expression parameters.   
     
     
         16 . The method of  claim 10 , wherein the processing of the first images and the second images comprises performing rigid stabilization to determine a relative pose of the user's face relative to the first image capturing device and the second image capturing device. 
     
     
         17 . A virtual or augmented reality system comprising:
 a head-mounted display unit, comprising:
 a first capturing device configured to capture an upper portion of a user's face including an eye region, 
 a second capturing device configured to capture a lower portion of the user's face, 
 a display device configured to display images to the user, 
 a body configured to mount the first and second capturing devices and the display device, and 
 a member under the body and configured to mount the second capturing device at a location below the first capturing device; and 
   a computing device communicatively coupled to the head-mounted display unit and configured to:
 receive the first and second images from the head-mounted display unit, and 
 process the first images and the second images to extract facial expression parameters representing a facial expression of the user. 
   
     
     
         19 . The virtual or augmented reality system of  claim 17 , wherein the member is slidable relative to the body. 
     
     
         20 . The virtual or augmented reality system of  claim 17 , wherein the body is connected to a strap configured to secure the head-mounted display to the user's head.