Head mounted image-sensing display device and composite image generating apparatus

Abstract

The invention provides a head mounted image-sensing display device including a pair of image-sensing units ( 18 R, 18 L) that stereoscopically capture a physical space and output a pair of stereoscopic images, and display units ( 13 R, 13 L) for displaying images for the right eye and images for the left eye. Image sensing parameters that are previously measured for the image-sensing units are stored in image sensing parameter storage units ( 26 R, 26 L). As a result, a difference in parallax between captured images of a physical space and images of a virtual object is reduced.

1 .- 12 . (canceled) 
     
     
         13 . A head mounted display device comprising:
 a pair of image-sensing units configured to stereoscopically capture a physical space;   an obtaining unit configured to obtain a pair of stereoscopic captured images from the pair of image-sensing units, and an image-sensing parameter which is previously measured;   a generation unit configured to generate a virtual image based on the image-sensing parameter and three-dimensional model information; and   a display unit configured to display the virtual image with the pair of stereoscopic captured image,   wherein the image-sensing parameter represents the positional relationship between one of the pair of image-sensing units and the other of the pair of image-sensing units.   
     
     
         14 . The head mounted display device according to  claim 13 , wherein one of the pair of image-sensing units is a left eye display unit, and the other of the pair of image-sensing units is a right eye display unit. 
     
     
         15 . The head mounted display device according to  claim 14 , further comprising an adjustment mechanism configured to change a base line length of the pair of image-sensing units. 
     
     
         16 . An apparatus comprising:
 an obtaining unit configured to obtain a pair of stereoscopic captured images from a pair of image-sensing units, which are configured to stereoscopically capture a physical space, and an image-sensing parameter which is previously measured;   a generation unit configured to generate a virtual image based on the image-sensing parameter and three-dimensional model information; and   a display unit configured to display the virtual image with the pair of stereoscopic captured image,   wherein the image-sensing parameter represents a positional relationship between one of the pair of image-sensing units and the other of the pair of image-sensing units.   
     
     
         17 . A head mounted display device comprising:
 an obtaining unit configured to obtain a captured image from an image-sensing unit capturing a physical space, and an image-sensing parameter which is previously measured;   a generation unit configured to generate a virtual image based on the image-sensing parameter and three-dimensional model information; and   a display unit configured to display the virtual image with the captured image,   wherein the image-sensing parameter includes a parameter obtained from the image-sensing unit by capturing a calibration pattern and a parameter for preventing the virtual image from deviating caused by misalignment of the image-sensing unit.   
     
     
         18 . An apparatus comprising:
 an obtaining unit configured to obtain a captured image from an image-sensing unit capturing a physical space, and an image-sensing parameter which is previously measured;   a generation unit configured to generate a virtual image based on the image-sensing parameter and three-dimensional model information; and   a display unit configured to display the virtual image with the captured image,   wherein the image-sensing parameter includes a parameter obtained from the image-sensing unit by capturing a calibration pattern and a parameter for preventing the virtual image from deviating caused by misalignment of the image-sensing unit.   
     
     
         19 . A head mounted display device comprising:
 an obtaining unit configured to obtain a unique parameter of an optical unit which is previously measured;   a generation unit configured to generate a virtual image based on the unique parameter and three-dimensional model information; and   a display unit configured to display the virtual image with a physical space by using the optical unit.   
     
     
         20 . An apparatus comprising:
 an obtaining unit configured to obtain a unique parameter of an optical unit which is previously measured;   a generation unit configured to generate a virtual image based on the unique parameter and three-dimensional model information; and   a display unit configured to display the virtual image with a physical space by using the optical unit.   
     
     
         21 . A method, comprising the steps of:
 obtaining a pair of stereoscopic captured images from a pair of image-sensing units configured to stereoscopically capture a physical space, and an image-sensing parameter which is previously measured;   generating a virtual image based on the image-sensing parameter and three-dimensional model information; and   displaying the virtual image with the pair of stereoscopic captured image on a display unit,   wherein the image-sensing parameter represents a positional relationship between one of the pair of image-sensing units and the other of the pair of image-sensing units.   
     
     
         22 . A method, comprising the steps of:
 obtaining a captured image from an image-sensing unit capturing a physical space, and an image-sensing parameter which is previously measured;   generating a virtual image based on the image-sensing parameter and three-dimensional model information; and   displaying the virtual image with the captured image on a display unit,   wherein the image-sensing parameter includes a parameter obtained from the image-sensing unit by capturing a calibration pattern and a parameter for preventing the virtual image from deviating caused by misalignment of the image-sensing unit.   
     
     
         23 . A method, comprising the steps of:
 obtaining a unique parameter of an optical unit which is previously measured;   generating a virtual image based on the unique parameter and three-dimensional model information; and   displaying the virtual image with a physical space by using the optical unit on a display unit.   
     
     
         24 . A system comprising:
 an image processing apparatus; and   a head mounted display device,   wherein   the image processing apparatus comprises:
 an obtaining unit configured to obtain a pair of stereoscopic captured images from a pair of image-sensing units, which are configured to stereoscopically capture a physical space, and an image-sensing parameter which is previously measured; and 
 a generation unit configured to generate a virtual image based on the image-sensing parameter and three-dimensional model information, wherein the image-sensing parameter represents a positional relationship between one of the pair of image-sensing units and the other of the pair of image-sensing units; and 
   the head mounted display device comprises a display unit configured to display the virtual image with the pair of stereoscopic captured image.   
     
     
         25 . A system comprising:
 an image processing apparatus; and   a head mounted display device,   wherein   the image processing apparatus comprises:
 an obtaining unit configured to obtain a captured image from an image-sensing unit capturing a physical space, and an image-sensing parameter which is previously measured; and 
 a generation unit configured to generate a virtual image based on the image-sensing parameter and three-dimensional model information, wherein the image-sensing parameter includes a parameter obtained from the image-sensing unit capturing a calibration pattern and a parameter for preventing the virtual image from deviating caused by misalignment of the image-sensing unit; and 
   the head mounted display device comprises a display unit configured to display the virtual image with the captured image.   
     
     
         26 . A system comprising:
 an image processing apparatus; and   a head mounted display device,   wherein   the image processing apparatus comprises:
 an obtaining unit configured to obtain a unique parameter of an optical unit which is previously measured; and 
 a generation unit configured to generate a virtual image based on the unique parameter and three-dimensional model information; and 
   the head mounted display device comprises a display unit configured to display the virtual image with a physical space by using the optical unit.