Image-capturing system for combining subject and three-dimensional virtual space in real time

Abstract

[Problem] To generate a highly realistic composite image. [Solution] This image-capturing system is provided with a camera ( 10 ) for capturing an image of a subject, a tracker ( 20 ) for detecting the position and orientation of the camera, a space image storage unit ( 30 ) in which an image of a three-dimensional virtual space is stored, and an image-forming unit ( 40 ) for generating a composite image in which an image of the subject captured using the camera and an image of the three-dimensional virtual space are combined. The image-forming unit ( 40 ) projects the three-dimensional virtual space specified by a world coordinate system (X, Y, Z) onto screen coordinates (U, V), in which the camera coordinate system (U, V, N) of the camera is taken as a reference, and combines the images of the three-dimensional virtual space and the subject on a screen specified by the screen coordinates (U, V). The camera coordinate system (U, V, N) is then set on the basis of the position and orientation of the camera detected by the tracker.

1 . An image-capturing system comprising:
 a camera for capturing an image of a subject;   a tracker for detecting a position and orientation of the camera;   a space image storage unit in which an image of a three-dimensional virtual space is stored; and   a rendering unit for generating a composite image in which the image of the subject captured using the camera and the image of the three-dimensional virtual space stored in the space image storage unit are combined,   wherein the rendering unit   projects the three-dimensional virtual space specified by a world coordinate system (X, Y, Z) onto screen coordinates (U, V), in which a camera coordinate system (U, V, N) of the camera is taken as a reference, and   combines the images of the three-dimensional virtual space and the subject on a screen specified by the screen coordinates (U, V), and   the camera coordinate system (U, V, N) is set on the basis of the position and orientation of the camera detected using the tracker.   
     
     
         2 . The image-capturing system according to  claim 1 , further comprising
 a monitor installed at a position visible from the subject being a human whose image is being captured by the camera,   wherein the rendering unit outputs the composite image to the monitor.   
     
     
         3 . The image-capturing system according to  claim 2 , further comprising:
 a motion sensor for detecting motion of the subject; and   a content storage unit in which a content including an image is stored in association with information relating to the motion of the subject,   wherein the rendering unit combines the content that is associated with the motion of the subject detected using the motion sensor with the image of the three-dimensional virtual space and the image of the subject on the screen, and outputs a composite image of the content and the images to the monitor.   
     
     
         4 . The image-capturing system, according to  claim 3 , changes the content depending on the calculation result, which is obtained from the rendering unit that obtains both or any one of a distance from the camera to the subject and an angle of the subject. 
     
     
         5 . The image-capturing system according to  claim 1 , comprises of a mirror type display installed at a position visible from the subject being a human whose image is being captured by the camera,
 wherein the mirror type display includes:   a display capable of displaying an image; and   a semitransparent mirror arranged at a display surface side of the display for transmitting light of the image displayed by the display and for reflecting light entering from an opposite side of the display.   
     
     
         6 . The image-capturing system, according to  claim 5 , outputs the image of the three-dimensional virtual space stored in the space image storage unit to the display, and comprises of
 a second rendering unit that projects the three-dimensional virtual space specified by the world coordinate system (X, Y, Z) onto the screen coordinates (U, V), which uses the reference from the camera coordinate system (U, V, N) that is set based on the position and orientation of the camera detected using the tracker.   
     
     
         7 . The image-capturing system according to  claim 5 ,
 wherein the second rendering unit reads the content that is associated with the motion of the subject detected using the motion sensor from the content storage unit, and outputs the content to the display.