Viewpoint position detection apparatus and method, and stereoscopic image display system

Abstract

This invention has as its object to provide a viewpoint detection apparatus and method, which can assure high-speed processing, high precision, and high tracking performance with a simple arrangement while suppressing adverse influences on the human body, and a stereoscopic image display apparatus using the same. The viewpoint position detection apparatus of this invention has an image sensing unit ( 1 ) and a viewpoint detection unit ( 2 ). The image sensing unit has a visible image sensing section ( 11 ) and infrared image sensing section ( 12 ). A pupil position detection processing section ( 24 ) detects the pupil position from an infrared image, and a template generation section ( 23 ) generates templates for a visible image using the pupil position obtained from the infrared image. A pattern matching discrimination section ( 22 ) executes pattern matching of a visible image. The infrared ray irradiation time upon capturing an infrared image can be minimized, and the load on processes can be reduced since pupil position information detected from the infrared image is used upon generating templates. An image display unit ( 3 ) as a stereoscopic image display apparatus is connected to the viewpoint position detection apparatus, and viewpoint position information is supplied to the image display unit, thus constructing a stereoscopic image display system having a broad stereovision range for the observer.

What is claimed is:  
     
         1 . A viewpoint position detection apparatus for detecting a viewpoint position of a person to be measured, and outputting viewpoint position information, comprising: 
 infrared image capturing means for capturing an infrared image of the person to be measured;    visible image capturing means for capturing a visible image of the person to be measured;    detection means for detecting a pupil position of the person to be measured from the infrared image captured by said infrared image capturing means;    template generation means for generating a template for pattern matching using the pupil position from the visible image captured by said visible image capturing means; and    matching means for detecting a viewpoint position of the person to be measured by pattern matching with the visible image captured by said visible image capturing means using the template generated by said template generation means, and outputting a result as the viewpoint position information.    
     
     
         2 . A viewpoint position detection apparatus for detecting a viewpoint position of a person to be measured, and outputting viewpoint position information, comprising: 
 infrared image capturing means for capturing an infrared image of the person to be measured;    visible image capturing means for capturing a visible image of the person to be measured;    detection means for detecting a pupil position of the person to be measured from the infrared image captured by said infrared image capturing means;    template generation means for generating a template for pattern matching using the pupil position from the visible image captured by said visible image capturing means;    matching means for detecting a viewpoint position of the person to be measured by pattern matching with the visible image captured by said visible image capturing means using the template generated by said template generation means, and outputting a detection result as the viewpoint position information; and    control means for controlling to generate the template again using said detection means and said template generation means when a predetermined condition is satisfied.    
     
     
         3 . The apparatus according to  claim 2 , wherein said control means evaluates the detection result of said matching means, and when said control means determines that detection has failed a predetermined number of times, said control means controls to generate the template again using said detection means and said template generation means.  
     
     
         4 . The apparatus according to  claim 2 , wherein said control means controls to generate the template again using said detection means or said template generation means at predetermined time intervals.  
     
     
         5 . The apparatus according to  claim 2 , wherein said control means evaluates the detection result-of said matching means, and when said control means determines that detection has failed a predetermined number of times and every time a predetermined period of time has elapsed, said control means controls to generate the template again using said detection means and said template generation means.  
     
     
         6 . The apparatus according to  claim 2 , further comprising reception means for receiving an instruction from the person to be measured, and wherein said control means controls to generate the template again using said detection means and said template generation means upon receiving a predetermined instruction via said reception means.  
     
     
         7 . The apparatus according to  claim 2 , wherein said template generation means generates a plurality of templates having different sizes, and said matching means executes pattern matching a plurality of number of times using the plurality of templates in turn.  
     
     
         8 . The apparatus according to  claim 2 , further comprising second template generation means for, when pupil position detection by said detection means has failed a predetermined number of times, generating a template for pattern matching using only the visible image or information obtained from the infrared image, and supplying the generated template to said matching means as the template generated by said template generation means.  
     
     
         9 . The apparatus according to  claim 8 , further comprising reception means for receiving an instruction from the person to be measured, and wherein said control means controls to execute pattern matching using said second template generation means upon receiving a predetermined instruction via said reception means.  
     
     
         10 . A viewpoint position detection method for detecting a viewpoint position of a person to be measured, and outputting viewpoint position information, comprising: 
 the infrared image capturing step of capturing an infrared image of the person to be measured;    the visible image capturing step of capturing a visible image of the person to be measured;    the detection step of detecting a pupil position of the person to be measured from the infrared image captured in the infrared image capturing step;    the template generation step of generating a template for pattern matching using the pupil position from the visible image captured in the visible image capturing step; and    the matching step of detecting a viewpoint position of the person to be measured by pattern matching with the visible image captured in the visible image capturing step using the template generated in the template generation step, and outputting a result as the viewpoint position information.    
     
     
         11 . A viewpoint position detection method for detecting a viewpoint position of a person to be measured, and outputting viewpoint position information, comprising: 
 the infrared image capturing step of capturing an infrared image of the person to be measured;    the visible image capturing step of capturing a visible image of the person to be measured;    the detection step of detecting a pupil position of the person to be measured from the infrared image captured in the infrared image capturing step;    the template generation step of generating a template for pattern matching using the pupil position from the visible image captured in the visible image capturing step;    the matching step of detecting a viewpoint position of the person to be measured by pattern matching with the visible image captured in the visible image capturing step using the template generated in the template generation step, and outputting a detection result as the viewpoint position information; and    the control step of controlling to generate the template again using the detection step and the template generation step when a predetermined condition is satisfied, and repeating the visible image capturing step and the matching step in other cases.    
     
     
         12 . The method according to  claim 11 , further comprising the evaluation step of evaluating the detection result in the matching step, and wherein when it is determined that detection has failed a predetermined number of times, the template is generated again using the detection step and the template generation step.  
     
     
         13 . The method according to  claim 11 , further comprising the time detection step of detecting a predetermined time, and wherein every time the predetermined time is detected in the time detection step, the template is generated again using the detection step and the template generation step.  
     
     
         14 . The method according to  claim 11 , further comprising the evaluation step of evaluating the detection result in the matching step, and the time detection step of detecting a predetermined time, and wherein when it is determined that detection has failed a predetermined number of times, or every time the predetermined time is detected in the time detection step, the template is generated again using the detection step and the template generation step.  
     
     
         15 . The method according to  claim 11 , further comprising the reception step of receiving an instruction from the person to be measured, and wherein the template is generated again using the detection step and the template generation step upon receiving a predetermined instruction in the reception step.  
     
     
         16 . The method according to  claim 11 , wherein the template generation step includes the step of generating a plurality of templates having different sizes, and the matching step includes the step of executing pattern matching a plurality of number of times using the plurality of templates in turn.  
     
     
         17 . The method according to  claim 11 , further comprising the second template generation step of generating a template for pattern matching using only the visible image or information obtained from the infrared image when pupil position detection in the detection step has failed a predetermined number of times, and supplying the generated template to the matching step as the template generated in the template generation step.  
     
     
         18 . The method according to  claim 17 , further comprising the reception step of receiving an instruction from the person to be measured, and wherein pattern matching is executed using the second template generation step upon receiving a predetermined instruction in the reception step.  
     
     
         19 . A computer readable storage medium storing a method recited in  claim 10  as a program that can be executed by a computer.  
     
     
         20 . A computer readable storage medium storing a method recited in  claim 11  as a program that can be executed by a computer.  
     
     
         21 . A viewpoint detection apparatus comprising: 
 image sensing means for capturing an image of an observer as video information;    video processing means having a function of detecting a face region from the video information of the observer captured by said image sensing means, and detecting two eyes of the observer from the face region, and a function of tracking the detected two eyes; and    camera control means for enlarging or reducing the face region detected by said video processing means.    
     
     
         22 . A stereoscopic image display apparatus for allowing an observer to stereoscopically observe disparity images using a viewpoint detection apparatus for detecting a viewpoint of the observer and a display device for displaying disparity images corresponding to right and left eyes of the observer while controlling the disparity image to track viewpoint information, 
 wherein said viewpoint detection apparatus is a viewpoint detection apparatus recited in any one of  claims 1  to  9  and  claim 21 .    
     
     
         23 . The apparatus according to  claim 22 , wherein said image sensing means has a video camera, and said camera control means has a mechanism for panning/tilting said video camera.  
     
     
         24 . The apparatus according to  claim 22 , further comprising signal switching means for externally outputting a video signal from said image sensing means and a zoom/pan/tilt control signal from said camera control means.  
     
     
         25 . The apparatus according to  claim 22 , wherein said video processing means identifies predetermined color information from the captured video information of the observer.  
     
     
         26 . The apparatus according to  claim 25 , wherein the predetermined color information is a face tone of the observer or a standard flesh tone.  
     
     
         27 . The apparatus according to  claim 25 , wherein when said video processing means identifies the predetermined color information, and when a region corresponding to the color is not detected from the captured video information, a focal length of said video camera is controlled to a short focal length side, and when the region is detected, the focal length of said video camera is controlled to a predetermined focal length.  
     
     
         28 . The apparatus according to  claim 25 , further comprising alarm means for generating an alarm to the observer when said video processing means identifies the predetermined color information, and when a region corresponding to the color is not detected from the captured video information.  
     
     
         29 . The apparatus according to  claim 28 , wherein said video processing means identifies a predetermined pattern region from the captured video information of the observer.  
     
     
         30 . The apparatus according to  claim 22 , wherein the predetermined pattern is an eye of the observer, a standard eye, a vicinity of an eye, or a partial image that forms an eye such as an iris or the like.  
     
     
         31 . The apparatus according to  claim 22 , further comprising video processing means for generating the predetermined color or pattern from face image information of the observer, and video recording means for recording the generated information.  
     
     
         32 . The apparatus according to  claim 22 , further comprising switching means for displaying face image information of the observer captured by said video camera on a display unit.  
     
     
         33 . The apparatus according to  claim 32 , further comprising operation means for allowing the observer to manually set the face image of the observer displayed on the display unit at a predetermined position and a size on a display screen.  
     
     
         34 . The apparatus according to  claim 22 , wherein said video processing means tracks a specific pattern by pattern recognition.  
     
     
         35 . The apparatus according to  claim 34 , wherein the specific pattern is an eye of the observer, a standard eye, a vicinity of an eye, or a partial image that forms an eye such as an iris or the like, and said apparatus further comprises alarm means for generating an alarm when a spacing between two eyes (captured from the video camera) is other than a prescribed value upon tracking the two eyes.  
     
     
         36 . A stereoscopic image display apparatus having a display device which includes an optical modulator having a discrete pixel structure, a mask pattern formed on a display surface of said optical modulator by aligning a plurality of transmitting and intercepting portions at a predetermined pitch in horizontal and vertical directions, light source means for irradiating said optical modulator with light, a display which has a discrete pixel structure and displays a synthesized disparity image using scanning lines, and which irradiates disparity images displayed on said display with a light beam patterned by said mask pattern, guides light beams based on the disparity images to right and left eyes of an observer, and allows the observer to stereoscopically observe image information displayed on said display, and a viewpoint detection apparatus for detecting viewpoint information of the observer, 
 wherein the synthesized disparity image is formed by two original disparity images corresponding to the right and left eyes, and a pattern shape of said mask pattern and original disparity images that form the synthesized disparity image are switched and displayed on the basis of the viewpoint information from said viewpoint detection apparatus.    
     
     
         37 . The apparatus according to  claim 36 , wherein the two original disparity images that form the synthesized disparity image are images observed from a viewpoint corresponding to a distance between eyes.  
     
     
         38 . The apparatus according to  claim 36 , wherein a horizontal element of each transmitting portion of said mask pattern of said optical modulator is composed of a plurality of pixels, and a stripe irradiated region to be projected at an observation position is controlled upon being segmented into a plurality of regions.  
     
     
         39 . A stereoscopic image display method comprising: the step of capturing an image of an observer who is observing a stereoscopic image based on disparity images displayed on a display as video information; the step of detecting a face region of the observer on the basis of the video information of the observer; the step of detecting eyeballs of the observer from the face region of the observer; the step of tracking the eyeballs of the observer; the step of detecting viewpoint information of the observer from the detected eyeballs of the observer; and the step of controlling to track the disparity images to be displayed on the display on the basis of the viewpoint information of the observer.  
     
     
         40 . The method according to  claim 39 , further comprising the step of identifying predetermined color information from the captured video information of the observer.  
     
     
         41 . The method according to  claim 39 , further comprising the step of changing a capturing method of the video information of the observer when predetermined color information is not present in the captured video information of the observer.  
     
     
         42 . The method according to claim  40 , further comprising the step of generating an alarm signal when the predetermined color information is not present in the captured video information of the observer.