System for displaying and controlling electronic objects

Abstract

The present invention is directed toward a system and process that controls a group of networked electronic components using a multimodal integration scheme in which inputs from a speech recognition subsystem, gesture recognition subsystem employing a wireless pointing device and pointing analysis subsystem also employing the pointing device, are combined to determine what component a user wants to control and what control action is desired. In this multimodal integration scheme, the desired action concerning an electronic component is decomposed into a command and a referent pair. The referent can be identified using the pointing device to identify the component by pointing at the component or an object associated with it, by using speech recognition, or both. The command may be specified by pressing a button on the pointing device, by a gesture performed with the pointing device, by a speech recognition event, or by any combination of these inputs.

1 . A computer-readable storage medium storing a video game program to be executed by a computer of a video game device for obtaining control data from a control device including image capturing means for capturing an image of a marker object, the control data being captured image data obtained by the image capturing means or data obtained by performing a predetermined operation on the captured image data, and for displaying a game image on a display device, the game image being an image of a virtual game space in which a controlled object is moved by a game process using the control data, the video game program instructing the computer to perform:
 a reference image display step of displaying a reference image at a predetermined position on a screen of the display device;   a pointer image display step of displaying a pointer image at a position on the screen pointed at by the control device; and   an object control step of controlling the controlled object to act according to a position of the marker object in the captured image based on the control data, after the pointer image reaches inside a predetermined area including at least a portion of the reference image.   
     
     
         2 . The storage medium according to  claim 1 , wherein the video game program instructs the computer to further perform an image erasing step of erasing the pointer image when the pointer image reaches inside a predetermined area defined based on a position of the reference image. 
     
     
         3 . The storage medium according to  claim 1 , wherein:
 the video game program instructs the computer to further perform a detection determination step of determining whether or not it is possible to detect a position of the marker object in the captured image; and   the reference image display step and the pointer image display step are performed when it is determined that the marker position cannot be detected.   
     
     
         4 . The storage medium according to  claim 3 , wherein it is determined in the detection determination step that the marker position cannot be detected when a part or whole of an image of the marker object is not included in the captured image. 
     
     
         5 . The storage medium according to  claim 1 , wherein:
 the controlled object is displayed as the reference image in the reference image display step; and   in the object control step, a position of the controlled object is controlled so that the controlled object is displayed at the predetermined position when it is determined that the marker position cannot be detected.   
     
     
         6 . The storage medium according to  claim 1 , wherein in the pointer image display step, the position on the screen pointed at by the control device is calculated based on a position of the marker object in the captured image. 
     
     
         7 . The storage medium according to  claim 1 , wherein the object control step includes:
 a position determination step of determining whether or not the pointer image has reached within a predetermined area defined based on a position of the reference image by comparing a position of the pointer image on the screen with the position of the reference image on the screen; and   a control start step of starting to control the controlled object when it is determined in the position determination step that the pointer image has reached within the predetermined area defined based on the position of the reference image.   
     
     
         8 . The storage medium according to  claim 1 , wherein:
 the controlled object is displayed as the reference image in the reference image display step; and   the controlled object displayed in the reference image display step is controlled in the object control step.   
     
     
         9 . The storage medium according to  claim 1 , wherein the controlled object is controlled to be moved in a three-dimensional virtual game space in the object control step. 
     
     
         10 . The storage medium according to  claim 9 , wherein:
 the video game program instructs the computer to further perform a distance calculation step of calculating a distance between two predetermined points in an image of the marker object included in a captured image obtained by the image capturing means; and   in the object control step, a movement of the controlled object in the three-dimensional virtual game space with respect to a depth direction of the screen is controlled based on the marker position and the distance.   
     
     
         11 . A video game system including a control device including image capturing means for capturing an image of a marker object, and a video game device for displaying a game image on a display device, the game image being an image of a virtual game space in which a controlled object is moved by a game process using control data, the control data being captured image data obtained by the image capturing means or data obtained by performing a predetermined operation on the captured image data, the video game system comprising:
 reference image display control means for displaying a reference image at a predetermined position on a screen of the display device;   pointer image display means for displaying a pointer image at a position on the screen pointed at by the control device;   image erasing means for erasing the pointer image when the pointer image reaches inside a predetermined area defined based on a position of the reference image; and   object control means for controlling the controlled object to act according to a position of the marker object in the captured image based on the control data, after the pointer image reaches inside a predetermined area defined based on a position of the reference image.   
     
     
         12 . A computer-implemented method for controlling an electronic object, comprising:
 capturing feature data indicating position of a control device;   displaying an image of a representation of an object to be controlled at a predetermined position on a screen of a display device;   displaying a cursor at a position on the screen pointed at by the control device; and   controlling the controlled object to act according to the captured feature data, after the cursor is displayed near a portion of the object image.   
     
     
         13 . The method according to  claim 12 , further comprising erasing the cursor when the cursor is displayed near a portion of the object image. 
     
     
         14 . The method according to  claim 12 , wherein:
 determining whether or not it is possible to detect a position of the control device based on the captured feature data, wherein displaying of the image representation of the object and displaying of the cursor are performed when it is determined that the position of the control device cannot be determined.   
     
     
         15 . The method according to  claim 14 , wherein it is determined that the feature data necessary to determine control device position cannot be detected when a part or whole of an image of the feature data necessary to determine position of the control device is not included in the captured feature data. 
     
     
         16 . The method according to  claim 12 , wherein:
 displaying the controlled object as the representation of the object to be controlled; and   controlling a position of the controlled object so that the controlled object is displayed at the predetermined position when it is determined that the position of the control device cannot be detected.   
     
     
         17 . The method according to  claim 12 , wherein the position on the screen pointed at by the control device is calculated based on the captured feature data. 
     
     
         18 . The method according to  claim 12 , wherein the object control step includes:
 determining whether or not the cursor has reached within a predetermined area defined based on a position of the object representation image by comparing a position of the cursor on the screen with the position of the object representation image on the screen; and   starting to control the controlled object when it is determined that the cursor has reached within the predetermined area defined based on the position of the object representation image.   
     
     
         19 . The method according to  claim 12 , wherein:
 displaying the controlled object as the object representation image; and   controlling the displayed controlled object in the object representation image according to the captured feature data.   
     
     
         20 . The method according to  claim 12 , wherein the controlled object is controlled to be moved in a three-dimensional virtual space. 
     
     
         21 . The method according to  claim 20 , wherein:
 calculating a distance between two predetermined image features included in the captured feature data; and   controlling a movement of the controlled object in the three-dimensional virtual space with respect to a depth direction of the screen based on the captured feature position data and the distance.   
     
     
         22 . A video game system comprising:
 feature data capture means for capturing feature data indicating position of a control device;   reference image display control means for displaying an image representative of an object to be controlled at a predetermined position on a screen of a display device;   cursor display means for displaying a cursor at a position on the screen pointed at by the control device;   image erasing means for erasing the cursor when the cursor reaches inside a predetermined area defined based on a position of the object representation image; and   object control means for controlling the controlled object to act according to the captured feature data, after the cursor is displayed near a portion of the object image.   
     
     
         23 . A storage medium storing a virtual position determining program for instructing a computer to perform a process on a virtual space based on input information obtained from an input device, the input information including target data that changes according to a change in a position of a predetermined target in a real space, the program instructing the computer to perform:
 an actual position data calculation step of calculating actual position data representing a position of the target in the real space by using the target data;   an effective range defining step of defining an effective range as a range extending from a first boundary to a second boundary, wherein the first boundary is represented by first position data and the second boundary is represented by second position data;   an effective range position data calculation step of calculating effective range position data representing a position of a value of the actual position data in the effective range;   a position determining step of determining a position of a controlled object in the virtual space through an operation based on a value of the effective range position data; and   a display process step of displaying the virtual space on a display device.   
     
     
         24 . The storage medium storing a virtual position determining program according to  claim 23 , wherein the virtual position determining program instructs the computer to further perform an effective range updating step of, when the value of the actual position data is outside the effective range, updating the effective range by moving the first boundary and the second boundary so that the value of the actual position data is included in the effective range. 
     
     
         25 . The storage medium storing a virtual position determining program according to  claim 23 , wherein the virtual position determining program instructs the computer to further perform an effective range updating step of, when the value of the actual position data is outside the effective range, updating the effective range by moving the first boundary and the second boundary so that one of the first boundary and the second boundary that is closer to the value of the actual position data becomes equal to the value of the actual position data. 
     
     
         26 . The storage medium storing a virtual position determining program according to  claim 23 , wherein the virtual position determining program instructs the computer to further perform a second effective range updating step of, when the value of the actual position data is outside the effective range, updating the effective range by moving the first boundary and the second boundary so that one of the first boundary and the second boundary that is closer to the value of the actual position data approaches the value of the actual position data at a predetermined rate. 
     
     
         27 . The storage medium storing a virtual position determining program according, to  claim 23 , wherein the effective range position data calculation step calculates the effective range position data so that the effective range position data takes a maximum value when the value of the actual position data is equal to the first boundary and the effective range position data takes a minimum value when the value of the actual position data is equal to the second boundary. 
     
     
         28 . The storage medium storing a virtual position determining program according to  claim 23 , wherein:
 the effective range defining step includes a step of further defining a play range having a predetermined length at a front end of the effective range and a rear end thereof; and   the virtual position determining program instructs the computer to further perform an effective range updating step of, when the actual position data takes a value outside a combined range including the effective range and the play ranges, updating the effective range by moving the values of the first boundary and the second boundary so that one of the first boundary and the second boundary that is closer to the value of the actual position data becomes equal to the value of the actual position data.   
     
     
         29 . The storage medium storing a virtual position determining program according to  claim 28 , wherein the effective range position data calculation step includes a step of calculating the effective range position data so that the effective range position data takes a maximum value when the value of the actual position data is equal to the first position data or exceeds the effective range in a direction of the first position data, and the effective range position data takes a minimum value when the value of the actual position data is equal to the second position data or exceeds the effective range in a direction of the second position data. 
     
     
         30 . The storage medium storing a virtual position determining program according to  claim 23 , wherein:
 the effective range defining step defines the second position data on a first side of the first position data, third position data on a second side of the first position data, and fourth position data on the first side of the second position data; and   the effective range defining step defines an upper limit boundary as position data variable from the first position data to the third position data and a lower limit boundary as position data variable from the second position data to the fourth position data, and defining the effective range as a range between the variable upper limit boundary and the variable lower limit boundary; and   the virtual position determining program instructs the computer to further perform:
 a step of, when the value of the actual position data is within the variable range of the upper limit boundary/lower limit boundary, updating the upper limit boundary/lower limit boundary so that the upper limit boundary/lower limit boundary approaches the value of the actual position data at a predetermined rate, and updating the lower limit boundary/upper limit boundary so that the lower limit boundary/upper limit boundary approaches the second position data/first position data at a predetermined rate; 
 a step of, when the value of the actual position data is within a range between the first position data and the second position data, updating the upper limit boundary/lower limit boundary so that the upper limit boundary/lower limit boundary approaches the first position data/second position data at a predetermined rate; 
 a step of, when the value of the actual position data is further on the second side of the third position data, moving the first, position data, the second position data, the third position data, the fourth position data and the lower limit boundary so that a value of the third position data approaches the value of the actual position data at a predetermined rate and updating the upper limit boundary to the new second position data; and 
 a step of, when the value of the actual position data is further on the first side of the fourth position data, moving the first position data, the second position data, the third position data, the fourth position data and the upper limit boundary so that a value of the fourth position data approaches the value of the actual position data at a predetermined rate and updating the lower limit boundary to the new fourth position data. 
   
     
     
         31 . The storage medium storing a virtual position determining program according to  claim 30 , wherein the effective range position data calculation step includes a step of calculating the effective range position data so that the effective range position data takes a maximum value when the value of the actual position data is equal to the first position data or exceeds the effective range in a direction of the first position data, and the effective range position data takes a minimum value when the value of the actual position data is equal to the second position data or exceeds the effective range in a direction of the second position data. 
     
     
         32 . A storage medium storing a virtual position determining program for, based on data from an input device including position information of a target in a real space, calculating a position of the target obtained when representing the real space in a predetermined coordinate system, and calculating a virtual position of a virtual object in a virtual space based on the calculated position, the virtual position determining program instructs a computer to function as:
 effective range defining means for defining an effective range, which is defined by at least a first position and a second position in a predetermined coordinate system of the real space;   normalization means for, when the position of the target is included in the effective range, normalizing the effective range and calculating a value corresponding to the position of the target obtained when the normalization is performed; and   calculation means for calculating the virtual position based on the value calculated by the normalization means.   
     
     
         33 . The storage medium storing a virtual position determining program according to  claim 32 , wherein the virtual position determining program instructs the computer to further function as effective range moving means for, when the position of the target is not included in the effective range, updating the first position and the second position to move the effective range to a position that includes the position of the target. 
     
     
         34 . A computer-implemented method for displaying a controlled object, comprising:
 capturing feature data indicating position of a control device in real space;   calculating actual position data representing a position of the control device in real space by using the captured feature data;   defining an effective range as a range extending from a first boundary to a second boundary, wherein the first boundary is represented by first position data and the second boundary is represented by second position data;   calculating effective range position data representing a position of a value of the actual position data in the effective range;   determining a position of a controlled object in virtual space through an operation based on a value of the effective range position data; and   displaying the virtual space on a display device.   
     
     
         35 . A computer-implemented system for calculating a position of an object in virtual space, comprising:
 effective range defining means for defining an effective range, which is defined by at least a first position and a second position in a predetermined coordinate system of a real space;   normalization means for, when the position of a control device is included in the effective range, normalizing the effective range and calculating a value corresponding to the position of the control device obtained when the normalization is performed; and   calculation means for calculating a virtual position based on the value calculated by the normalization means.