Coordinating Haptics with Visual Images in a Human-Computer Interface

Abstract

The present invention provides a method of generating haptic interface experiences that is capable of generating haptic companions to existing two-dimensional images. The method exploits the dominance of visual over haptic in humans: human tactile, proprioceptive, and kinesthetic senses do not supply as much fine detail as human vision, and consequently haptic representations do not have to be as detailed as the corresponding visual images to be believable to a human user. An existing two-dimensional image can be used as the visual component of a human-computer interface. A three-dimensional haptic model that is topologically similar to the image can be selected, where topological similarity means that the haptic model provides an acceptable approximation to important characteristics of the haptic experience desired to accompany the two-dimensional image. A relationship can then be established between the image in the visual component of the interface and the haptic model in the haptic component of the interface such that interaction with the image in the visual component is linked with interaction with the haptic model in the haptic component. As an example, a two-dimensional image of a ball can have a haptic model of a sphere associated therewith, and the center and radius established as a relationship such that visual cursor motion at the boundary of the image of the two-dimensional image of the ball is linked with haptic force feedback as the cursor encounters the surface of the haptic model.

1 . A method of generating a three-dimensional haptic component of a human-computer interface corresponding to a two-dimensional image of an object, comprising: 
 a) Selecting from a library of haptic models an initial haptic model that is topologically similar to the object;    b) Establishing correspondence between a set of correspondence points in the three-dimensional initial haptic model and a set of correspondence points in the two-dimensional image of the object;    c) Generating a final haptic model by modifying the initial haptic model so that the two-dimensional projection of the correspondence points in the final haptic model are substantially aligned with the correspondence points in the two-dimensional image of the object.    
   
   
       2 . The method of  claim 1 , wherein selecting from a library comprises using automated image recognition to select three-dimensional models having two-dimensional projections similar to the object.  
   
   
       3 . The method of  claim 1 , wherein establishing correspondence comprises presenting a representation of the initial haptic model to a user, and accepting from the user specification of points in the initial haptic model to place in correspondence with points in the display of the object.  
   
   
       4 . The method of  claim 3 , wherein accepting from the user specification of points in the initial haptic model to place in correspondence with points in the display of the object comprises monitoring the position of a cursor, and identifying the position of the cursor in the haptic model as a correspondence point responsive to an indication from the user.  
   
   
       5 . The method of  claim 1 , wherein modifying the initial haptic model comprises modifying the initial haptic model according to a set of parameterized operations, and determining parameters for the operations that produce a final haptic model whose correspondence points in a two-dimensional projection are collectively within an error metric of the correspondence points in the image of the object.  
   
   
       6 . The method of  claim 1 , wherein the image of the object is one of a sequence of images, and further comprising determining correspondence points in the image of the object in other images in the sequence, and further comprising generating haptic models for the object in other images in the sequence according to the haptic model correspondence points and the correspondence points determined for the object in each of the other images in the sequence.  
   
   
       7 . A method of providing a human-computer interface comprising a two-dimensional visual space and a generated three-dimensional haptic space; comprising: 
 a) Selecting from a library of haptic models an initial haptic model that is topologically similar to the object;    b) Establishing correspondence between a set of correspondence points in the three-dimensional initial haptic model and a set of correspondence points in the two-dimensional image of the object;    c) Generating a final haptic model by modifying the initial haptic model so that the two-dimensional projection of the correspondence points in the final haptic model are substantially aligned with the correspondence points in the two-dimensional image of the object.    d) Establishing a correspondence between a haptic cursor in the haptic space and a visual cursor in the visual space;    e) changing the haptic cursor position and visual cursor position responsive to user input;    f) Providing force feedback to the user according to the haptic model and the position of the haptic cursor relative to the haptic model.    
   
   
       8 . The method of  claim 7 , wherein providing force feedback comprises determining the position of the haptic cursor in the two-dimensional projection of the haptic model corresponding to the two-dimensional image, and establishing the depth relative to that projection of the haptic cursor to be the depth in the haptic space of the part of the haptic model that the haptic cursor first contacts.  
   
   
       9 . A method of generating a haptic model corresponding to an object in a visual image, comprising: 
 a) Selecting a haptic model from a set of haptic models, wherein the selected haptic model approximates desired haptic aspects of the object;    b) Establishing a correspondence between features of the selected haptic model and the representation of the object in the image; and    c) Generating a haptic model from the selected haptic model and the correspondence.    
   
   
       10 . The method of  claim 9 , wherein the set of haptic models comprises a plurality of haptic models, each associated with a class of objects and each representing a simplified haptic model adaptable to generate haptic models within the haptic perception range of objects within the associated class.  
   
   
       11 . The method of  claim 9 , wherein the image comprises a two-dimensional image.  
   
   
       12 . The method of  claim 9 , wherein the image comprises a two-dimensional image within a sequence of two-dimensional images.  
   
   
       13 . A method of generating a haptic model corresponding to a plurality of objects in a visual image, comprising, for each object in the visual image: 
 a) Selecting a haptic model from a set of haptic models, wherein the selected haptic model approximates desired haptic aspects of the object;    b) Establishing a correspondence between features of the selected haptic model and the representation of the object in the image; and    c) Generating a haptic model from the selected haptic model and the correspondence.    
   
   
       14 . A method of generating a three-dimensional haptic component of a human-computer interface corresponding to a two-dimensional image of an object, comprising: 
 a) Selecting from a library of haptic models an initial haptic model that is topologically similar to the object;    b) Establishing correspondence between a set of correspondence points in the three-dimensional initial haptic model and a set of correspondence points in the two-dimensional image of the object;    c) Determining an error between the two dimensional projection of a haptic model and the two-dimensional image of the object, where the haptic model is determined by applying a set of parameterized operations to the initial haptic model;    d) Determining parameters for the parameterized operations that produce a haptic model corresponding to an error within an error bound.    
   
   
       15 . A method of generating a three-dimensional haptic component of a human-computer interface corresponding to a two-dimensional image, comprising: 
 a) Selecting from a library of haptic models a three-dimensional haptic model that is topologically similar to the object;    b) Establishing a relationship between the two-dimensional image and the three-dimensional haptic model such that interaction with the two-dimensional image is linked to interaction with the three-dimensional haptic model.    
   
   
       16 . The method of  claim 15  wherein interaction with the two-dimensional image comprises motion of a cursor in a visual representation of the image responsive to user manipulation of an input device, and wherein interaction with the three-dimensional haptic model comprises force fed back to the input device responsive to user manipulation thereof.  
   
   
       17 . The method of  claim 15  wherein interactions are linked such that forces applied by the user to the three-dimensional haptic model affect the display of the two-dimensional image.  
   
   
       18 . The method of  claim 15  wherein establishing a relationship comprises establishing a correspondence between the three-dimensional haptic model and the two-dimensional image such that perceptive discrepancy therebetween is minimized.  
   
   
       19 . The method of  claim 15  wherein establishing a relationship comprises establishing a correspondence between the three-dimensional haptic model and the two-dimensional image such that an error representing the difference between the two-dimensional projection of the haptic model into to plane of the image and the two-dimensional image is minimized.