Systems and methods to interact with discoverable objects by pointing a light beam using a handheld device

Abstract

Systems and methods are described in which one or more attributes or cues related to a discoverable object are conveyed to a user of handheld device via audible, haptic and/or visual means. Using a light beam generated by the handheld device and pointed in the same direction as a device camera, the user may point toward one or more objects in the user's environment believed to be associated with the one or more attributes or cues. Based on the region being pointed at within camera images, the handheld device may classify objects within images to determine if an object matches one or more templates or classifications of the discoverable object. A match, or mismatch, of objects being pointed at may determine subsequent actions. Systems and methods may provide simple and intuitive methods for human-machine interactions that may be particularly well-suited for learners.

We claim: 
     
         1 . A method to indicate a discoverable object by a human using a handheld device including a device processor, a device light beam source configured to generate a light beam producing one or more light beam reflections off one or more visible objects viewable by the human, a device camera aligned such that a camera field-of-view includes a beam location region of the one or more light beam reflections and operatively coupled to the device processor, and a device speaker operatively coupled to the device processor, the method comprising:
 playing, by the device speaker, one or more audible cues related to the discoverable object;   acquiring, by the device camera, a camera image when the handheld device is manipulated by the human such that a projected light beam points from the device light beam source to one or more visible objects;   isolating, by the device processor, one or more indicated objects at the beam location region within the camera image; and   determining, by the device processor, whether one or more of the one or more indicated objects match a predetermined template of the discoverable object.   
     
     
         2 . The method of  claim 1 , wherein the device light beam source is one of a light-emitting diode and a lasing diode. 
     
     
         3 . The method of  claim 1 , wherein the device light beam source is operatively coupled to the device processor and wherein an intensity of the device light beam source is controlled by one or more of regulating a magnitude of a light beam driving current and a pulse width modulation of the light beam driving current. 
     
     
         4 . The method of  claim 1 , wherein the projected light beam is one or more of collimated, non-coherent, diverging, and patterned. 
     
     
         5 . The method of  claim 1 , wherein the device light beam source is operatively coupled to the device processor, the method further comprising turning the device light beam source off upon one or both of the acquiring of the camera image and the determining the match with the predetermined template of the discoverable object. 
     
     
         6 . The method of  claim 1 , wherein the beam location region within the camera image is determined by:
 acquiring, by the device camera, a baseline image that includes no reflection by the projected light beam;   acquiring, by the device camera, a light beam reflection image that includes one or more reflections produced by the projected light beam;   computing, by the device processor, a subtracted pixel intensity image based at least in part on the baseline image subtracted from the light beam reflection image; and   assigning to the beam location region, by the device processor, light beam sensed pixels within the subtracted pixel intensity image that exceed a predetermined light intensity threshold.   
     
     
         7 . The method of  claim 1 , wherein the one or more audible cues related to the discoverable object include one or more of: one or more sounds generated by the discoverable object, one or more names of the discoverable object, one or more questions about the discoverable object, one or more descriptions of the discoverable object, one or more described functions of the discoverable object, a mathematical problem related to the discoverable object, a musical score related to the discoverable object, and one or more related object descriptions related to the discoverable object. 
     
     
         8 . The method of  claim 1 , wherein the predetermined template of the discoverable object includes one or more of: one or more shapes of the discoverable object, one or more sizes of the discoverable object, one or more colors of the discoverable object, one or more textures of the discoverable object, and one or more patterns within the discoverable object. 
     
     
         9 . The method of  claim 1 , wherein the discoverable object is a printed object within one of a book, a book cover, a brochure, a box, a sign, a newspaper and a magazine. 
     
     
         10 . The method of  claim 1 , further comprising performing an action by the device processor based at least in part on the determining one of the match and a mismatch, with the predetermined template of the discoverable object. 
     
     
         11 . The method of  claim 10 , wherein the action comprises one or more of:
 transmitting, to one or more remote processors, one or more of the one or more audible cues, the camera image, an acquisition time of the acquiring the camera image, the predetermined camera image light beam pointing region, the predetermined template of the discoverable object, and the one or more indicated objects;   playing one or more sounds on the device speaker;   displaying one or more illumination patterns on one or more device displays operatively coupled to the device processor; and   activating a device haptic unit operatively coupled to the device processor.   
     
     
         12 . The method of  claim 1 , wherein a device switch is operatively coupled to the device processor, the method further comprising determining the match with the predetermined template of the discoverable object occurring upon detecting, by the device processor, a change in a switch state of the device switch. 
     
     
         13 . The method of  claim 1 , wherein a device microphone is operatively coupled to the device processor, the method further comprising determining the match with the predetermined template of the discoverable object occurring upon identifying, by the device processor, one or more identifiable sounds produced by the human within data acquired by the device microphone. 
     
     
         14 . The method of  claim 1 , wherein a device inertial measurement unit is operatively coupled to the device processor, the method further comprising determining the match with the predetermined template of the discoverable object occurring upon identifying, by the device processor, one of a predetermined handheld device gesture motion and a predetermined handheld device orientation within data acquired by the device inertial measurement unit. 
     
     
         15 . A method to indicate a discoverable object by a human using a handheld device including a device processor, a device light beam source configured to generate a light beam producing one or more light beam reflections off one or more visible objects viewable by the human, a device camera aligned such that a camera field-of-view includes a beam location region of the one or more light beam reflections and operatively coupled to the device processor, and one or more device displays operatively coupled to the device processor, the method comprising:
 displaying, by the one or more device displays, one or more visual cues related to the discoverable object;   acquiring, by the device camera, a camera image when the handheld device is manipulated by the human such that a projected light beam points from the device light beam source to the one or more visible objects;   isolating, by the device processor, one or more indicated objects at the beam location region within the camera image; and   determining, by the device processor, whether one or more of the one or more indicated objects match a predetermined template of the discoverable object.   
     
     
         16 . The method of  claim 15 , further comprising performing an action by the device processor based on the determining one of the match and a mismatch, with the predetermined template of the discoverable object. 
     
     
         17 . The method of  claim 16 , wherein the action comprises one or more of:
 transmitting, to one or more remote processors, one or more of: the one or more visual cues, the camera image, an acquisition time of the acquiring the camera image, the predetermined camera image light beam pointing region, the predetermined template of the discoverable object, and the one or more indicated objects;   playing one or more sounds on a device speaker operatively coupled to the device processor;   displaying one or more illumination patterns on the one or more device displays; and   activating a device haptic unit operatively coupled to the device processor.   
     
     
         18 . A method to indicate a discoverable object by a human using a handheld device including a device processor, a device light beam source configured to generate a light beam producing one or more light beam reflections off one or more visible objects viewable by the human, a device camera aligned such that a camera field-of-view includes a beam location region of the one or more light beam reflections and operatively coupled to the device processor, and a device haptic unit operatively coupled to the device processor, the method comprising:
 producing, by the device haptic unit, sensed haptic vibrations at one or more haptic frequencies related to one or more of motions and sounds associated with the discoverable object;   acquiring, by the device camera, a camera image when the handheld device is manipulated such that a projected light beam points from the device light beam source to the one or more visible objects;   isolating, by the device processor, one or more indicated objects at the beam location region within the camera image; and   determining, by the device processor, whether one or more of the one or more indicated objects match a predetermined template of the discoverable object.   
     
     
         19 . The method of  claim 18 , further comprising performing an action by the device processor based on the determining one of the match and a mismatch, with the predetermined template of the discoverable object. 
     
     
         20 . The method of  claim 19 , wherein the action comprises one or more of:
 transmitting, to one or more remote processors, one or more of: the one or more haptic frequencies, the camera image, an acquisition time of acquiring the camera image, the predetermined camera image light beam pointing region, the predetermined template of the discoverable object, and the one or more indicated objects;   playing one or more sounds on a device speaker operatively coupled to the device processor;   displaying one or more illumination patterns on one or more device displays operatively coupled to the device processor; and   activating a device haptic unit operatively coupled to the device processor.