Systems, Methods, and Devices for Utilizing Radar with Smart Devices

Abstract

An electronic device includes an image sensor having a field of view, a radar module, one or more processors, and memory. The electronic device captures image data of a scene that is within the field of view of the image sensor. The electronic device receives radar signals from one or more objects in the scene. The device determines respective location and/or movement of the one or more objects based on the radar signals. It further identifies a first object of the one or more objects based on a combination of the image data and the radar signals.

1 . A method performed at an electronic device, that includes an image sensor having a field of view, a radar module, one or more processors, and memory, the method comprising:
 capturing image data of a scene that is within the field of view of the image sensor;   receiving radar signals from one or more objects in the scene;   determining respective location and/or movement of the one or more objects based on the radar signals; and   identifying a first object of the one or more objects based on a combination of the image data and the radar signals.   
     
     
         2 . The method of  claim 1 , wherein identifying a first object of the one or more objects based on a combination of the image data and the radar signals comprises:
 employing the radar signals to enhance image processing of the electronic device in low light conditions; and/or   employing the radar signals to enhance image processing of the electronic device when the first object is obstructed from the field of view of the image sensor.   
     
     
         3 . The method of  claim 1 , wherein identifying the first object comprises detecting an anomaly associated with the first object. 
     
     
         4 . The method of  claim 3 , wherein the scene is part of a home environment, the method further comprising:
 comparing the determined location and/or movement of the one or more objects with stored data of known objects found in the home environment;   and wherein detecting the anomaly associated with the first object is in accordance with the comparing.   
     
     
         5 . The method of  claim 3 , further comprising:
 in accordance with detecting the anomaly, directing focus of the image sensor to a first portion of the scene based on a first determined location and/or movement of the first object; and   capturing via the image sensor an image of the first portion of the scene.   
     
     
         6 . The method of  claim 3 , further comprising: generating a notification in response to detecting the anomaly associated with the first object. 
     
     
         7 . The method of  claim 6 , wherein generating a notification comprises sending an alert to a personal device of an occupant of the home environment. 
     
     
         8 . The method of  claim 3 , wherein detecting an anomaly associated with a first object includes determining that the first object has one or more of: an unexpected velocity, an unexpected acceleration, and an unexpected location. 
     
     
         9 . An electronic device comprising:
 an image sensor having a field of view;   a radar module;   one or more processors;   memory; and   one or more programs stored in the memory for execution by the one or more processors, the one or more programs comprising instructions for:
 capturing image data of a scene that is within the field of view of the image sensor; 
 receiving radar signals from one or more objects in the scene; 
 determining respective location and/or movement of the one or more objects based on the radar signals; and 
 identifying a first object of the one or more objects based on a combination of the image data and the radar signals. 
   
     
     
         10 . The electronic device of  claim 9 , the one or more programs further comprising instructions for:
 time-synchronizing the radio signals and the image data.   
     
     
         11 . The electronic device of  claim 9 , the one or more programs further comprising instructions for:
 determining occupancy of at least a portion of the home environment based at least in part on the radar data.   
     
     
         12 . The electronic device of  claim 9 , wherein the camera device includes a plurality of LEDs; and
 the instructions for capturing the image data includes instructions for illuminating the scene using the plurality of LEDs.   
     
     
         13 . The electronic device of  claim 9 , wherein the instructions for identifying the first object comprises instructions for identifying a type of the object. 
     
     
         14 . The electronic device of  claim 12 , wherein the type of one of: a person, an animal or a vehicle. 
     
     
         15 . The electronic device of  claim 9 , wherein the instructions for identifying the first object comprises instructions for assigning an event category to the first object. 
     
     
         16 . A non-transitory computer readable storage medium storing one or more programs configured for execution by an electronic device that includes an image sensor having a field of view, a radar module, one or more processors, and memory, the one or more programs comprising instructions for:
 capturing image data of a scene that is within the field of view of the image sensor;   receiving radar signals from one or more objects in the scene;   determining respective location and/or movement of the one or more objects based on the radar signals; and   identifying a first object of the one or more objects based on a combination of the image data and the radar signals.   
     
     
         17 . The non-transitory computer readable storage medium of  claim 16 , wherein the radar signals comprise biometric characteristics, the one or more programs further comprising instructions for:
 recognizing individuals in the scene based on the biometric characteristics encoded in the radar data.   
     
     
         18 . The non-transitory computer readable storage medium of  claim 16 , wherein the radar module that is configured to operate with a constrained detection range, less than the full detection range of the radar module, the constrained detection range corresponding to a perimeter of the home environment. 
     
     
         19 . The non-transitory computer readable storage medium of  claim 16 , wherein the instructions for identifying a first object of the one or more objects based on a combination of the image data and the radar signals comprises instructions for:
 employing the radar signals to enhance image processing of the electronic device in low light conditions; and/or   employing the radar signals to enhance image processing of the electronic device when the first object is obstructed from the field of view of the image sensor.   
     
     
         20 . The non-transitory computer readable storage medium of  claim 16 , wherein the instructions for identifying the first object comprises instructions for detecting an anomaly associated with the first object.