Spatio-temporal awareness engine for priority tree based region selection across multiple input cameras and multimodal sensor empowered awareness engine for target recovery and object path prediction

Abstract

A spatio-temporal awareness engine combines a low-resolution tracking process and high resolution tracking process to employ an array of imaging sensors to track an object within the visual field. The system utilizes a low-resolution conversion through noise filtering and feature consolidation to load-balance the more computationally-intensive aspects of object tracking, allowing for a more robust system, while utilizing less computer resources. A process for target recovery and object path prediction in a robotic drone may include tracking targets using a combination of visual and acoustic multimodal sensors, operating a camera as a main tracking sensor of the multimodal sensors and feeding output of the camera to a spatiotemoral engine, complementing the main tracking sensor with non-visual, fast secondary sensors to assign rough directionality to a target tracking signal, and applying the rough directionality to prioritize visual scanning by the main tracking sensor.

1 . (canceled) 
     
     
         2 . A system, comprising:
 an image capture module;   a sensor module;   a flight control module; and   one or more processors configured to:
 monitor multiple regions of interest in image data from the image capture module; 
 detect a change in one of the regions of interest at a first resolution; 
 in response to detecting the change in one of the regions of interest, analyze at least a portion of the image data at a second resolution different from the first resolution to detect an item of interest; and 
 in response to detecting the item of interest, determine a response direction for the flight control module based at least in part on the sensor module, wherein the response direction is associated with a location for the item of interest. 
   
     
     
         3 . The system of  claim 2 , wherein a navigation system is controlled based on the location for the item of interest. 
     
     
         4 . The system of  claim 2 , wherein the image capture module is controlled based on the location for the item of interest. 
     
     
         5 . The system of  claim 2 , wherein the change in one of the regions of interest includes an anomaly in one of the regions of interest. 
     
     
         6 . The system of  claim 2 , wherein the change in one of the regions of interest includes no longer detecting the item of interest in one of the regions of interest. 
     
     
         7 . The system of  claim 2 , wherein the sensor module includes at least one of the following: a radar sensor, a radio frequency sensor, a proximity sensor, an acoustic sensor, a thermal sensor, a night vision sensor, or a global positioning sensor. 
     
     
         8 . The system of  claim 2 , wherein the one or more processors are further configured to track the item of interest. 
     
     
         9 . The system of  claim 2 , wherein the one or more processors are further configured to use a set of processing schemes, wherein each processing scheme of the set of processing schemes is associated with a corresponding performance score that indicates a confidence in the corresponding processing scheme, wherein an active processing scheme is repeatedly determined based on the processing scheme with the highest performance score from the set of processing schemes. 
     
     
         10 . The system of  claim 9 , wherein the one or more processors are configured to use the set of processing schemes including by being configured to select another processing scheme to process video data from the capture module in response to the corresponding performance score of the active processing scheme failing to satisfy a threshold. 
     
     
         11 . The system of  claim 9 , wherein the one or more processors are configured to use the set of processing schemes including by being configured to select another processing scheme based on determining a number of items of interest in the image data. 
     
     
         12 . The system of  claim 9 , wherein the set of processing schemes utilize different resource levels to process data from the image capture module. 
     
     
         13 . The system of  claim 9 , wherein the one or more processors are configured to use the set of processing schemes including by being configured to perform one or more of the following: produce an object location estimation independent of historical information, use a last known location of an object location estimation and using a fast validation method to validate the estimation, and use a known location of the object location estimation and/or a fast validation method to validate the estimation. 
     
     
         14 . A method, comprising:
 monitoring multiple regions of interest in image data from an image capture module;   detecting a change in one of the regions of interest at a first resolution;   in response to detecting the change in one of the regions of interest, analyzing at least a portion of the image data at a second resolution different from the first resolution to detect an item of interest; and   in response to detecting the item of interest, determining a response direction for a flight control module based at least in part on a sensor module, wherein the response direction is associated with a location for the item of interest.   
     
     
         15 . The method of  claim 14 , wherein a navigation system is controlled based on the location for the item of interest. 
     
     
         16 . The method of  claim 14 , wherein the image capture module is controlled based on the location for the item of interest. 
     
     
         17 . The method of  claim 14 , wherein the change in one of the regions of interest includes an anomaly in one of the regions of interest. 
     
     
         18 . The method of  claim 14 , wherein the change in one of the regions of interest includes no longer detecting the item of interest in one of the regions of interest. 
     
     
         19 . The method of  claim 14 , further comprising tracking the item of interest. 
     
     
         20 . The method of  claim 14 , further comprising using a set of processing schemes, wherein each processing scheme of the set of processing schemes is associated with a corresponding performance score that indicates a confidence in the corresponding processing scheme, wherein an active processing scheme is repeatedly determined based on the processing scheme with the highest performance score from the set of processing schemes. 
     
     
         21 . A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
 monitoring multiple regions of interest in image data from an image capture module;   detecting a change in one of the regions of interest at a first resolution;   in response to detecting the change in one of the regions of interest, analyzing at least a portion of the image data at a second resolution different from the first resolution to detect an item of interest; and   in response to detecting the item of interest, determining a response direction for a flight control module based at least in part on a sensor module, wherein the response direction is associated with a location for the item of interest.