Systems and Methods for Improving Performance of a Robotic Vehicle by Managing On-board Camera Defects

Abstract

Embodiments include methods performed by a processor of a robotic vehicle for detecting and responding to defects on an on-board imaging device that includes an image sensor. Various embodiments may include causing the imaging device to capture at least one image, determining whether a defect to the imaging device is detected based at least in part on the at least one captured image, and, in response to determining that a defect to the imaging device is detected, identifying an area of the image sensor corresponding to the defect and masking image data received from the identified area of the image sensor.

What is claimed is: 
     
         1 . A method performed by a processor of a robotic vehicle for detecting and responding to defects in an image sensor, the method comprising:
 capturing at least one image comprising a reference element by the image sensor;   determining whether there is a defect in the image sensor based at least in part on the at least one captured image, by:
 comparing features of the reference element in the captured image to features of a baseline image of the reference element; 
 determining whether features of the captured image of the reference element differ from features of a baseline image of the reference element by more than a threshold amount in one or more regions of the captured image; and 
   in response to determining that there is a defect in the image sensor:
 identifying an area of the image sensor corresponding to the defect; 
 masking image data received from the identified area of the image sensor; and 
 altering at least one of an operating mode or a path of the robotic vehicle based on an unmasked area of the image sensor. 
   
     
     
         2 . The method of  claim 1 , further comprising continuing active operation of the robotic vehicle in response to determining that there is a defect in the image sensor. 
     
     
         3 . The method of  claim 2 , wherein continuing active operation of the robotic vehicle comprises navigating the robotic vehicle using image data received from the unmasked area of the image sensor. 
     
     
         4 . The method of  claim 2 , wherein continuing active operation of the robotic vehicle comprises utilizing a second image sensor of the robotic vehicle to perform navigation. 
     
     
         5 . The method of  claim 1 , wherein masking image data received from the identified area of the image sensor comprises developing a protocol for excluding use by an application of image data received from the identified area of the image sensor in response to determining that there is a defect in the image sensor. 
     
     
         6 . The method of  claim 5 , wherein excluding use by the application of image data received from the identified area of the image sensor comprises excluding use of a region of the identified area of the image sensor. 
     
     
         7 . The method of  claim 1 , further comprising causing motion of the image sensor, wherein:
 capturing at least one image comprises capturing at least two temporally-separated images; and   determining whether there is a defect in the image sensor comprises:
 identifying features within the at least two temporally-separated images; and 
 determining whether any of the identified features maintains a fixed position between the at least two temporally-separated images. 
   
     
     
         8 . The method of  claim 1 , wherein determining whether there is a defect in the image sensor is further based on data received from an inertial sensor of the robotic vehicle. 
     
     
         9 . A robotic vehicle, comprising:
 an image sensor; and   a processor configured with processor-executable instructions to:
 use the image sensor to capture at least one image comprises an image of a reference element; 
 determine whether there is a defect in the image sensor based at least in part on the at least one captured image by:
 comparing features of the reference element in the captured image to features of a baseline image of the reference element; and 
 determining whether features of the reference element in the captured image differ from features of the baseline image of the reference element by more than a threshold amount in one or more regions of the captured image; and 
 
 in response to determining that there is a defect in the image sensor:
 identify an area of the image sensor corresponding to the defect; 
 mask image data received from the identified area of the image sensor; and 
 alter at least one of an operating mode or a path of the robotic vehicle based on an unmasked area of the image sensor. 
 
   
     
     
         10 . The robotic vehicle of  claim 9 , wherein the processor is further configured with processor-executable instructions to continue active operation of the robotic vehicle in response to determining that there is a defect in the image sensor. 
     
     
         11 . The robotic vehicle of  claim 10 , wherein the processor is further configured with processor-executable instructions to continue active operation of the robotic vehicle by navigating the robotic vehicle using the image data received from the unmasked area of the image sensor. 
     
     
         12 . The robotic vehicle of  claim 10 , wherein the processor is further configured with processor-executable instructions to continue active operation of the robotic vehicle by utilizing a second image sensor of the robotic vehicle to perform navigation. 
     
     
         13 . The robotic vehicle of  claim 9 , wherein the processor is further configured with processor-executable instructions to developing a protocol for excluding use by an application of image data received from the identified area of the image sensor in response to determining that there is a defect in the image sensor. 
     
     
         14 . The robotic vehicle of  claim 13 , wherein the processor is further configured with processor-executable instructions to cause motion of the image sensor, wherein the processor is further configured with processor-executable instructions to:
 capture at least one image by capturing at least two temporally-separated images; and   determine whether there is a defect in the image sensor by:
 identifying features within the at least two temporally-separated images; and 
 determining whether any of the identified features maintains a fixed position between the at least two temporally-separated images. 
   
     
     
         15 . The robotic vehicle of  claim 9 , wherein the processor is further configured with processor-executable instructions to determine whether there is a defect in the image sensor is detected is further based on data received from an inertial sensor of the robotic vehicle. 
     
     
         16 . The robotic vehicle of  claim 9 , wherein the processor is further configured with processor-executable instructions to determine whether there is a defect in the image sensor based on data received from an inertial sensor of the robotic vehicle. 
     
     
         17 . A processing device configured for use in a robotic vehicle and configured to:
 use an image sensor to capture at least one image comprises an image of a reference element;   determine whether there is a defect in the image sensor based at least in part on the at least one captured image by:
 comparing features of the reference element in the captured image to features of a baseline image of the reference element; and 
 determining whether features of the reference element in the captured image differ from features of the baseline image of the reference element by more than a threshold amount in one or more regions of the captured image; and 
   in response to determining that there is a defect in the image sensor:
 identify an area of the image sensor corresponding to the defect; 
 mask image data received from the identified area of the image sensor; and 
 alter at least one of an operating mode or a path of the robotic vehicle based on an unmasked area of the image sensor. 
   
     
     
         18 . The processing device of  claim 17 , wherein the processing device is further configured to continue active operation of the robotic vehicle in response to determining that there is a defect in the image sensor. 
     
     
         19 . The processing device of  claim 18 , wherein the processing device is further configured to continue active operation of the robotic vehicle by navigating the robotic vehicle using the image data received from the unmasked area of the image sensor. 
     
     
         20 . The processing device of  claim 18 , wherein the processing device is further configured to continue active operation of the robotic vehicle by utilizing a second image sensor of the robotic vehicle to perform navigation. 
     
     
         21 . The processing device of  claim 17 , wherein the processing device is further configured to developing a protocol for excluding use by an application of image data received from the identified area of the image sensor in response to determining that there is a defect in the image sensor. 
     
     
         22 . The processing device of  claim 21 , wherein the processing device is further configured to cause motion of the image sensor, wherein the processing device is further configured with processor-executable instructions to:
 capture at least one image by capturing at least two temporally-separated images; and   determine whether there is a defect in the image sensor by:
 identifying features within the at least two temporally-separated images; and 
 determining whether any of the identified features maintains a fixed position between the at least two temporally-separated images. 
   
     
     
         23 . A non-transitory processor readable medium having stored thereon processor-executable instructions configured to cause a processor of a robotic vehicle to perform operations, comprising:
 capturing at least one image comprising a reference element by an image sensor on board a robotic device;   determining whether there is a defect in the image sensor based at least in part on the at least one captured image, by:
 comparing features of the reference element in the captured image to features of a baseline image of the reference element; 
 determining whether features of the captured image of the reference element differ from features of a baseline image of the reference element by more than a threshold amount in one or more regions of the captured image; and 
   in response to determining that there is a defect in the image sensor:
 identifying an area of the image sensor corresponding to the defect; 
 masking image data received from the identified area of the image sensor; and 
 altering at least one of an operating mode or a path of the robotic vehicle based on an unmasked area of the image sensor. 
   
     
     
         24 . The non-transitory processor readable medium of  claim 23 , wherein the stored processor-executable instructions are configured to cause a processor to perform operations such that masking image data received from the identified area of the image sensor comprises developing a protocol for excluding use by an application of image data received from the identified area of the image sensor in response to determining that there is a defect in the image sensor. 
     
     
         25 . The non-transitory processor readable medium of  claim 24 , wherein the stored processor-executable instructions are configured to cause a processor to perform operations such that excluding use by the application of image data received from the identified area of the image sensor comprises excluding use of a region of the identified area of the image sensor. 
     
     
         26 . The non-transitory processor readable medium of  claim 23 , wherein the stored processor-executable instructions are configured to cause a processor to perform operations further comprising causing motion of the image sensor, and the stored processor-executable instructions are configured to cause a processor to perform operations such that:
 capturing at least one image comprises capturing at least two temporally-separated images; and   determining whether there is a defect in the image sensor comprises:
 identifying features within the at least two temporally-separated images; and 
 determining whether any of the identified features maintains a fixed position between the at least two temporally-separated images.