Object recognition method of autonomous driving device, and autonomous driving device

Abstract

Disclosed is an object recognition method including: obtaining a first RGB image by using a camera; predicting at least one first region, in which an object is unrecognizable, in the first RGB image based on brightness information of the first RGB image; determining at least one second region, in which an object exists, from among the at least one first region, based on object information obtained through a dynamic vision sensor; obtaining an enhanced second RGB image by controlling photographic configuration information of the camera in relation to the at least one second region; and recognizing the object in the second RGB image.

The invention claimed is: 
     
       1. A method, performed by an autonomous driving device, of recognizing an object, the method comprising:
 obtaining a first red, green, blue (RGB) image by using a camera arranged in the autonomous driving device; 
 predicting at least one first region in which an object is unrecognizable in the first RGB image based on brightness information of the first RGB image; 
 determining at least one second region in which an object exists, from among the at least one first region, based on object information obtained through a dynamic vision sensor (DVS) arranged in the autonomous driving device; 
 obtaining an enhanced second RGB image by controlling photographic configuration information of the camera in relation to the at least one second region; and 
 recognizing the object in the second RGB image. 
 
     
     
       2. The method of  claim 1 , wherein the predicting of the at least one first region in which the object is unrecognizable comprises predicting a region in which brightness values are out of a threshold range in the first RGB image as the at least one first region. 
     
     
       3. The method of  claim 1 , wherein the predicting of the at least one first region in which the object is unrecognizable comprises:
 analyzing the first RGB image to obtain a histogram of the first RGB image; 
 determining whether an object-unrecognizable region exists in the first RGB image by using the histogram of the first RGB image; and 
 when it is determined that the object-unrecognizable region exists in the first RGB image, predicting the at least one first region in the first RGB image based on the brightness information of the first RGB image. 
 
     
     
       4. The method of  claim 1 , wherein the predicting of the at least one first region in which the object is unrecognizable comprises:
 determining whether an object-unrecognizable region exists in the first RGB image by using a first artificial intelligence model that has learned a plurality of RGB images; and 
 when it is determined that the object-unrecognizable region exists in the first RGB image, predicting the at least one first region in the first RGB image by using the first artificial intelligence model. 
 
     
     
       5. The method of  claim 1 , wherein the object information comprises at least one of a dynamic vision sensor (DVS) image obtained by the dynamic vision sensor and position information of at least one object detected from the DVS image. 
     
     
       6. The method of  claim 1 , wherein the determining of the at least one second region in which the object exists, from among the at least one first region, comprises determining the at least one second region by applying a DVS image obtained by the dynamic vision sensor and the first RGB image to a second artificial intelligence model. 
     
     
       7. The method of  claim 1 , wherein the obtaining of the second RGB comprises controlling at least one of an exposure, a focus, and a white balance with respect to the at least one second region. 
     
     
       8. The method of  claim 1 , wherein the obtaining of the second RGB comprises adjusting at least one of a gain, an aperture, and an exposure time of the camera. 
     
     
       9. The method of  claim 1 , further comprising obtaining, when the second RGB image is composed of a plurality of frames, position information of the autonomous driving device by tracking a feature included in the object recognized from each of the plurality of frames. 
     
     
       10. The method of  claim 1 , further comprising determining a route of the autonomous driving device based on information about the recognized object. 
     
     
       11. The method of  claim 1 , further comprising:
 tracking the recognized object by using the camera; 
 detecting a new object appearing around the autonomous driving device by using the dynamic vision sensor; 
 determining, in response to the new object being detected, a candidate region in which a probability of recognizing the new object in a third RGB image obtained through the camera is greater than a threshold value; and 
 recognizing the new object from the third RGB image by performing image processing on the candidate region. 
 
     
     
       12. An autonomous driving device comprising:
 a camera; 
 a dynamic vision sensor (DVS); and 
 at least one processor, 
 wherein the at least one processor is configured to: 
 obtain a first red, green, blue (RGB) image by using the camera; 
 predict at least one first region in which an object is unrecognizable in the first RGB image based on brightness information of the first RGB image; 
 determine at least one second region in which an object exists, from among the at least one first region, based on object information obtained through the dynamic vision sensor; 
 obtain an enhanced second RGB image by controlling photographing configuration information of the camera in relation to the at least one second region; and 
 recognize the object in the second RGB image. 
 
     
     
       13. The autonomous driving device of  claim 12 , wherein the at least one processor is further configured to predict a region in which brightness values are out of a threshold range in the first RGB image as the at least one first region. 
     
     
       14. The autonomous driving device of  claim 12 , wherein the at least one processor is further configured to:
 analyze the first RGB image to obtain a histogram of the first RGB image; 
 determine whether an object-unrecognizable region exists in the first RGB image by using the histogram of the first RGB image; and 
 when it is determined that the object-unrecognizable region exists in the first RGB image, predict the at least one first region in the first RGB image based on the brightness information of the first RGB image. 
 
     
     
       15. The autonomous driving device of  claim 12 , wherein the at least one processor comprises an artificial intelligence processor configured to determine whether an object-unrecognizable region exists in the first RGB image by using a first artificial intelligence model that has learned a plurality of RGB images, and when it is determined that the object-unrecognizable region exists in the first RGB image, predict the at least one first region in the first RGB image by using the first artificial intelligence model. 
     
     
       16. The autonomous driving device of  claim 12 , wherein the at least one processor is further configured to control photographic configuration information of the camera by adjusting at least one of a gain, aperture, and exposure time of the camera. 
     
     
       17. The autonomous driving device of  claim 12 , wherein the at least one processor is further configured to:
 track the recognized object by using the camera; 
 detect, by using the dynamic vision sensor, a new object appearing around the autonomous driving device; 
 determine, in response to the new object being detected, a candidate region in which a probability of recognizing the new object in a third RGB image obtained through the camera is greater than a threshold value; and 
 recognize the new object from the third RGB image by performing image processing on the candidate region. 
 
     
     
       18. The autonomous driving device of  claim 12 , wherein the at least one processor is further configured to set a frame rate of the dynamic vision sensor to be equal to a frame rate of the camera. 
     
     
       19. The autonomous driving device of  claim 12 , further comprising at least one of an autonomous driving vehicle, an autonomous flying device, and an autonomous driving robot. 
     
     
       20. A computer program product comprising a recording medium having recorded thereon a program for:
 obtaining a first red, green, blue (RGB) image by using a camera; 
 predicting at least one first region in which an object is unrecognizable in the first RGB image based on brightness information of the first RGB image; 
 determining at least one second region in which an object exists, from among the at least one first region, based on object information obtained through a dynamic vision sensor (DVS); 
 obtaining an enhanced second RGB image by controlling photographing configuration information of the camera in relation to the at least one second region; and 
 recognizing the object in the second RGB image.