Advanced driver assist system, method of calibrating the same, and method of detecting object in the same

Abstract

An advanced driver assist system (ADAS) includes a processing circuit and a memory storing instructions executable by the processing circuit. The processing circuit executes the instructions to cause the ADAS to: obtain, from a vehicle, a video sequence including a plurality of frames captured while driving the vehicle, where each of the frames corresponds to a stereo image including a first viewpoint image and a second viewpoint image; determine depth information in the stereo image based on reflected signals received while driving the vehicle; fuse the stereo image and the depth information to generated fused information, and detect at least one object included in the stereo image based on the fused information.

What is claimed is: 
     
       1. A processing circuit comprising:
 an image pre-processor configured to generate a pre-processed stereo image from a stereo image including a first viewpoint image and a second viewpoint image, the stereo image corresponding to each of a plurality of frames; 
 a first depth information generation engine configured to generate a first depth information in the stereo image based on radar reflected signals received from at least one radar; and 
 an object detection module configured to:
 extract features from the pre-processed stereo image having a first resolution to generate feature vectors; 
 increase a resolution of the first depth information having a second resolution according to the first resolution to generate a resized depth image; 
 fuse the feature vectors and the resized depth image using a plurality of convolutional layers of a convolutional neural network to generate fused feature vectors; 
 input the fused feature vectors to a feature pyramid network to generate feature maps; and 
 use a box predictor to detect at least one object included in the stereo image based on the feature maps to provide a final image or to provide a bounding box indicating the detected at least one object. 
 
 
     
     
       2. The processing circuit of  claim 1 , wherein the first resolution is greater than the second resolution. 
     
     
       3. The processing circuit of  claim 1 , wherein the object detection module is configured to perform down-sampling on the resized depth image at least once to generate a down-sampled depth image and configured to fuse the down-sampled depth image and the feature vectors. 
     
     
       4. The processing circuit of  claim 1 , wherein the object detection module is configured to mark the at least one object with the bounding box by using at least one of a single shot detector (SSD) and a faster recurrent convolution neural network (R-CNN). 
     
     
       5. The processing circuit of  claim 1 , wherein the first resolution is the same as the second resolution. 
     
     
       6. The processing circuit of  claim 1 , wherein the object detection module is further configured to:
 obtain a first trained model based on a result of detecting a learning object from a video sequence including a plurality of learning frames captured while driving a learning vehicle; and 
 detect the at least one object in the stereo image by using the obtained first trained model. 
 
     
     
       7. The processing circuit of  claim 1 , wherein the object detection module comprises:
 a feature extractor including a plurality of layers, and configured to extract features of the at least one object from the stereo image having the first resolution by using the plurality of layers to provide feature vectors; and 
 a sensor fusion engine configured to fuse the feature vectors and the depth information having the second resolution to generate fused feature vectors on the at least one object. 
 
     
     
       8. The processing circuit of  claim 7 , wherein:
 the first resolution is greater than the second resolution; and 
 the sensor fusion engine is configured to increase a size and a resolution of the depth information having the second resolution with respect to the first resolution to generate the resized depth image and configured to fuse the feature vectors and the resized depth image by using a plurality of convolution layers. 
 
     
     
       9. The processing circuit of  claim 7 , wherein:
 the first resolution is the same as the second resolution; and 
 the sensor fusion engine is configured perform down-sampling on the resized depth image at least once to generate a down-sampled depth image and configured to fuse the down-sampled depth image and the feature vectors by using a plurality of convolution layers. 
 
     
     
       10. The processing circuit of  claim 1 , further comprising:
 a second depth information generation engine configured to generate a second depth information in the stereo image based on the pre-processed stereo image, and wherein the object detection module is configured to generate the resized depth image further based on the second depth information. 
 
     
     
       11. A processing circuit comprising:
 an image pre-processor configured to generate a pre-processed stereo image from a stereo image including a first viewpoint image and a second viewpoint image, the stereo image corresponding to each of a plurality of frames; 
 a depth information generation engine configured to generate a depth information in the stereo image based on the pre-processed stereo image; and 
 an object detection module configured to:
 extract features from the pre-processed stereo image having a first resolution to generate feature vectors; 
 increase a resolution of the depth information having a second resolution according to the first resolution to generate a resized depth image; 
 fuse the feature vectors and the resized depth image using a plurality of convolutional layers of a convolutional neural network to generate fused feature vectors; 
 input the fused feature vectors to a feature pyramid network to generate feature maps; and 
 use a box predictor to detect at least one object included in the stereo image based on the feature maps to provide a final image or to provide a bounding box indicating the detected at least one object.