Multi-stage autonomous localization architecture for charging electric vehicles

Abstract

An automated charging system for an electric vehicle is disclosed that includes a plug with a built-in camera assembly. The camera assembly captures images of a charging port of the electric vehicle, which are processed by one or more processors to estimate the location of the charging port relative to the plug. A multi-stage localization architecture is described that includes a gross localization procedure and a fine localization procedure. The gross localization procedure can implement a first convolutional neural network (CNN) to estimate a position of an object in the image. The fine localization procedure can implement a second CNN to estimate a position and orientation of the object. Actuators for moving the plug in a three-dimensional space can be controlled by the multi-stage localization architecture.

What is claimed is: 
     
       1. A method for estimating localization of an object in an image relative to a position and orientation of a camera, the method comprising:
 performing a gross localization procedure to estimate a target position in three-dimensional space based on a first image of the object captured by the camera; 
 moving the camera based on the estimated target position; 
 capturing a second image of the object by the camera after moving the camera based on the estimated target position; 
 performing a second gross localization procedure to estimate a second target position in the three-dimensional space based on the second image of the object captured by the camera; 
 moving the camera based on the second estimated target position; 
 capturing a third image of the object by the camera after moving the camera based on the second estimated target position; and 
 performing a fine localization procedure to estimate a new target position and a target orientation in the three-dimensional space based on the third image of the object. 
 
     
     
       2. The method of  claim 1 , wherein the gross localization procedure comprises processing the first image by a first convolutional neural network configured to generate a three-element output vector that represents the target position for the camera in the three-dimensional space relative to a current position of the camera, and wherein the fine localization procedure comprises processing the second image by a second convolutional neural network configured to generate an output vector with at least one position coordinate and at least one orientation coordinate. 
     
     
       3. The method of  claim 2 , wherein, prior to performing the gross localization procedure, the convolutional neural network is trained based on a set of training data that includes a set of input images and corresponding target output vectors. 
     
     
       4. The method of  claim 2 , wherein the three-element output vector includes a radial coordinate, an angular coordinate, and an azimuth coordinate. 
     
     
       5. The method of  claim 2 , wherein the at least one position coordinate includes at least one of a radial coordinate, an angular coordinate, and a height coordinate, and wherein the at least one orientation coordinate includes an angular rotation coordinate associated with a corresponding axis. 
     
     
       6. The method of  claim 1 , wherein the fine localization procedure comprises processing the second image to apply feature detection and/or feature matching algorithms to locate the object in the second image. 
     
     
       7. The method of  claim 1 , wherein the gross localization procedure comprises processing the first image by a neural network configured to perform object detection, wherein an output of the neural network comprises at least one of coordinates for a bounding box or a segmentation mask. 
     
     
       8. The method of  claim 7 , wherein the gross localization procedure further comprises processing the coordinates for the bounding box to calculate the estimated target position. 
     
     
       9. The method of  claim 1 , wherein the gross localization procedure comprises processing the first image by a first convolutional neural network, and the fine localization procedure comprises processing the second image by a second convolutional neural network. 
     
     
       10. The method of  claim 9 , wherein the first convolutional neural network includes fewer convolution layers than the second convolutional neural network. 
     
     
       11. The method of  claim 1 , wherein the gross localization procedure is performed by a processor, and wherein the fine localization procedure is performed by a machine learning (ML) accelerator connected to the processor. 
     
     
       12. The method of  claim 11 , wherein the ML accelerator is configured to implement a convolutional neural network configured to generate an output vector that includes three position coordinates and at least one orientation coordinate. 
     
     
       13. A system comprising:
 a camera assembly; 
 a memory; and 
 at least one processor coupled to the memory and configured to:
 perform a gross localization procedure to estimate a target position in three-dimensional space based on a first image of an object captured by the camera assembly; 
 move the camera assembly based on the estimated target position; 
 capture a second image of the object by the camera after moving the camera based on the estimated target position; 
 perform a second gross localization procedure to estimate a second target position in the three-dimensional space based on the second image of the object captured by the camera; 
 move the camera based on the second estimated target position; 
 capture a third image of the object by the camera assembly after moving the camera based on the second estimated target position; and 
 perform a fine localization procedure to estimate a new target position and a target orientation in the three-dimensional space based on third image of the object. 
 
 
     
     
       14. The system of  claim 13 , wherein the gross localization procedure comprises processing the first image by a first convolutional neural network configured to generate a three-element output vector that represents the target position for the camera in the three-dimensional space relative to a current position of the camera, and wherein the fine localization procedure comprises processing the second image by a second convolutional neural network configured to generate an output vector with at least one position coordinate and at least one orientation coordinate. 
     
     
       15. The system of  claim 14 , the system further comprising: a
 machine learning (ML) accelerator coupled to the at least one processor and configured to execute at least one of the first convolutional neural network or the second convolutional neural network. 
 
     
     
       16. The system of  claim 13 , wherein the fine localization procedure comprises processing the second image to apply feature detection and/or feature matching algorithms to locate the object in the second image. 
     
     
       17. The system of  claim 13 , wherein the camera assembly is mounted on a plug associated with a charging port of an electric vehicle, and wherein moving the camera assembly comprises generating signals for one or more actuators configured to move the plug in the three-dimensional space. 
     
     
       18. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to:
 perform a gross localization procedure to estimate a target position in three-dimensional space based on a first image of an object captured by a camera assembly; 
 move the camera assembly based on the estimated target position; 
 capture a second image of the object by the camera assembly after moving the camera assembly based on the estimated target position; 
 perform a second gross localization procedure to estimate a second target position in the three-dimensional space based on the second image of the object captured by the camera assembly, 
 move the camera assembly based on the second estimated target position; 
 capture a third image of the object by the camera assembly after moving the camera assembly based on the second estimated target position; and 
 perform a fine localization procedure to estimate a new target position and a target orientation in the three-dimensional space based on the third image of the object. 
 
     
     
       19. The non-transitory computer-readable storage medium of  claim 18 , wherein the gross localization procedure comprises processing the first image by a first convolutional neural network configured to generate a three-element output vector that represents the target position for the camera in the three-dimensional space relative to a current position of the camera. 
     
     
       20. The non-transitory computer-readable storage medium of  claim 19 , wherein the fine localization procedure comprises processing the second image by a second convolutional neural network configured to generate an output vector with at least one position coordinate and at least one orientation coordinate. 
     
     
       21. The non-transitory computer-readable storage medium of  claim 18 , wherein the fine localization procedure comprises processing the second image to apply feature detection and/or feature matching algorithms to locate the object in the second image. 
     
     
       22. The method of  claim 1 , further comprising:
 capturing a third image of the object by the camera after moving the camera based on the new target position and the target orientation; and 
 performing a second fine localization procedure to estimate a further new target position and a further new target orientation in the three-dimensional space based on the third image of the object.