Image based localization system

Abstract

Systems and methods for determining a location based on image data are provided. A method can include receiving, by a computing system, a query image depicting a surrounding environment of a vehicle. The query image can be input into a machine-learned image embedding model and a machine-learned feature extraction model to obtain a query embedding and a query feature representation, respectively. The method can include identifying a subset of candidate embeddings that have embeddings similar to the query embedding. The method can include obtaining a respective feature representation for each image associated with the subset of candidate embeddings. The method can include determining a set of relative displacements between each image associated with the subset of candidate embeddings and the query image and determining a localized state of a vehicle based at least in part on the set of relative displacements.

What is claimed is: 
     
       1. A computer-implemented method for determining a localized state of an autonomous vehicle, the method comprising:
 receiving, by a computing system comprising one or more computing devices, a query image collected by the autonomous vehicle and depicting a surrounding environment of the autonomous vehicle; 
 inputting, by the computing system, the query image into a machine-learned image embedding model to receive a query embedding as an output of the machine-learned image embedding model; 
 accessing, by the computing system, a database of pre-computed image embeddings, the pre-computed image embeddings previously computed for a plurality of images by the machine-learned image embedding model; 
 obtaining, by the computing system, a plurality of candidate embeddings from the database of pre-computed image embeddings based at least in part on vehicle location data associated with the autonomous vehicle and image location data associated with each pre-computed image embedding in the database of pre-computed image embeddings; 
 comparing, by the computing system, the query embedding to the plurality of candidate embeddings to identify a subset of candidate embeddings that have embeddings that satisfy a similarity threshold; and 
 determining, by the computing system, the localized state of the autonomous vehicle based at least in part on the image location data associated with each pre-computed image embedding in the subset of candidate embeddings. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein determining the localized state of the autonomous vehicle based at least in part on the image location data associated with each pre-computed image embedding in the subset of candidate embeddings further comprises:
 inputting, by the computing system, the query image into a machine-learned feature extraction model to obtain a query feature representation for the query image; 
 obtaining, by the computing system, a respective feature representation for a respective image associated with each candidate embedding in the subset of candidate embeddings; 
 for each candidate embedding in the subset of candidate embeddings, inputting, by the computing system, the query feature representation and the respective feature representation for the respective image associated with the candidate embedding into a machine-learned regression model to obtain a respective relative displacement between the query image and the image associated with the candidate embedding; 
 determining, by the computing system, the localized state of the autonomous vehicle based at least in part on a set of relative displacements that comprises the respective relative displacement between the query image and the respective image associated with each of the candidate embeddings in the subset of candidate embeddings. 
 
     
     
       3. The computer-implemented method of  claim 2 , wherein the respective feature representation for the respective image associated with each candidate embedding in the subset of candidate embeddings is previously computed by the machine-learned feature extraction model and obtaining, by the computing system, each respective feature representation comprises obtaining, by the computing system, the respective feature representation from a database of feature representations. 
     
     
       4. The computer-implemented method of  claim 2 , wherein determining the localized state of the autonomous vehicle based at least in part on the set of relative displacements comprises aggregating the set of relative displacements to obtain the localized state. 
     
     
       5. The computer-implemented method of  claim 4 , wherein aggregating the set of relative displacements comprises determining one or more median location coordinates and a median heading angle associated with the set of relative displacements. 
     
     
       6. The computer-implemented method of  claim 2 , wherein the machine-learned regression model and the machine-learned feature extraction model have been jointly trained end-to-end on a set of training data that comprises a plurality of pairs of training images, each pair of training images having a known ground truth displacement between the pair of training images. 
     
     
       7. The computer-implemented method of  claim 1 , wherein the vehicle location data associated with the autonomous vehicle and the image location data associated with each of the pre-computed image embeddings comprise geolocation coordinates. 
     
     
       8. The computer-implemented method of  claim 1 , wherein the machine-learned image embedding model is previously trained using a triplet training scheme, the triplet training scheme utilizing a plurality of image triplets, each image triplet in the plurality of image triplets comprising an anchor image, a positive image, and a negative image, wherein:
 the anchor image is associated with a respective geolocation that is closer to a respective geolocation associated with the positive image than a respective geolocation associated with the negative image; and 
 the positive image is associated with a respective heading angle within a respective heading angle associated with the anchor image by a heading threshold. 
 
     
     
       9. The computer-implemented method of  claim 1 , further comprising:
 controlling, by the computing system, motion of the autonomous vehicle based at least in part on the localized state of the autonomous vehicle. 
 
     
     
       10. A computing system comprising:
 one or more processors; and 
 one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the computing system to perform operations comprising:
 receiving a query image collected by an autonomous vehicle and depicting a surrounding environment of the autonomous vehicle; 
 inputting the query image into a machine-learned image embedding model to receive a query embedding as an output of the machine-learned image embedding model; 
 accessing a database of pre-computed image embeddings, the pre-computed image embeddings previously computed for a plurality of images by the machine-learned image embedding model; 
 obtaining a plurality of candidate embeddings from the database of pre-computed image embeddings based at least in part on vehicle location data associated with the autonomous vehicle and image location data associated with each pre-computed image embedding in the database of pre-computed image embeddings; 
 comparing the query embedding to the plurality of candidate embeddings to identify a subset of candidate embeddings that satisfy a threshold; and 
 determining a localized state of the autonomous vehicle based at least in part on the image location data associated with each pre-computed image embedding in the subset of candidate embeddings. 
 
 
     
     
       11. The computing system of  claim 10 , wherein determining the localized state of the autonomous vehicle further comprises:
 inputting the query image into a machine-learned feature extraction model to obtain a query feature representation for the query image; 
 for each candidate embedding in the subset of candidate embeddings,
 obtaining a respective feature representation for a respective image associated with the candidate embedding; and 
 inputting the query feature representation and the respective feature representation into a machine-learned regression model to obtain a respective relative displacement between the query image and the respective image associated with the candidate embedding; and 
 
 determining the localized state of the autonomous vehicle based at least in part on a set of relative displacements that comprise the respective relative displacement between the query image and the respective image associated with each of the candidate embeddings in the subset of candidate embeddings. 
 
     
     
       12. The computing system of  claim 11 , wherein the respective feature representation for a respective image associated with each candidate embedding in the subset of candidate embeddings is previously computed for each of the plurality of images by the machine-learned feature extraction model and obtaining each respective feature representation comprises obtaining the respective feature representation from a database of feature representations. 
     
     
       13. The computing system of  claim 10 , wherein the vehicle location data associated with the autonomous vehicle and the image location data associated with each of the pre-computed image embeddings in the database of pre-computed image embeddings comprise geolocation coordinates. 
     
     
       14. The computing system of  claim 13 , wherein obtaining the plurality of candidate embeddings from the database of pre-computed image embeddings comprises:
 determining a Euclidean distance between the geolocation coordinates associated with the autonomous vehicle and the geolocation coordinates associated with each pre-computed image embedding in the database of pre-computed image embeddings; and 
 obtaining the plurality of candidate embeddings associated with a Euclidean distance below a distance threshold. 
 
     
     
       15. An autonomous vehicle comprising:
 one or more vehicle sensors; 
 one or more processors; 
 a machine-learned feature extraction model; 
 a machine-learned regression model; and 
 one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising:
 collecting, via the one or more vehicle sensors, a query image depicting a surrounding environment of the autonomous vehicle; 
 obtaining, via the machine-learned feature extraction model, a query feature representation by inputting the query image into the machine-learned feature extraction model; 
 for each of a plurality of candidate images:
 obtaining a respective feature representation associated with the candidate image; and 
 obtaining, via the machine-learned regression model, a respective relative displacement by inputting the query feature representation and the respective feature representation into the machine-learned regression model; and 
 
 determining a localized state of the autonomous vehicle based at least in part on the respective relative displacement obtained for each of the plurality of candidate images. 
 
 
     
     
       16. The autonomous vehicle of  claim 15 , wherein the autonomous vehicle further comprises a machine-learned embedding model; and the operations further comprise:
 obtaining, via the machine-learned image embedding model, a query embedding by inputting the query image into the machine-learned image embedding model; 
 obtaining a plurality of candidate embeddings from a database of pre-computed image embeddings based at least in part on vehicle location data associated with the autonomous vehicle; and 
 comparing the query embedding to the plurality of candidate embeddings to identify a subset of candidate embeddings that satisfy a threshold, wherein the plurality of candidate images comprise images that are respectively associated with the subset of candidate embeddings. 
 
     
     
       17. The autonomous vehicle of  claim 16 , wherein the database of pre-computed image embeddings is remotely located from the autonomous vehicle. 
     
     
       18. The autonomous vehicle of  claim 15 , wherein the respective feature representation for each of the plurality of candidate images is previously computed by the machine-learned feature extraction model. 
     
     
       19. The autonomous vehicle of  claim 18 , wherein the respective feature representation for each of the plurality of candidate images is obtained from a feature representation database remotely located from the autonomous vehicle. 
     
     
       20. The autonomous vehicle of  claim 19 , wherein the autonomous vehicle further comprises one or more communication interfaces; and
 obtaining the respective feature representation for each of a plurality of candidate images further comprises accessing, via the one or more communication interfaces, the feature representation database to obtain the respective feature representation associated with each candidate image in the plurality of candidate images.