Vehicle localization

Abstract

In one aspect, a vehicle localization system implements the following steps: receiving a predetermined road map; receiving at least one captured image from an image capture device of a vehicle; processing, by a road detection component, the at least one captured image, to identify therein road structure for matching with corresponding structure of the predetermined road map, and determine a location of the vehicle relative to the identified road structure; and using the determined location of the vehicle relative to the identified road structure to determine a location of the vehicle on the road map, by matching the road structure identified in the at least one captured image with the corresponding road structure of the predetermined road map.

1 .- 45 . (canceled) 
     
     
         46 . A vehicle localization method implemented in a computer system, the method comprising:
 receiving a predetermined road map;   receiving at least one road image for determining a vehicle location;   processing, by a road detection component, the at least one road image, to identify therein road structure for matching with corresponding structure of the predetermined road map, and determine the vehicle location relative to the identified road structure; and   using the determined vehicle location relative to the identified road structure to determine a vehicle location on the road map, by matching the road structure identified in the at least one road image with the corresponding road structure of the predetermined road map;   wherein the road structure identified in the at least one road image comprises:
 a centre line for matching with a corresponding centre line of the road map, wherein determining the vehicle location relative thereto comprises determining a lateral separation between the vehicle and the centre line in a direction perpendicular to the centre line; and/or 
 a junction region for matching with a corresponding junction region of the predetermined road map, wherein determining the vehicle location relative thereto comprises determining a longitudinal separation between the vehicle and the junction region in a direction along a road being travelled by the vehicle. 
   
     
     
         47 . A method according to  claim 46 , wherein the road structure identified in the at least one road image comprises a centre line for matching with a corresponding centre line of the road map and a junction region for matching with a corresponding junction region of the predetermined road map;
 wherein determining the vehicle location relative thereto comprises determining a lateral separation between the vehicle and the centre line in a direction perpendicular to the centre line and a longitudinal separation between the vehicle and the junction region in a direction along a road being travelled by the vehicle.   
     
     
         48 . A method according to  claim 46 , comprising:
 using the determined vehicle location on the predetermined road map to determine a location, relative to the vehicle location, of expected road structure indicated by the predetermined road map; and   merging the road structure identified in the at least one road image with the expected road structure indicated by the predetermined road map, to determine merged road structure and a location of the merged road structure relative to the vehicle location on the predetermined road map.   
     
     
         49 . A method according to  claim 46 , wherein the road detection component identifies the road structure in the at least one road image and the vehicle location relative to the identified road structure by assigning, to each of a plurality of spatial points within the image, at least one road structure classification value, and determining a location of those spatial points in a vehicle frame of reference. 
     
     
         50 . A method according to  claim 49 , comprising:
 using the determined vehicle location on the predetermined road map to determine a location, relative to the vehicle location, of expected road structure indicated by the predetermined road map; and   merging the road structure identified in the at least one road image with the expected road structure indicated by the predetermined road map, to determine merged road structure and a location of the merged road structure relative to the vehicle location on the predetermined road map;   wherein the merging comprises merging the road structure classification value assigned to each of those spatial points with a corresponding road structure value determined from the predetermined road map for a corresponding spatial point on the predetermined road map.   
     
     
         51 . A method according to  claim 46 , comprising:
 determining an approximate vehicle location on the road map and using the approximate vehicle location to determine a target area of the map containing the corresponding road structure for matching with the road structure identified in the at least one road image, wherein the vehicle location on the predetermined road map that is determined by matching those structures has a greater accuracy than the approximate vehicle location.   
     
     
         52 . A method according to  claim 46 , wherein the road image comprises 3D image data and the vehicle location relative to the identified road structure is determined using depth information of the 3D image data. 
     
     
         53 . A method according to  claim 52 , wherein the predetermined road map is a two dimensional road map and the method comprises a step of using the depth information to geometrically project the identified road structure onto a plane of the two dimensional road map for matching with the corresponding road structure of the two dimensional road map. 
     
     
         54 . A method according to  claim 46 , wherein the road map is a three dimensional road map, the vehicle location on the predetermined road map being a three dimensional location in a frame of reference of the predetermined road map. 
     
     
         55 . A method according to  claim 46 , comprising:
 determining an error estimate for the determined vehicle location on the predetermined road map, based on the matching of the visually identified road structure with the corresponding road structure of the road map.   
     
     
         56 . A method according to  claim 55 , comprising:
 receiving one or more further vehicle location estimates on the road map, each with an associated indication of error; and   applying a filter to: (i) the vehicle location on the road map as determined from the structure matching and the error estimate determined therefor, and (ii) the one or more further vehicle location estimates and the indication(s) of error received therewith, in order to determine an overall vehicle location estimate on the road map.   
     
     
         57 . A method according to  claim 55 , comprising:
 using the determined vehicle location on the predetermined road map to determine a location, relative to the vehicle location, of expected road structure indicated by the predetermined road map;   wherein determining the location of the expected road structure comprises determining, based on the road map and the error estimate, a plurality of expected road structure confidence values for a plurality of spatial points in a vehicle frame of reference.   
     
     
         58 . A method according to  claim 57 , comprising merging the road structure identified in the at least one road image with the expected road structure indicated by the predetermined road map, to determine merged road structure and a location of the merged road structure relative to the vehicle location on the predetermined road map, wherein the merging is performed in dependence on the expected road structure confidence values for those spatial points. 
     
     
         59 . A method according to  claim 46 , wherein the road detection component comprises a convolutional neural network, the road structure being identified by applying the convolutional neural network to the at least one road image. 
     
     
         60 . A method according to  claim 59 , wherein the road structure identified in the at least one road image comprises a road shape identified by applying a first convolutional neural network to the at least one road image, and a junction region for matching with a corresponding junction region of the predetermined road map, the junction region identified by applying a second convolutional neural network to the at least one road image. 
     
     
         61 . A method according to  claim 46 , wherein the matching is performed by determining an approximate vehicle location on the road map, determining a region of the road map corresponding to the at least one image based on the approximate location, computing an error between the captured at least one image and the corresponding region of the road map, and adapting the approximate location using an optimization algorithm to minimize the computed error, and thereby determining the said vehicle location on the road map. 
     
     
         62 . A method according to  claim 46 , comprising determining an error estimate for the determined vehicle location on the predetermined road map, based on the matching of the visually identified road structure with the corresponding road structure of the road map, wherein the determined error estimate comprises or is derived from the error between the road image and the corresponding region of the road map as computed upon completion of the optimization algorithm. 
     
     
         63 . A method according to  claim 46 , wherein the road structure identified in the at least one road image is matched with the corresponding road structure of the predetermined road map by matching a shape of the identified road structure with a shape of the corresponding road structure. 
     
     
         64 . A computer system, comprising:
 a map input configured to receive a predetermined road map;   an image input configured to receive at least one road image for determining a vehicle location; and one or more hardware processors configured to:   process the at least one road image, to identify therein road structure for matching with corresponding structure of the predetermined road map, and determine the vehicle location relative to the identified road structure; and   use the determined vehicle location relative to the identified road structure to determine a vehicle location on the road map, by matching the road structure identified in the at least one road image with the corresponding road structure of the predetermined road map; wherein the road structure identified in the at least one road image comprises:   a centre line for matching with a corresponding centre line of the road map, wherein determining the vehicle location relative thereto comprises determining a lateral separation between the vehicle and the centre line in a direction perpendicular to the centre line; and/or   a junction region for matching with a corresponding region of the predetermined road map, wherein determining the vehicle location relative thereto comprises determining a separation between the vehicle and the junction region in a direction along a road being travelled by the vehicle.   
     
     
         65 . A computer program comprising executable instructions stored on a non-transitory computer-readable storage medium and configured, when executed on one or more processors, to implement operations comprising:
 receiving a predetermined road map;   receiving at least one road image for determining a vehicle location;   processing, by a road detection component, the at least one road image, to identify therein road structure for matching with corresponding structure of the predetermined road map, and determine the vehicle location relative to the identified road structure; and   using the determined vehicle location relative to the identified road structure to determine a vehicle location on the road map, by matching the road structure identified in the at least one road image with the corresponding road structure of the predetermined road map;   wherein the road structure identified in the at least one road image comprises:
 a centre line for matching with a corresponding centre line of the road map, wherein determining the vehicle location relative thereto comprises determining a lateral separation between the vehicle and the centre line in a direction perpendicular to the centre line; and/or 
 a junction region for matching with a corresponding junction region of the predetermined road map, wherein determining the vehicle location relative thereto comprises determining a longitudinal separation between the vehicle and the junction region in a direction along a road being travelled by the vehicle.