Multi-sensor sequential calibration system

Abstract

Techniques for performing a sensor calibration using sequential data is disclosed. An example method includes receiving, from a first camera located on a vehicle, a first image comprising at least a portion of a road comprising lane markers, where the first image is obtained by the camera at a first time; obtaining a calculated value of a position of an inertial measurement (IM) device at the first time; obtaining an optimized first extrinsic matrix of the first camera by adjusting a function of a first actual pixel location of a location of a lane marker in the first image and an expected pixel location of the location of the lane marker; and performing autonomous operation of the vehicle using the optimized first extrinsic matrix of the first camera when the vehicle is operated on another road or at another time.

What is claimed is: 
     
         1 . A method of calibrating sensors, comprising:
 obtaining, from a first sensor located on a vehicle, a first sensor data comprising at least a first portion of a road comprising a lane marker;   obtaining, from a second sensor located on the vehicle, a second sensor data comprising at least a second portion of the road; and   calibrating the first sensor and the second sensor based on an association between the first sensor data and the second sensor data, a stored map information of the road, and one or more values based on one or more measurements provided by one or more sensors on the vehicle.   
     
     
         2 . The method of  claim 1 ,
 wherein the calibrating the first sensor includes obtaining a first set of parameters of the first sensor,   wherein the first set of parameters include extrinsic parameters of the first sensor, and   wherein the first set of parameters are obtained by minimizing a difference between a first pixel location of a location of the lane marker on the road determined from the first sensor data and a first expected pixel location of the location of the lane marker on the road determined from the stored map information of the road.   
     
     
         3 . The method of  claim 2 , wherein the difference between the first pixel location and the first expected pixel location is minimized by adjusting a previously known third set of parameters of the first sensor. 
     
     
         4 . The method of  claim 2 , wherein the first expected pixel location is determined from the stored map information of the road, a pose value of an inertial measurement (IM) device based on measurements provided by the IM device and a global positioning system (GPS) device, a matrix that describes optical characteristics of the first sensor, and a previously known third set of parameters of the first sensor. 
     
     
         5 . The method of  claim 1 ,
 wherein the calibrating the second sensor includes obtaining a second set of parameters of the second sensor,   wherein the second set of parameters include extrinsic parameters of the second sensor, and   wherein the second set of parameters are obtained by adjusting a previously known fourth set of parameters of the second sensor that causes a matrix multiplication of an equation to equal zero, and   wherein the equation describes a constraint between the first sensor and the second sensor.   
     
     
         6 . The method of  claim 5 , wherein the equation is based on:
 a first matrix that describes a first ray from the first sensor to a first pixel location of a location of the lane marker on the road determined from the first sensor data,   a second matrix that describes a second ray from the second sensor to a second pixel location of the location of the lane marker on the road determined from the second sensor data, and   a third matrix that describes a relationship between the first pixel location and the second pixel location.   
     
     
         7 . The method of  claim 6 ,
 wherein the first sensor data and the second sensor data are obtained at a same time, and   wherein the third matrix is based on a previously known third set of parameters of the first sensor and the previously known fourth set of parameters of the second sensor.   
     
     
         8 . The method of  claim 6 ,
 wherein the first sensor data and the second sensor data are obtained at different times, and   wherein the third matrix is based on a previously known third set of parameters of the first sensor, the previously known fourth set of parameters of the second sensor, a first pose value of an inertial measurement (IM) device when the first sensor data is obtained, and a second pose value of the IM device when the second sensor data is obtained.   
     
     
         9 . The method of  claim 6 , wherein the first matrix is based on intrinsic parameters of the first sensor and the first pixel location. 
     
     
         10 . The method of  claim 6 , wherein the second matrix is based on intrinsic parameters of the second sensor and the second pixel location. 
     
     
         11 . A system for calibrating sensors, the system comprising a processor configured to:
 obtain, from a first sensor located on a vehicle, a first sensor data comprising at least a first portion of a road comprising a lane marker;   obtain, from a second sensor located on the vehicle, a second sensor data comprising at least a second portion of the road; and   calibrate the first sensor and the second sensor based on an association between the first sensor data and the second sensor data, a stored map information of the road, and one or more values based on one or more measurements provided by one or more sensors on the vehicle.   
     
     
         12 . The system of  claim 11 , wherein the obtain the first sensor data, the obtain the second sensor data, and the calibrate the first sensor and the second sensor are performed while the vehicle is driven on the road. 
     
     
         13 . The system of  claim 12 , wherein the vehicle is driven between a pre-defined starting location and a pre-defined ending location on the road. 
     
     
         14 . The system of  claim 11 , wherein the first sensor and the second sensor include a first camera and a second camera, respectively. 
     
     
         15 . The system of  claim 14 , wherein the first camera and the second camera have overlapping field of views. 
     
     
         16 . A non-transitory computer readable storage medium having code stored thereon, the code, when executed by a processor, causing the processor to implement a method comprising:
 obtaining, from a first sensor located on a vehicle, a first sensor data comprising at least a first portion of a road comprising a lane marker;   obtaining, from a second sensor located on the vehicle, a second sensor data comprising at least a second portion of the road; and   calibrating the first sensor and the second sensor based on an association between the first sensor data and the second sensor data, a stored map information of the road, and one or more values based on one or more measurements provided by one or more sensors on the vehicle.   
     
     
         17 . The non-transitory computer readable storage medium of  claim 16 , wherein the lane marker includes a solid line on the road. 
     
     
         18 . The non-transitory computer readable storage medium of  claim 17 , wherein the stored map information includes world coordinates of the solid line at an interval. 
     
     
         19 . The non-transitory computer readable storage medium of  claim 16 , wherein the lane marker includes a lane block from a plurality of lane blocks on the road. 
     
     
         20 . The non-transitory computer readable storage medium of  claim 19 ,
 wherein a first pixel location of a location of the lane marker on the road determined from the first sensor data, and   wherein the first pixel location of the location of the lane marker includes the first pixel location of a corner of the lane block.