Monocular 2D semantic keypoint detection and tracking

Abstract

A method for 2D semantic keypoint detection and tracking is described. The method includes learning embedded descriptors of salient object keypoints detected in previous images according to a descriptor embedding space model. The method also includes predicting, using a shared image encoder backbone, salient object keypoints within a current image of a video stream. The method further includes inferring an object represented by the predicted, salient object keypoints within the current image of the video stream. The method also includes tracking the inferred object by matching embedded descriptors of the predicted, salient object keypoints representing the inferred object within the previous images of the video stream based on the descriptor embedding space model.

What is claimed is: 
     
       1. A method for 2D semantic keypoint detection and tracking, comprising:
 learning embedded descriptors of salient vehicle object keypoints detected in previous images to train a shared descriptor embedding space model; 
 predicting, using a shared image encoder backbone, salient vehicle object keypoints within a current image of a video stream; 
 inferring a vehicle represented by the predicted, salient vehicle object keypoints within the current image of the video stream; 
 processing, by a trained shared descriptor embedding space model, embedded descriptors of the salient vehicle object keypoints detected in the previous frames and embedded descriptors of the predicted, salient vehicle object keypoints representing the inferred vehicle within the current frame; 
 matching the predicted, salient vehicle object keypoints representing the inferred vehicle within the current frame with salient vehicle object keypoints representing the inferred vehicle within the previous images of the video stream based on the trained descriptor embedding space model; and 
 tracking the inferred vehicle within the previous images, the current image, and subsequent images of the video stream based on the matching. 
 
     
     
       2. The method of  claim 1 , in which learning the embedded descriptors comprises:
 predicting the salient vehicle object keypoints and the descriptors within frames of the video stream; and 
 identifying, using the descriptors, associated keypoints at different frames of the video stream corresponding to the salient vehicle object keypoints based on the embedding space model. 
 
     
     
       3. The method of  claim 2 , further comprising:
 generating known transformations of an input image; 
 warping the input image to form a warped image; 
 extracting keypoints and the descriptors from the input image and the warped image; 
 computing corresponding keypoints through the known transformations between the input image and the warped image; and 
 ensuring the descriptors of the corresponding keypoints match the extracted keypoints. 
 
     
     
       4. The method of  claim 1 , in which predicting the salient vehicle object keypoints comprises:
 extracting, using the shared image encoder backbone, the salient vehicle object keypoints within the current image of the video stream based on relevant appearance and geometric features of the current image; and 
 generating a keypoint heatmap based on the salient vehicle object keypoints extracted using the shared image encoder backbone. 
 
     
     
       5. The method of  claim 1 , further comprising:
 generating, using a descriptor head, the embedded descriptors of the salient vehicle object keypoints; and 
 computing the salient vehicle object keypoints in the previous images of the video stream using the embedded descriptors generated using the descriptor head. 
 
     
     
       6. The method of  claim 1 , in which the salient vehicle object keypoints approximate geometry/spatial relationships of a rigid-body object of the vehicle. 
     
     
       7. The method of  claim 1 , further comprising planning a trajectory of an ego vehicle according to the tracking of the inferred vehicle. 
     
     
       8. A non-transitory computer-readable medium having program code recorded thereon for 2D semantic keypoint detection and tracking, the program code being executed by a processor and comprising:
 program code to learn embedded descriptors of salient vehicle object keypoints detected in previous images to train a shared descriptor embedding space model; 
 program code to predict, using a shared image encoder backbone, salient vehicle object keypoints within a current image of a video stream; 
 program code to infer a vehicle represented by the predicted, salient vehicle object keypoints within the current image of the video stream; 
 program code to process, by a trained shared descriptor embedding space model, embedded descriptors of the salient vehicle object keypoints detected in the previous frames and embedded descriptors of the predicted, salient vehicle object keypoints representing the inferred object within the current frame; 
 program code to match the predicted, salient vehicle object keypoints representing the inferred object vehicle within the current frame with salient vehicle object keypoints representing the inferred vehicle within the previous images of the video stream based on the descriptor embedding space model; and 
 program code to track the inferred vehicle within the previous images, the current image, and subsequent images of the video stream based on the program code to match. 
 
     
     
       9. The non-transitory computer-readable medium of  claim 8 , in which the program code to learn the embedded descriptors comprises:
 program code to predict the salient vehicle object keypoints and the descriptors within frames of the video stream; and 
 program code to identify, using the descriptors, associated keypoints at different frames of the video stream corresponding to the salient vehicle object keypoints based on the embedding space model. 
 
     
     
       10. The non-transitory computer-readable medium of  claim 9 , further comprising:
 program code to generate known transformations of an input image; 
 program code to warp the input image to form a warped image; 
 program code to extract keypoints and the descriptors from the input image and the warped image; 
 program code to compute corresponding keypoints through the known transformations between the input image and the warped image; and 
 program code to ensure the descriptors of the corresponding keypoints match extracted keypoints. 
 
     
     
       11. The non-transitory computer-readable medium of  claim 8 , in which the program code to predict the salient vehicle object keypoints comprises:
 program code to extract, using the shared image encoder backbone, the salient vehicle object keypoints within the current image of the video stream based on relevant appearance and geometric features of the current image; and 
 program code to generate a keypoint heatmap based on the salient vehicle object keypoints extracted using the shared image encoder backbone. 
 
     
     
       12. The non-transitory computer-readable medium of  claim 8 , further comprising:
 program code to generate, using a descriptor head, the embedded descriptors of the salient vehicle object keypoints; and 
 program code to compute the salient vehicle object keypoints in the previous images of the video stream using the embedded descriptors generated using the descriptor head. 
 
     
     
       13. The non-transitory computer-readable medium of  claim 8 , in which the salient vehicle object keypoints approximate geometry/spatial relationships of a rigid-body object of the vehicle. 
     
     
       14. The non-transitory computer-readable medium of  claim 8 , further comprising program code to plan a trajectory of an ego vehicle according to the program code to track the inferred vehicle. 
     
     
       15. A system for 2D semantic keypoint detection and tracking, the system comprising:
 a semantic keypoint detection module to learn embedded descriptors of salient vehicle object keypoints detected in previous images to train a shared descriptor embedding space model; 
 a semantic keypoint descriptor module to predict, using a shared image encoder backbone, salient vehicle object keypoints within a current image of a video stream; 
 a keypoint inference model to infer a vehicle represented by the predicted, salient vehicle object keypoints within the current image of the video stream; 
 a trained shared descriptor embedding space model to process embedded descriptors of the salient vehicle object keypoints detected in the previous frames and embedded descriptors of the predicted, salient vehicle object keypoints representing the inferred vehicle within the current frame; and 
 a keypoint tracking module to match the predicted, salient vehicle object keypoints representing the inferred object vehicle within the current frame with salient vehicle object keypoints representing the inferred vehicle within the previous images of the video stream based on the descriptor embedding space model and to track the inferred vehicle within the previous images, the current image, and subsequent images of the video stream based on the match. 
 
     
     
       16. The system of  claim 15 , in which the salient vehicle object keypoints approximate geometry/spatial relationships of a rigid-body object of the vehicle. 
     
     
       17. The system of  claim 15 , further comprising a planner module to plan a trajectory of an ego vehicle according to the tracking of the inferred-object vehicle.