Generating a Descriptor of Second Modality Data Associated With Data of a First Modality

Abstract

The present invention relates to a computer-implemented method of generating a descriptor associated with data of a first modality. The method comprises: receiving first data associated with a first modality, and second data associated of a second modality, wherein the first and second modalities are different; generating respective first and second descriptors corresponding to the respective first and second modalities by encoding the first and second data using respective first and second encoders, the first and second encoders respectively trained based on first training data of the first modality and second training data of the second modality; transforming the first descriptor into a third descriptor, the third descriptor corresponding to the second modality; and storing the third descriptor in a database

1 . A computer-implemented method of generating a descriptor associated with data of a first modality, the method comprising:
 receiving first data associated with a first modality, and second data associated of a second modality, wherein the first and second modalities are different;   generating respective first and second descriptors corresponding to the respective first and second modalities by encoding the first and second data using respective first and second encoders, the first and second encoders respectively trained based on first training data of the first modality and second training data of the second modality;   transforming the first descriptor into a third descriptor, the third descriptor corresponding to the second modality; and   storing the third descriptor in a database.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein transforming the first descriptor into the third descriptor comprises:
 decoding the first descriptor into decoded third data corresponding to the second modality using a decoder, the decoder trained to generate data of the second modality; and   encoding the third data to generate the third descriptor corresponding to the second modality using a third encoder.   
     
     
         3 . The computer-implemented method of  claim 1 , wherein the first modality and the second modality respectively correspond to modality of a sensor used to record the respective data. 
     
     
         4 . The computer-implemented method of  claim 3 , wherein the sensor comprises one of a LIDAR sensor, a RADAR sensor, and a camera. 
     
     
         5 . The computer-implemented method of  claim 1 , wherein the first encoder and the second encoder are from respective auto encoders, and wherein the respective first descriptor and second descriptor is a representation corresponding to a bottle neck of the respective auto encoder. 
     
     
         6 . The computer-implemented method of  claim 5 , wherein at least one of the respective auto encoders is a variational auto encoder. 
     
     
         7 . A computer-implemented method of training a descriptor generator for generating a descriptor associated with data of a first modality, the descriptor generator comprising a first encoder, a second encoder, and a transformer, the method comprising:
 training the first encoder to generate a first descriptor corresponding to first data of a first modality using first training data of the first modality;   training the second encoder to generate a second descriptor corresponding to second data of a second modality using second training data of the second modality, wherein the first and second modalities are different; and   training the transformer to transform the first descriptor into a third descriptor corresponding to the second modality using contrastive learning based on the first and second training data.   
     
     
         8 . The computer-implemented method of  claim 7 , wherein training the transformer using contrastive learning comprises:
 determining a distance between the second descriptor and the third descriptor; and   modifying the transformer to reduce the distance.   
     
     
         9 . The computer-implemented method of  claim 7 , wherein the training transformer using contrastive learning comprises:
 comparing, using a discriminator, the second descriptor and the second descriptor; and   determining if the third descriptor is real or fake using the discriminator.   
     
     
         10 . The computer-implemented method of  claim 7 , wherein the descriptor generator further comprises an inverse transformer, and wherein the method further comprises:
 transforming the third descriptor into a fourth descriptor using the inverse transformer;   computing a distance between the first descriptor and the fourth descriptor; and   modifying the inverse transformer to reduce the distance.   
     
     
         11 . The computer-implemented method of  claim 7 , wherein the first modality and the second modality respectively correspond to a modality used to record the respective data. 
     
     
         12 . The computer-implemented method of  claim 11 , wherein the sensor comprises one of a LIDAR sensor, a RADAR sensor, and a camera. 
     
     
         13 . The computer-implemented method of  claim 7 , wherein the first encoder and the second encoder are each an encoder of an autoencoder. 
     
     
         14 . The computer-implemented method of  claim 13 , wherein the autoencoder is a variational autoencoder. 
     
     
         15 . A computer-implemented method of retrieving data of a first modality, the method comprising:
 retrieving a descriptor from a database of descriptors associated with data of a second modality;   transforming, using an inverse transformer, the retrieved descriptor into a transformed descriptor, the transformed descriptor associated with the first modality, wherein the first and second modalities are different; and   retrieving data of the first modality associated with the transformed descriptor from a database.   
     
     
         16 . The computer-implemented method of  claim 15 , wherein the first and second modalities each correspond to a modality of a sensor used to record the respective data. 
     
     
         17 . The computer-implemented method of  claim 16 , wherein the sensor is one of a LiDAR sensor, a RADAR sensor, or a camera. 
     
     
         18 . The computer-implemented method of  claim 15 , further comprising:
 obtaining real-time data from a sensor of the first modality;   comparing the real-time data to the retrieved data; and   operating an autonomous vehicle to move based on the comparison.   
     
     
         19 . A transitory, or non-transitory, computer-readable medium including instructions stored thereon that when executed by a processor, cause the processor to perform a method comprising:
 receiving first data associated with a first modality, and second data associated of a second modality, wherein the first and second modalities are different;   generating respective first and second descriptors corresponding to the respective first and second modalities by encoding the first and second data using respective first and second encoders, the first and second encoders respectively trained based on first training data of the first modality and second training data of the second modality;   transforming the first descriptor into a third descriptor, the third descriptor corresponding to the second modality; and   storing the third descriptor in a database.