System and method for performing collaborative learning of machine representations for a target concept

Abstract

Embodiments provide a system and method for performing collaborative learning of machine representations of a concept. During operation, the system can receive a user-specified object associated with a user's concept of interest. The system can compute a similarity score between a target feature vector associated with the user-specified object and a respective feature vector for a set of candidate objects. The system can determine, based on the similarity score, a first subset of candidate objects that satisfy a similarity threshold. The system can receive, via a GUI, a first user-feedback associated with a visual representation of the first subset of candidate objects. The first user-feedback can represent an elaboration of a current user's concept of interest. The system can then modify, based on the first user-feedback, the target feature vector and the similarity function, thereby providing an improved model for machine representations of a current user's concept of interest.

What is claimed is: 
     
       1. A computer-implemented method, comprising:
 receiving a user-specified object associated with a user's concept of interest; 
 computing, based on a similarity function, a similarity score between a target feature vector associated with the user-specified object and a respective feature vector for a set of candidate objects; 
 determining, based on the similarity score, a first subset of candidate objects that satisfy a similarity threshold within a specified tolerance; 
 presenting, via a graphical user interface (GUI), a visual representation of the first subset of candidate objects; 
 receiving a first user-feedback corresponding to the visual representation of the first subset of candidate objects, wherein the first user-feedback represents an elaboration of a current user's concept of interest; and 
 modifying, based on the first user-feedback, the target feature vector and the similarity function, thereby providing an improved model for machine representations of the current user's concept of interest; and 
 wherein computing the similarity score comprises computing, based on multiple types of deep neural networks (DNN) associated with each candidate object and user-specified object, a weighted ensemble average of similarities, wherein each type of DNN is associated with a different weight, wherein a respective weight is updated based on a user feedback. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein the user-specified object represents one or more of:
 a video clip; 
 an image; 
 text; and 
 others types of objects for which deep neural network embedded features exist. 
 
     
     
       3. The computer-implemented method of  claim 1 , wherein the first user-feedback includes:
 a selection of one or more non-matching candidate objects in the first subset of candidate objects; and 
 a de-selection of one or more matching candidate objects in the first subset of candidate objects. 
 
     
     
       4. The computer-implemented method of  claim 1 , further comprising:
 iteratively performing following operations until a user request to terminate searching for matching candidate objects is received:
 computing, based on the modified similarity function and the modified target feature vector, an updated similarity score between the modified target feature vector and a respective feature vector for the set of candidate objects; 
 determining, based on the updated similarity score, a second subset of candidate objects that satisfy an updated similarity threshold within the specified tolerance; 
 receiving a second user-feedback corresponding to a visual representation of the second subset of candidate objects, wherein the second user-feedback indicates a current user's concept of interest; and 
 modifying, based on the second user-feedback, the modified target feature vector and the modified similarity function, thereby further improving the model for machine representations of the current user's concept of interest in collaboration with the user. 
 
 
     
     
       5. The computer-implemented method of  claim 1 , wherein computing, based on the similarity function, the similarity score between the target feature vector associated with the user-specified object and the respective feature vector for a set of candidate objects comprises:
 computing inner products of respective feature vectors of candidate objects and the target feature vector. 
 
     
     
       6. The computer-implemented method of  claim 1 , wherein the visual representation of the first subset of candidate objects includes:
 one or more matching candidate objects; and 
 one or more non-matching candidate objects which are near misses. 
 
     
     
       7. The computer-implemented method of  claim 1 , wherein modifying, based on the first user-feedback, the target feature vector and the similarity function comprises:
 applying, based on the first user-feedback, a target bootstrapping technique to the target feature vector to obtain the modified target feature vector that is consistent with user selected candidate objects from the first subset of candidate objects, wherein the modified target feature vector emphasizes features that are similar to the user selected candidate objects and de-emphasizes features that are different. 
 
     
     
       8. The computer-implemented method of  claim 1 , wherein the first subset of candidate objects includes a plurality of candidate objects that is consistent with the first user-feedback. 
     
     
       9. The computer-implemented method of  claim 1 , wherein the first subset of candidate objects includes a plurality of candidate objects that enables the user to expand a scope of the current user's concept of interest. 
     
     
       10. A computer system, comprising:
 a processor; and 
 a storage device coupled to the processor and storing instructions which when executed by the processor cause the processor to perform a method, the method comprising:
 receiving a user-specified object associated with a user's concept of interest; 
 computing, based on a similarity function, a similarity score between a target feature vector associated with the user-specified object and a respective feature vector for a set of candidate objects; 
 determining, based on the similarity score, a first subset of candidate objects that satisfy a similarity threshold within a specified tolerance; 
 presenting, via a graphical user interface (GUI), a visual representation of the first subset of candidate objects; 
 receiving a first user-feedback corresponding to the visual representation of the first subset of candidate objects, wherein the first user-feedback represents an elaboration of a current user's concept of interest; and 
 modifying, based on the first user-feedback, the target feature vector and the similarity function, thereby providing an improved model for machine representations of the current user's concept of interest; and 
 wherein computing the similarity score comprises computing, based on multiple types of deep neural networks (DNN) associated with each candidate object and user-specified object, a weighted ensemble average of similarities, wherein each type of DNN is associated with a different weight, wherein a respective weight is updated based on a user feedback. 
 
 
     
     
       11. The computer system of  claim 10 , wherein the user-specified object represents one or more of:
 a video clip; 
 an image; 
 text; and 
 others types of objects for which deep neural network embedded features exist. 
 
     
     
       12. The computer system of  claim 10 , wherein the first user-feedback includes:
 a selection of one or more non-matching candidate objects in the first subset of candidate objects; and 
 a de-selection of one or more matching candidate objects in the first subset of candidate objects. 
 
     
     
       13. The computer system of  claim 10 , the method further comprising:
 iteratively performing following operations until a user request to terminate searching for matching candidate objects is received:
 computing, based on the modified similarity function and the modified target feature vector, an updated similarity score between the modified target feature vector and a respective feature vector for the set of candidate objects; 
 determining, based on the updated similarity score, a second subset of candidate objects that satisfy an updated similarity threshold within the specified tolerance; 
 receiving, from the user, a second user-feedback associated with a visual representation of the second subset of candidate objects, wherein the second user-feedback indicates a current user's concept of interest; and 
 modifying, based on the second user-feedback, the modified target feature vector and the modified similarity function, thereby further improving the model for machine representations of the current user's concept of interest in collaboration with the user. 
 
 
     
     
       14. The computer system of  claim 10 , wherein computing, based on the similarity function, the similarity score between the target feature vector associated with the user-specified object and the respective feature vector for a set of candidate objects comprises:
 computing inner products of respective feature vectors of candidate objects and the target feature vector. 
 
     
     
       15. The computer system of  claim 10 , wherein the visual representation of the first subset of candidate objects includes:
 one or more matching candidate objects; and 
 one or more non-matching candidate objects which are near misses. 
 
     
     
       16. The computer system of  claim 10 , wherein modifying, based on the first user-feedback, the target feature vector and the similarity function comprises:
 applying, based on the first user-feedback, a target bootstrapping technique to the target feature vector to obtain the modified target feature vector that is consistent with user selected candidate objects from the first subset of candidate objects, wherein the modified target feature vector emphasizes features that are similar to the user selected candidate objects and de-emphasizes features that are different. 
 
     
     
       17. The computer system of  claim 10 , wherein the first subset of candidate objects includes a plurality of candidate objects that is consistent with the first user-feedback. 
     
     
       18. The computer system of  claim 10 , wherein the first subset of candidate objects includes a plurality of candidate objects that enables the user to expand a scope of the current user's concept of interest. 
     
     
       19. A non-transitory computer-readable storage medium storing instructions to:
 receive a user-specified object associated with a user's concept of interest; 
 compute, based on a similarity function, a similarity score between a target feature vector associated with the user-specified object and a respective feature vector for a set of candidate objects; 
 determine, based on the similarity score, a first subset of candidate objects that satisfy a similarity threshold within a specified tolerance; 
 present, via a graphical user interface (GUI), a visual representation of the first subset of candidate objects; 
 receive a first user-feedback corresponding to the visual representation of the first subset of candidate objects, wherein the first user-feedback represents an elaboration of a current user's concept of interest; and 
 modify, based on the first user-feedback, the target feature vector and the similarity function, thereby providing an improved model for machine representations of the current user's concept of interest; and 
 wherein computing the similarity score comprises computing, based on multiple types of deep neural networks (DNN) associated with each candidate object and user-specified object, a weighted ensemble average of similarities, wherein each type of DNN is associated with a different weight, wherein a respective weight is updated based on a user feedback. 
 
     
     
       20. The non-transitory computer-readable storage medium of  claim 19 , wherein the instructions are further to compute the similarity score by computing inner products of respective feature vectors of candidate objects and the target feature vector.