Intelligent online personal assistant with multi-turn dialog based on visual search

Abstract

Systems, methods, and computer program products for identifying a relevant candidate product in an electronic marketplace. Embodiments perform a visual similarity comparison between candidate product image visual content and input query image visual content, process formal and informal natural language user inputs, and coordinate aggregated past user interactions with the marketplace stored in a knowledge graph. Visually similar items and their corresponding product categories, aspects, and aspect values can determine suggested candidate products without discernible delay during a multi-turn user dialog. The user can then refine the search for the most relevant items available for purchase by providing responses to machine-generated prompts that are based on the initial search results from visual, voice, and/or text inputs. An intelligent online personal assistant can thus guide a user to the most relevant candidate product more efficiently than existing search tools.

What is claimed is: 
     
       1. A method comprising:
 receiving an input query image; 
 assembling a list of candidate products based on a visual similarity measure between the input query image and one or more candidate products in the list of candidate products; 
 providing, for at least a subset of candidate products in the list of candidate products, corresponding knowledge graph information comprising aggregate historical electronic marketplace user interaction information; 
 determining, based on the corresponding knowledge graph information, a mismatch between the list of candidate products and a non-image input; 
 outputting an instruction to cause display of a user prompt on a graphical user interface of a client device, wherein the user prompt requests a specified user input in a multi-turn dialog; 
 receiving, from the client device via the multi-turn dialog, the specified user input; 
 generating a filtered list of candidate products based at least upon the specified user input; and 
 providing the filtered list of candidate products. 
 
     
     
       2. The method of  claim 1 , wherein the user prompt comprises a statement prompt denoting the mismatch between the list of candidate products and the non-image input. 
     
     
       3. The method of  claim 1 , further comprising outputting a second instruction to cause display of a question prompt associated with the list of candidate products. 
     
     
       4. The method of  claim 1 , wherein the user prompt comprises at least one of:
 a selectable candidate product image; or 
 a question requesting additional natural language input. 
 
     
     
       5. The method of  claim 1 , wherein the corresponding knowledge graph information comprises product information. 
     
     
       6. The method of  claim 5 , wherein the product information comprises one or more of:
 category data; 
 product aspect data; or 
 aspect values frequently used when searching for particular products. 
 
     
     
       7. The method of  claim 1 , wherein the input query image comprises one of:
 a photograph; 
 a video frame; 
 a sketch; or 
 a diagram. 
 
     
     
       8. The method of  claim 1 , wherein the user prompt is generated using machine learning to formulate a question directed to understanding user intent. 
     
     
       9. A non-transitory computer-readable storage medium having embedded therein a set of instructions which, when executed by one or more processors of a computer, causes the computer to execute operations comprising:
 receiving an input query image; 
 assembling a list of candidate products based on a visual similarity measure between the input query image and one or more candidate products in the list of candidate products; 
 providing, for at least a subset of candidate products in the list of candidate products, corresponding knowledge graph information comprising aggregate historical electronic marketplace user interaction information; 
 determining, based on the corresponding knowledge graph information, a mismatch between the list of candidate products and a non-image input; 
 outputting an instruction to cause display of a user prompt on a graphical user interface of a client device, wherein the user prompt requests a specified user input in a multi-turn dialog; 
 receiving, from the client device via the multi-turn dialog, the specified user input; 
 generating a filtered list of candidate products based at least upon the specified user input; and 
 providing the filtered list of candidate products. 
 
     
     
       10. The non-transitory computer-readable storage medium of  claim 9 , wherein the user prompt comprises a statement prompt denoting the mismatch between the list of candidate products and the non-image input. 
     
     
       11. The non-transitory computer-readable storage medium of  claim 9 , wherein the user prompt comprises at least one of:
 a selectable candidate product image; or 
 a question requesting additional natural language input. 
 
     
     
       12. The non-transitory computer-readable storage medium of  claim 9 , wherein the corresponding knowledge graph information comprises product information. 
     
     
       13. The non-transitory computer-readable storage medium of  claim 12 , wherein the product information comprises one or more of:
 category data; 
 product aspect data; or 
 aspect values frequently used when searching for particular products. 
 
     
     
       14. The non-transitory computer-readable storage medium of  claim 9 , wherein the input query image comprises one of:
 a photograph; 
 a video frame; 
 a sketch; or 
 a diagram. 
 
     
     
       15. The non-transitory computer-readable storage medium of  claim 9 , wherein the user prompt is generated using machine learning to formulate a question directed to understanding user intent. 
     
     
       16. A system comprising:
 at least one processor; and 
 at least one memory encoding computer-executable instructions that, when executed by the at least one processor, perform operations comprising:
 receiving an input query image; 
 assembling a list of candidate products based on a visual similarity measure between the input query image and one or more candidate products in the list of candidate products; 
 providing, for at least a subset of candidate products in the list of candidate products, corresponding knowledge graph information comprising aggregate historical electronic marketplace user interaction information; 
 determining, based on the corresponding knowledge graph information, a mismatch between the list of candidate products and a non-image input; 
 outputting an instruction to cause display of a user prompt on a graphical user interface of a client device, wherein the user prompt requests a specified user input in a multi-turn dialog; 
 receiving, from the client device via the multi-turn dialog, the specified user input; 
 generating a filtered list of candidate products based at least upon the specified user input; and 
 providing the filtered list of candidate products. 
 
 
     
     
       17. The system of  claim 16 , wherein the user prompt comprises a statement prompt denoting the mismatch between the list of candidate products and the non-image input. 
     
     
       18. The system of  claim 16 , wherein the user prompt comprises at least one of:
 a selectable candidate product image; or 
 a question requesting additional natural language input. 
 
     
     
       19. The system of  claim 16 , wherein the input query image comprises one of:
 a photograph; 
 a video frame; 
 a sketch; or 
 a diagram. 
 
     
     
       20. The system of  claim 16 , wherein the user prompt is generated using machine learning to formulate a question directed to understanding user intent.