Obtaining and utilizing feedback for agent-assist systems

Abstract

Techniques for agent-assist systems to provide context-aware, subdocument-granularity recommended answers to agents that are attempting to answer queries of users. The agent-assist system may obtain collections of documents that include information for responding to queries, and analyze those documents to identify subdocuments that are associated with different semantics or meanings. Subsequently, any queries received can be analyzed to identify their semantics, and relevant subdocuments can be identified as having similar semantics. When the agent-assist system presents the agent with the relevant documents, it may highlight or otherwise indicate the relevant subdocument within the document for quick identification by the agent. Further, the agent-assist system may collect feedback from the agent and/or user to determine a relevancy of the recommended answers. The agent-assist system can use the feedback to improve the quality of the recommended answers provided to the agents.

What is claimed is: 
     
         1 . A method performed at least partly by an agent-assist system, the method comprising:
 obtaining a plurality of documents at the agent-assist system, the plurality of documents including subdocuments;   converting subdocuments into mathematical representations of semantic meanings of the subdocuments;   storing the mathematical representations of the semantic meanings of the subdocuments in a knowledge base of the agent-assist system;   establishing a communication session between a user device and an agent device, wherein the agent device presents supplemental information to an agent engaged in the communication session for responding to a user associated with the user device;   receiving, from the user device, first input from the user engaged in the communication session;   identifying, from the first input, a query that the user has for the agent to answer;   retrieving, using a model trained on historical interactions, a recommended answer from the knowledge base of the agent-assist system, wherein the retrieving the recommended answer for responding to the query includes:
 identifying a prior query in the communication session that is related to the query; 
 using a conversation-context-aware semantic search algorithm, ranking candidate answers from the model based on contextual similarity between the prior query and the query, wherein the candidate answers have corresponding mathematical representations determined to be mathematically similar to the prior query and the query; 
   causing presentation of the recommended answer on a display associated with the agent device;   receiving, from the agent device, second input that includes an answer provided by the agent to the query;   determining, using automated data tracking techniques, implicit feedback indicating a relevancy of the recommended answer for responding to the query, the determining the implicit feedback includes determining (i) a duration the recommended answer remains in a viewport of the agent device, (ii) a modification made by the agent to the recommended answer before sending a response, and (iii) sentiment analysis of user responses following answer presentation; and   based at least in part on the implicit feedback, adjusting a confidence value associated with the recommended answer being used for responding to the query, wherein the confidence value indicates a likelihood that the recommended answer is relevant to use for responding to the query, the adjusting the confidence value including decreasing the confidence value, wherein adjusting the confidence value includes dynamically updating the model to refine future recommendations and reducing presentation frequency for low-confidence answers.   
     
     
         2 . The method of  claim 1 , wherein determining the implicit feedback includes:
 determining that the recommended answer was presented at least partially in the viewport of the display for a period of time; and   determining whether the period of time was greater than or equal to a threshold period of time; and   wherein adjusting the confidence value comprises:   in response to determining that the period of time being greater than or equal to the threshold period of time, increasing the confidence value; or   in response to determining that the period of time being less than the threshold period of time, decreasing the confidence value.   
     
     
         3 . The method of  claim 1 , wherein determining the implicit feedback includes:
 obtaining first text data representing the recommended answer;   obtaining second text data representing the answer;   determining a similarity score indicating a measure of similarity between the first text data and the second text data; and   determining whether the similarity score is greater than or equal to a threshold similarity score; and   wherein adjusting the confidence value comprises:   in response to determining that the similarity score is greater than or equal to the threshold similarity score, increasing the confidence value; or   in response to determining that the similarity score is less than the threshold similarity score, decreasing the confidence value.   
     
     
         4 . The method of  claim 3 , further comprising:
 determining that the second input was received from the agent device within a threshold amount of conversation turns of the communication session; and   determining to adjust the confidence value based at least in part on the second input being received within the threshold amount of conversation turns.   
     
     
         5 . The method of  claim 1 , wherein determining the implicit feedback includes:
 identifying, using a sentiment-analysis model of the agent-assist system, a first sentiment of the user prior to receiving the second input that includes the answer provided by the agent;   identifying, using the sentiment-analysis model, a second sentiment of the user subsequent to receiving the second input that includes the answer provided by the agent; and   determining a sentiment change for the user using the first sentiment and the second sentiment, and   wherein adjusting the confidence value comprises:   in response to determining that the sentiment change indicates the sentiment of the user has increased, increasing the confidence value; or   in response to determining that the sentiment change indicates the sentiment of the user has decreased, decreasing the confidence value.   
     
     
         6 . The method of  claim 1 , further comprising:
 determining explicit feedback indicating relevancy of the recommended answer for responding to the query; and   adjusting the confidence value based at least in part on the explicit feedback.   
     
     
         7 . The method of  claim 1 , further comprising:
 receiving third input from the user of the user device, the third input including a second query for the agent to answer;   determining that the knowledge base does not include recommended answers for the second query; and   outputting, to an administrative account associated with the knowledge base, an indication that the knowledge base does not include recommended answers for the second query.   
     
     
         8 . A system comprising:
 one or more processors; and   one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:   obtaining a plurality of documents at an agent-assist system, the plurality of documents including subdocuments;   converting subdocuments into mathematical representations of semantic meanings of the subdocuments;   storing the mathematical representations of the semantic meanings of the subdocuments in a knowledge base of the agent-assist system;   establishing a communication session between a user device and an agent device, wherein the agent device presents supplemental information to an agent engaged in the communication session for responding to a user associated with the user device;   receiving, from the user device, first input from the user engaged in the communication session;   identifying, from the first input, a query that the user has for the agent to answer;   retrieving, using a model trained on historical interactions, a recommended answer from the knowledge base of the agent-assist system, wherein the retrieving the recommended answer for responding to the query includes:
 identifying a prior query in the communication session that is related to the query; 
 using a conversation-context-aware semantic search algorithm, ranking candidate answers from the model based on contextual similarity between the prior query and the query, wherein the candidate answers have corresponding mathematical representations determined to be mathematically similar to the prior query and the query; 
   causing presentation of the recommended answer on a display associated with the agent device;   receiving, from the agent device, second input that includes an answer provided by the agent to the query;   determining, using automated data tracking techniques, implicit feedback indicating a relevancy of the recommended answer for responding to the query, the determining the implicit feedback including determining(i) a duration the recommended answer remains in a viewport of the agent device, (ii) a modification made by the agent to the recommended answer before sending a response, and (iii) that the second input was received from the agent device within a threshold amount of conversation turns of the communication session from when the recommended answer was presented; and   based at least in part on the implicit feedback and on the second input being received within the threshold amount of conversation turns from when the recommended answer was presented, adjusting a confidence value associated with the recommended answer being used for responding to the query, wherein the confidence value indicates a likelihood that the recommended answer is relevant to use for responding to the query, wherein adjusting the confidence value includes dynamically updating the model to refine future recommendations and reducing presentation frequency for low-confidence answers.   
     
     
         9 . The system of  claim 8 , wherein determining the implicit feedback includes:
 determining that the recommended answer was presented at least partially in the viewport of the display for a period of time; and   determining whether the period of time was greater than or equal to a threshold period of time; and   wherein adjusting the confidence value comprises:   in response to determining that the period of time being greater than or equal to the threshold period of time, increasing the confidence value; or   in response to determining that the period of time being less than the threshold period of time, decreasing the confidence value.   
     
     
         10 . The system of  claim 8 , wherein determining the implicit feedback includes:
 obtaining first text data representing the recommended answer;   obtaining second text data representing the answer;   determining a similarity score indicating a measure of similarity between the first text data and the second text data; and   determining whether the similarity score is greater than or equal to a threshold similarity score; and   wherein adjusting the confidence value comprises:   in response to determining that the similarity score is greater than or equal to the threshold similarity score, increasing the confidence value; or   in response to determining that the similarity score is less than the threshold similarity score, decreasing the confidence value.   
     
     
         11 . The system of  claim 8 , wherein determining the implicit feedback includes:
 identifying, using a sentiment-analysis model of the agent-assist system, a first sentiment of the user prior to receiving the second input that includes the answer provided by the agent;   identifying, using the sentiment-analysis model, a second sentiment of the user subsequent to receiving the second input that includes the answer provided by the agent; and   determining a sentiment change for the user using the first sentiment and the second sentiment, and   wherein adjusting the confidence value comprises:   in response to determining that the sentiment change indicates the sentiment of the user has increased, increasing the confidence value; or   in response to determining that the sentiment change indicates the sentiment of the user has decreased, decreasing the confidence value.   
     
     
         12 . The system of  claim 8 , wherein:
 determining the implicit feedback includes determining that the recommended answer has been presented to the agent more than a threshold number of times during a period of time; and   adjusting the confidence value includes decreasing the confidence value based at least in part on the recommended answer having been presented to the agent more than the threshold number of times during the period of time.   
     
     
         13 . The system of  claim 8 , the operations further comprising:
 determining explicit feedback indicating relevancy of the recommended answer for responding to the query; and   adjusting the confidence value based at least in part on the explicit feedback.   
     
     
         14 . The system of  claim 8 , the operations further comprising:
 receiving third input from the user of the user device, the third input including a second query for the agent to answer;   determining that the knowledge base does not include recommended answers for the second query; and   outputting, to an administrative account associated with the knowledge base, an indication that the knowledge base does not include recommended answers for the second query.   
     
     
         15 . A method comprising:
 obtaining a plurality of documents at an agent-assist system, the plurality of documents including subdocuments;   converting subdocuments into mathematical representations of semantic meanings of the subdocuments;   storing the mathematical representations of the semantic meanings of the subdocuments in a knowledge base of the agent-assist system;   establishing, partly by an agent device, a communication session with a user device, the communication session facilitating a conversation between a user of the user device and an agent associated with the agent device;   presenting, on a display of the agent device, one or more user interfaces (UIs) that represents at least the conversation between the agent and the user;   receiving first input from the user device, the first input representing a query from the user for the agent to answer;   retrieving, using a model trained on historical interactions, a recommended answer from the knowledge base of the agent-assist system, wherein the retrieving the recommended answer for responding to the query includes:
 identifying a prior query in the communication session that is related to the query; 
 using a conversation-context-aware semantic search algorithm, ranking candidate answers from the model based on contextual similarity between the prior query and the query, wherein the candidate answers have corresponding mathematical representations determined to be mathematically similar to the prior query and the query; 
   presenting the recommended answer in the one or more UIs on the display;   identifying second input at the agent device, the second input representing an answer provided by the agent to respond to the query;   determining implicit feedback indicating a relevancy of the recommended answer for responding to the query, the determining the implicit feedback including:
 determining that the recommended answer was presented at least partially in a viewport of the display for a period of time and determining whether the period of time was greater than or equal to a threshold period of time; 
 determining a duration the recommended answer remains in a viewport of the agent device; 
 determining a modification made by the agent to the recommended answer before sending a response; and 
 determining a sentiment analysis of user responses following answer presentation; 
   providing the implicit feedback to the agent-assist system; and   adjusting, by the agent-assist system, a confidence value associated with the recommended answer being used for responding to the query, the adjusting comprising:
 in response to determining that the period of time being greater than or equal to the threshold period of time, increasing the confidence value; or 
 in response to determining that the period of time being less than the threshold period of time, decreasing the confidence value, 
 wherein adjusting the confidence value includes dynamically updating the model to refine future recommendations and reducing presentation frequency for low-confidence answers. 
   
     
     
         16 . The method of  claim 15 , wherein determining the implicit feedback includes:
 obtaining first text data representing the recommended answer;   obtaining second text data representing the answer;   determining a similarity score indicating a measure of similarity between the first text data and the second text data; and   determining whether the similarity score is greater than or equal to a threshold similarity score,   further comprising adjusting, by the agent-assist system, a confidence value associated with the recommended answer being used for responding to the query, the adjusting comprising:   in response to determining that the similarity score is greater than or equal to the threshold similarity score, increasing the confidence value; or   in response to determining that the similarity score is less than the threshold similarity score, decreasing the confidence value.   
     
     
         17 . The method of  claim 15 , wherein determining the implicit feedback includes:
 identifying, using a sentiment-analysis model of the agent-assist system, a first sentiment of the user prior to receiving the second input that includes the answer provided by the agent;   identifying, using the sentiment-analysis model, a second sentiment of the user subsequent to receiving the second input that includes the answer provided by the agent; and   determining a sentiment change for the user using the first sentiment and the second sentiment,   further comprising adjusting, by the agent-assist system, a confidence value associated with the recommended answer being used for responding to the query, the adjusting comprising:   in response to determining that the sentiment change indicates the sentiment of the user has increased, increasing the confidence value; or   in response to determining that the sentiment change indicates the sentiment of the user has decreased, decreasing the confidence value.