Generating training data from scenario driven simulated users

Abstract

A system that generates relevant and vetted training data using intelligent simulated users and evaluation of conversation data. A simulated user and an automated agent engage in a conversation to generate conversation and/or interaction data. The simulated user is guided by scenarios which are generated based on one or more controls to be followed by the automated agent. Using a simulated user driven by control-derived scenarios ensures the ensuing conversation data is relevant to the desired scope of operation for the automated agent. The conversation data is evaluated based on the controls to confirm the automated agent actions and responses followed the controls properly. Evaluating the conversation data based on the controls ensures that conversation data associated with properly followed controls is used as subsequent training data.

1 . A method for generating training data using a simulated user, comprising:
 generating one or more scenarios by a first application on a first server based on a control;   providing a simulated user based on the scenario, the simulated user provided by a simulated user application;   accessing an example of an interaction between an automated agent and the simulated users by the first application, wherein each example is associated with an action by the automated agent, wherein the action is associated with the control, wherein the example including a subset of the interaction; and   selecting the example as training data for a subsequent learning process based on whether the automated agent action in the example is validated to be proper based on the control.   
     
     
         2 . The method of  claim 1 , wherein a scenario includes one or more parameters set to test whether the automated agent follows the control in response to a simulated user request. 
     
     
         3 . The method of  claim 1 , wherein a scenario is generated by a large language model in response to a prompt provided to the large language model, the prompt including the control, the role of the automated agent, and a prompt request to generate one or more scenarios based on the control and role. 
     
     
         4 . The method of  claim 1 , further comprising evaluating whether the automated agent followed the control when responding to a simulated user request, the simulated user request generated based on the scenario. 
     
     
         5 . The method of  claim 1 , wherein evaluating includes processing interaction data and the control by a machine learning model. 
     
     
         6 . The method of  claim 5 , wherein the machine learning model includes a large language model. 
     
     
         7 . The method of  claim 1 , wherein the automated agent action is evaluated during the conversation with the simulated user. 
     
     
         8 . The method of  claim 1 , wherein the example is stored as training data in a validated pool. 
     
     
         9 . A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to generate training data using a simulated user, the method comprising:
 generating one or more scenarios by a first application on a first server based on a control;   providing a simulated user based on the scenario, the simulated user provided by a simulated user application;   accessing an example of an interaction between an automated agent and the simulated users by the first application, wherein each example is associated with an action by the automated agent, wherein the action is associated with the control, wherein the example including a subset of the interaction; and   selecting the example as training data for a subsequent learning process based on whether the automated agent action in the example is validated to be proper based on the control.   
     
     
         10 . The non-transitory computer readable storage medium of  claim 9 , wherein a scenario includes one or more parameters set to test whether the automated agent follows the control in response to a simulated user request. 
     
     
         11 . The non-transitory computer readable storage medium of  claim 9 , wherein a scenario is generated by a large language model in response to a prompt provided to the large language model, the prompt including the control, the role of the automated agent, and a prompt request to generate one or more scenarios based on the control and role. 
     
     
         12 . The non-transitory computer readable storage medium of  claim 9 , further comprising evaluating whether the automated agent followed the control when responding to a simulated user request, the simulated user request generated based on the scenario. 
     
     
         13 . The non-transitory computer readable storage medium of  claim 9 , wherein evaluating includes processing interaction data and the control by a machine learning model. 
     
     
         14 . The non-transitory computer readable storage medium of  claim 13 , wherein the machine learning model includes a large language model. 
     
     
         15 . The non-transitory computer readable storage medium of  claim 9 , wherein the automated agent action is evaluated during the conversation with the simulated user. 
     
     
         16 . The non-transitory computer readable storage medium of  claim 9 , wherein the example is stored as training data in a validated pool. 
     
     
         17 . A system for generating training data using a simulated user, comprising:
 one or more servers, wherein each server includes a memory and a processor; and   one or more modules stored in the memory and executed by at least one of the one or more processors to generate one or more scenarios by a first application on a first server based on a control, provide a simulated user based on the scenario, the simulated user provided by a simulated user application, access an example of an interaction between an automated agent and the simulated users by the first application, wherein each example is associated with an action by the automated agent, wherein the action is associated with the control, wherein the example including a subset of the interaction, and select the example as training data for a subsequent learning process based on whether the automated agent action in the example is validated to be proper based on the control.   
     
     
         18 . The system of  claim 17 , wherein a scenario includes one or more parameters set to test whether the automated agent follows the control in response to a simulated user request. 
     
     
         19 . The system of  claim 17 , wherein a scenario is generated by a large language model in response to a prompt provided to the large language model, the prompt including the control, the role of the automated agent, and a prompt request to generate one or more scenarios based on the control and role. 
     
     
         20 . The system of  claim 17 , further comprising evaluating whether the automated agent followed the control when responding to a simulated user request, the simulated user request generated based on the scenario.