Multi-modal conversational agent platform

Abstract

A method includes receiving data characterizing an utterance of a query associated with a tenant; providing, to an automated speech recognition engine, the received data and a profile selected from a plurality of profiles based on the tenant, the profile configuring the automated speech recognition engine to process the received data; receiving, from the automated speech recognition engine, a text string characterizing the query; and processing, via an ensemble of natural language agents configured based on the tenant, the text string characterizing the query to determine a textual response to the query, the textual response including at least one word from a first lexicon associated with the tenant. Related systems, methods, apparatus, and computer readable mediums are also described.

1 . A method comprising:
 receiving, by a multitenant remote server including executable instances of natural language resources, data characterizing a query by a first user and associated with a first tenant, the multitenant remote server including a tenant portal enabling the first user configuration of tenant data;   deploying, responsive to the receiving, a first instance of an executable natural language resource configured to receive a text string characterizing the query and determine a textual response to the query;   providing, to an automated speech recognition engine via the multitenant remote server, the received data;   receiving, from the automated speech recognition engine, a text string characterizing the query; and   processing, via the first instance of the executable natural language agent ensemble, the text string characterizing the query to determine a textual response to the query, the textual response including at least one word from a lexicon associated with the first tenant.   
     
     
         2 . The method of  claim 1 , further comprising
 providing, to a text-to-speech synthesis engine, the textual response;   receiving, from the text-to-speech synthesis engine, a verbalized query response determined by the text-to-speech synthesis engine based on the textual response; and   providing the verbalized query response.   
     
     
         3 . The method of  claim 2 , wherein the text-to-speech synthesis engine is configured to receive the textual response, and to generate, in response to the receiving, the verbalized query response including audio data corresponding to the received textual response, the text-to-speech synthesis engine being selected from one or more inter-changeable speech processing engines included in the profile. 
     
     
         4 . The method of  claim 2 , further comprising:
 receiving, prior to receiving data characterizing the query, an input to a web site provided via a web browser configured on a first computing device, the input causing the web browser to be authenticated and registered at a second computing device coupled to the first computing device via a network.   
     
     
         5 . The method of  claim 4 , further comprising:
 receiving, by the second computing device, validation data associated with the first computing device, the validation data including audio and graphical rendering settings configured on with the first computing device;   generating, in response to confirming the validation data, an initial conversation prompt by the second computing device and providing the initial conversation prompt to the web site configured on the first computing device;   receiving, at an input device coupled to the first computing device and in response to providing the initial conversation prompt via the web site, the data characterizing an utterance of the query, the query associated with an item available via the web site;   transmitting the provided verbalized query response to the first computing device; and   providing the verbalized query response via an output device coupled to the first computing device.   
     
     
         6 . The method of  claim 1 , further comprising
 providing a first configuration of a graphical user interface on a first client device, the client device configured to receive the utterance from a user.   
     
     
         7 . The method of  claim 1 , wherein processing the text string characterizing the query further comprises:
 generating a sematic interpretation associated with the text string;   determining a first contextual sequence associated with text string based on one or more previously processed text strings;   generating a first response action based on the determined first contextual sequence; and   generating the textual response based on the generated first response action.   
     
     
         8 . The method of  claim 7 , wherein the semantic interpretation is generated using a first data structure representing the lexicon associated with the first tenant. 
     
     
         9 . The method of  claim 8 , wherein the first data structure is generated based on at least one of: a catalog of items associated with the first tenant and including a first item title and a first item description, one or more reviews associated with a first item, interactive user data associated with a first item, or a combination thereof. 
     
     
         10 . The method of  claim 9 , wherein generating the first data structure includes
 determining one or more attributes associated with a first item from the catalog of items;   determining one or more synonyms associated with the first item from the catalog of items;   determining one or more referring expressions associated with the first item from the catalog of items and/or the interactive user data associated with the first item;   generating the first data structure based on the determining steps, the first data structure including a name, one or more attributes, one or more synonyms, one or more referring expressions, and/or one or more dialogs corresponding to the first item.   
     
     
         11 . The method of  claim 8 , wherein the first data structure is used to train the at least one of a plurality of classification algorithms. 
     
     
         12 . The method of  claim 1 , further comprising:
 receiving second data characterizing an utterance of a second query associated with a second tenant;   providing, to a second automated speech recognition engine, the received second data;   receiving, from the automated speech recognition engine, a second text string characterizing the second query; and   processing, via a second instance of the natural language agent ensemble configured based on the second tenant, the second text string characterizing the second query to determine a second textual response to the second query, the second textual response including at least one word from a second lexicon associated with the second tenant.   
     
     
         13 . The method of  claim 1 , wherein the query includes a plurality of natural language words spoken by the first user and received by an input device of a first computing device, the query provided by the first user in regard to a first context associated with a first item provided by the first tenant. 
     
     
         14 . The method of  claim 1 , wherein the received data is provided to the automated speech recognition engine with a profile selected from a plurality of profiles based on the first tenant, the profile configuring the automated speech recognition engine to process the received data. 
     
     
         15 . The method of  claim 14 , wherein the profile includes one or more configuration settings associated with the first instance of the natural language agent ensemble configured on a server including a data processor, one or more configuration settings associated with the natural language agent ensemble configured on a first computing device, or one or more configuration settings specifying one or more speech processing engines configured on a server including a data processor. 
     
     
         16 . The method of  claim 1 , wherein the first tenant includes at least one of a retail entity, a service provider entity, a financial entity, a manufacturing entity, an entertainment entity, an information storage entity, and a data processing entity. 
     
     
         17 . The method of  claim 1 , wherein the automated speech recognition engine is configured to receive audio data corresponding to the query and to generate, in response to the receiving, the text string including textual data corresponding to the received audio data, the automatic speech recognition engine being selected from one or more inter-changeable speech processing engines. 
     
     
         18 . The method of  claim 1 , wherein the data characterizing the query associated with the first tenant is provided via a textual interaction modality or via a speech interaction modality. 
     
     
         19 . A system comprising:
 at least one data processor; and   memory storing instructions, which, when executed by the at least one data processor cause the at least one data processor to perform operations comprising:   receiving, by a multitenant remote server including executable instances of natural language resources, data characterizing a query by a first user and associated with a first tenant, the multitenant remote server including a tenant portal enabling the first user configuration of tenant data;   deploying, responsive to the receiving, a first instance of an executable natural language resource configured to receive a text string characterizing the query and determine a textual response to the query;   providing, to an automated speech recognition engine via the multitenant remote server, the received data;   receiving, from the automated speech recognition engine, a text string characterizing the query; and   processing, via the first instance of the executable natural language agent ensemble, the text string characterizing the query to determine a textual response to the query, the textual response including at least one word from a lexicon associated with the first tenant.   
     
     
         20 . The system of  claim 19 , the operations further comprising:
 providing, to a text-to-speech synthesis engine, the textual response;   receiving, from the text-to-speech synthesis engine, a verbalized query response determined by the text-to-speech synthesis engine based on the textual response; and   providing the verbalized query response.