Method and system of context window engineering for large language models fine-tuned for conversations

Abstract

Methods and systems for context window management of large language models are disclosed. An example method includes: receiving a message in a user language; preprocessing the message; setting a static context based on a topic of the message for a text sequence including the message, if the topic is new or different from a topic of a preceding message; attaching the static context to a context window; attaching a long term dynamic context to the context window based on the message and one or more previous messages in the text sequence; attaching a short term dynamic context to the context window based on the message; providing the context window to a language model server; receiving a database access command based on the context window from the language model server; providing the database access command to a database; and receiving a result response to the database access command from the database.

What is claimed is: 
     
         1 . A method comprising:
 receiving a message in a user language;   preprocessing the message;   setting a static context based on a topic of the message for a text sequence including the message, if the topic is new or different from a topic of an immediately preceding message;   attaching the static context to a context window;   attaching a dynamic context of long term to the context window based on the message and one or more previous messages, if any, in the text sequence;   attaching a dynamic context of short term to the context window based on the message;   providing the context window to a language model server;   receiving a database access command based on the context window from the language model server;   providing the database access command to a database; and   receiving a result response to the database access command from the database.   
     
     
         2 . The method of  claim 1 , further comprising:
 providing the result to the language model server;   receiving the result in the user language from the language model server; and   providing the response in the user language.   
     
     
         3 . The method of  claim 1 , splitting the context window into:
 the static context;   the dynamic context of long term; and   the dynamic context of short term.   
     
     
         4 . The method of  claim 1 , wherein said attaching the dynamic context comprises providing relevant information to the message. 
     
     
         5 . The method of  claim 1 , further comprising:
 storing the one or more contexts in a queue; and   processing a next message using the message associated with the one or more contexts.   
     
     
         6 . The method of  claim 1 , further comprising loading a static context in memory based on one or more topics of the message. 
     
     
         7 . The method of  claim 6 , wherein the static context is based on at least one of multiple topics including an order, sales or appointment. 
     
     
         8 . The method of  claim 6 , wherein said attaching the dynamic context of long term comprises providing the dynamic context of long term over a period of the text sequence including the message on the one or more topics, based on one or more contents of the message and one or more previous messages, and
 wherein the message is a follow-up message of the previous message.   
     
     
         9 . The method of  claim 1 , wherein the dynamic context of short term is specific to the message,
 wherein the dynamic context of short term expires after receiving the result response.   
     
     
         10 . The method of  claim 1 , further comprising deleting one or more messages in the dynamic context long term on a first-in-first-out (FIFO) basis, when a token count in the context window becomes equal to a maximum number of tokens for the context window. 
     
     
         11 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:
 receive a message in a user language;   preprocessing the message;   set a static context based on a topic of the message for a text sequence including the message, if the topic is new or different from a topic of an immediately preceding message;   attach the static context to a context window;   attach a dynamic context of long term to the context window based on the message and one or more previous messages, if any, in the text sequence;   attach a dynamic context of short term to the context window based on the message;   provide the context window to a language model server;   receive a database access command based on the context window from the language model server;   provide the database access command to a database; and   receive a result response to the database access command from the database.   
     
     
         12 . The computer-readable storage medium of  claim 11 , wherein the instructions further configure the computer to:
 provide the result to the language model server;   receive the result in the user language from the language model server; and   provide the response in the user language.   
     
     
         13 . The computer-readable storage medium of  claim 11 , wherein said attach the dynamic context comprises provide relevant information to the message. 
     
     
         14 . The computer-readable storage medium of  claim 11 , wherein the instructions further configure the computer to:
 store the one or more contexts in a queue; and   process a next message using the message associated with the one or more contexts.   
     
     
         15 . The computer-readable storage medium of  claim 11 , wherein the instructions further configure the computer to load a static context in memory based on one or more topics of the message. 
     
     
         16 . The computer-readable storage medium of  claim 15 , wherein the static context is based on at least one of multiple topics including at least one of an order, sales or appointment. 
     
     
         17 . The computer-readable storage medium of  claim 15 , wherein said attach the dynamic context of long term comprises provide the dynamic context of long term over a period of the text sequence including the message on the one or more topics, based on one or more contents of the message and one or more previous messages, and
 wherein the message is a follow-up message of the previous message.   
     
     
         18 . The computer-readable storage medium of  claim 11 , wherein the dynamic context of short term is specific to the message, and
 wherein the dynamic context of short term expires after receiving the result response.   
     
     
         19 . The computer-readable storage medium of  claim 11 , wherein the instructions further configure the computer to delete one or more messages in the dynamic context long term on a FIFO basis, when a token count in the context window becomes equal to a maximum number of tokens for the context window. 
     
     
         20 . A computing apparatus comprising:
 a processor; and   a memory storing instructions that, when executed by the processor, configure the apparatus to:   receive a message in a user language;   preprocess the message;   set a static context based on a topic of the message for a text sequence including the message, if the topic is new or different from a topic of an immediately preceding message;   attach the static context to a context window;   attach a dynamic context of long term to the context window based on the message and one or more previous messages, if any, in the text sequence;   attach a dynamic context of short term to the context window based on the message;   provide the context window to a language model server;   receive a database access command based on the context window from the language model server;   provide the database access command to a database; and   receive a result response to the database access command from the database.   
     
     
         21 . The computing apparatus of  claim 20 , wherein the instructions further configure the apparatus to:
 provide the result to the language model server;   receive the result in the user language from the language model server; and   provide the response in the user language.   
     
     
         22 . The computing apparatus of  claim 20 , wherein said attach the dynamic context comprises provide relevant information to the message. 
     
     
         23 . The computing apparatus of  claim 20 , wherein the instructions further configure the apparatus to:
 store the one or more contexts in a queue; and   process a next message using the message associated with the one or more contexts.   
     
     
         24 . The computing apparatus of  claim 20 , wherein the instructions further configure the apparatus to load a static context in memory based on one or more topics of the message. 
     
     
         25 . The computing apparatus of  claim 24 , wherein the static context is based on at least one of multiple topics including at least one of an order, sales or appointment. 
     
     
         26 . The computing apparatus of  claim 24 , wherein said attach the dynamic context of long term comprises provide the dynamic context of long term over a period of the text sequence including the message on the one or more topics, based on one or more contents of the message and one or more previous messages, and
 wherein the message is a follow-up message of the previous message.   
     
     
         27 . The computing apparatus of  claim 20 , wherein the dynamic context of short term is specific to the message, and
 wherein the dynamic context of short term expires after receiving the result response.   
     
     
         28 . The computing apparatus of  claim 20 , wherein the instructions further configure the apparatus to delete one or more messages in the dynamic context long term on a FIFO basis, when a token count in the context window becomes equal to a maximum number of tokens for the context window.