AI Gateway - GenAI Platform Services

Abstract

AI gateways are provided. An AI service request for an AI model may be received by an AI gateway from a client. The AI service request may be routed to an AI model deployment, where routing the AI service request includes selecting the AI model deployment from AI model deployments based on a quality of service.

What is claimed is: 
     
         1 . A method comprising:
 receiving an AI service request for an AI model from a client; and   routing the AI service request to an AI model deployment, wherein the routing includes selecting the AI model deployment from a plurality of AI model deployments based on a quality of service.   
     
     
         2 . The method of  claim 1 , wherein to maintain the quality of service, the AI model deployment is selected based on a token refill rate of the AI model deployments. 
     
     
         3 . The method of  claim 2 , wherein the AI model deployment is selected at random from the AI model deployments, and a chance of selecting each respective one of the AI model deployments is proportional to the token refill rate of the respective one of the AI model deployments. 
     
     
         4 . The method of  claim 2 , wherein the selecting comprises excluding the AI model deployments having no remaining tokens in a current interval from being selected. 
     
     
         5 . The method of  claim 1 , wherein to maintain the quality of service, the AI model deployment is selected over one or more of the AI model deployments because the AI model deployment is assigned provisioned throughput units and the one or more of the AI model deployments are Pay as You Go (PGO). 
     
     
         6 . The method of  claim 5  further comprising routing a second AI service request to one of the AI model deployments that is Pay as You Go (PGO) in response to a determination that the AI model deployment is unavailable. 
     
     
         7 . The method of  claim 1 , wherein the selecting comprises selecting the AI model deployment from the AI model deployments based on the quality of service assigned on a per identity basis to the AI model deployments. 
     
     
         8 . A computer readable storage medium comprising computer executable instructions, the computer executable instructions executable by a processor, the computer executable instructions comprising:
 instructions executable by the processor to receive an AI service request for an AI model from a client; and   instructions executable by the processor to route the AI service request to an AI model deployment, wherein to route the AI service request, the AI model deployment is selected from a plurality of AI model deployments based on a quality of service.   
     
     
         9 . The computer readable storage medium of  claim 8  further comprising instructions executable by the processor to extract and normalize information about a plurality of AI service requests and to deliver that normalized information to a metadata bus. 
     
     
         10 . The computer readable storage medium of  claim 8  further comprising instructions executable by the processor to send a request that includes predefined test data to a set of the AI model deployments, and to measure a completion time of the request. 
     
     
         11 . The computer readable storage medium of  claim 10  further comprising instructions executable by the processor to stop routing any AI service requests to an underperforming AI model deployment in the set of the AI model deployments, wherein the underperforming AI model deployment is identified by the completion time of the AI service request exceeding a threshold level. 
     
     
         12 . The computer readable storage medium of  claim 8 , wherein access to the AI model deployments is controlled on a per identity basis. 
     
     
         13 . The computer readable storage medium of  claim 8  further comprising instructions executable by the processor to restrict access to the AI model deployments per entity by enforcing a “valid no later than” epoch time for an entire identity. 
     
     
         14 . The computer readable storage medium of  claim 8 , wherein access to the AI model deployments is controlled on a per identity basis, and wherein an identity corresponds to a bearer token included in the AI service request. 
     
     
         15 . An AI gateway comprising:
 a processor; and   a request handler executable by the processor to receive an AI service request for an AI model from a client and to proxy the AI service request to an AI model deployment, wherein the request handler is executable by the processor to select the AI model deployment from a plurality of AI model deployments based on a quality of service.   
     
     
         16 . The AI gateway of  claim 15  further comprising a token bucket refiller executable by the processor to regularly refill a plurality of token buckets, the token buckets assigned to the AI model deployments, wherein the AI model deployment is selected from the AI model deployments based on the token buckets. 
     
     
         17 . The AI gateway of  claim 15  further comprising a PTU refiller executable by the processor to regularly update a prepaid tokens setting for any of the AI model deployments that are PTU deployments. 
     
     
         18 . The AI gateway of  claim 17 , wherein the request handler is executable by the processor to proxy the AI service request to a preferred target before any other of the AI model deployments, and wherein the preferred target includes any of the AI model deployments that are PTU deployments. 
     
     
         19 . The AI gateway of  claim 15 , wherein the request handler is executable by the processor to authorize access to AI services based on an identity indicated in the AI service request. 
     
     
         20 . The AI gateway of  claim 15 , wherein the request handler is executable by the processor to enable resource sharing across a plurality of classes of service, wherein the quality of service depends on the classes of services.