Distributed execution budget management system

Abstract

Systems and methods are provided for managing execution budgets for computing tasks in distributed computing systems. Execution budgets may include a budgeted number of operation retries that may be performed by the distributed computing system in response to failure of an operation. Execution budgets may also or alternatively include a budgeted amount of time in which the distributed computing system may perform the operations of a computing task. When a distributed computing system exhausts the execution budget allotted for a computing task, then further execution of the computing task or portions thereof may be terminated.

What is claimed is: 
     
       1. A system for managing distributed execution of computing services, comprising a plurality of services implemented by one or more computing devices;
 wherein a first service of the plurality of services is configured to:
 receive a first request to execute a first function, wherein the first request is associated with first retry budget data representing a first maximum number of communication retry attempts permitted in response to inter service communication failures, and wherein execution of the first function comprises sending, to a second service of the plurality of services, a second request to execute a second function; 
 determine that a first attempt to communicate with the second service has failed; 
 determine that the first maximum number of communication retry attempts has not been exhausted; 
 generate second retry budget data based at least partly on reducing the first maximum number of communication retry attempts, wherein the second retry budget data represents a second maximum number of communication retry attempts; 
 determine that a second attempt to retry communicating with the second service has been successful; and 
 send the second request to the second service, wherein the second request is associated with the second retry budget data, and wherein execution of the second function comprises sending a third request to a third service of the plurality of services; and 
 
 wherein the second service is configured to:
 determine that a third attempt to communicate with the third service has failed; 
 determine that the second maximum number of communication retry attempts has been exhausted; and 
 send an error notification to the first service. 
 
 
     
     
       2. The system of  claim 1 , wherein the first service comprises one of: a first virtual machine executing on a host computing device, a first application executing on the host computing device, or a first thread executing on the host computing device; and wherein the second service comprises one of a second virtual machine executing on a host computing device, a second application executing on the host computing device, or a second thread executing on the host computing device. 
     
     
       3. The system of  claim 1 , further comprising a communication network, wherein the first attempt to communicate with the second service comprises an attempt to establish a connection with a first computing device of the second service via the communication network, and wherein the first service is configured to determine that the second attempt to communicate with the second service is to be an attempt to establish the connection with a second computing device of the second service via the communication network based at least partly on a failure of the first attempt. 
     
     
       4. The system of  claim 1 , wherein the first request is further associated with first duration budget data representing a maximum duration of time for execution of the first function. 
     
     
       5. A computer-implemented method comprising:
 under control of a first computing service comprising one or more computing devices configured to execute specific instructions,
 receiving a first request to execute a first function of a plurality of functions, wherein the first request is associated with first execution budget data representing a first execution budget of retry attempts for performing one or more operations during execution of the first function, and wherein the first execution budget is different from a maximum permitted quantity of retry attempts associated with execution of the plurality of functions; 
 determining that execution of the first function comprises requesting a second computing service to execute a second function of the plurality of functions; 
 generating second execution budget data representing a second execution budget of retry attempts for performing one or more operations during execution of the second function, wherein the second execution budget data is generated based at least partly on the first execution budget data and a quantity of retry attempts performed during execution of the first function, and wherein the second execution budget is different from the maximum permitted quantity of retry attempts associated with execution of the plurality of functions; and 
 generating a second request to the second computing service to execute the second function, wherein the second request is associated with the second execution budget data. 
 
 
     
     
       6. The computer-implemented method of  claim 5 , further comprising updating the first execution budget data based at least partly on removing, from the first execution budget, a portion of the first execution budget consumed during at least a portion of the one or more operations performed during execution of the first function, wherein the second execution budget data is generated subsequent to updating the first execution budget data. 
     
     
       7. The computer-implemented method of  claim 5 , further comprising:
 determining that execution of the first function further comprises requesting a third computing service to execute a third function concurrently with execution of at least a portion of the second function by the second computing service; 
 generating third execution budget data representing a third execution budget for performing one or more operations during execution of the third function; and 
 allocating, among at least the second execution budget and the third execution budget, at least a portion of the first execution budget. 
 
     
     
       8. The computer-implemented method of  claim 5 , wherein generating the second execution budget data further comprises determining a maximum duration of time for execution of the second function. 
     
     
       9. The computer-implemented method of  claim 8 , further comprising:
 determining that execution of the first function further comprises requesting a third computing service to execute a third function concurrently with execution of at least a portion of the second function by the second computing service; and 
 generating third execution budget data representing a third execution budget for performing one or more operations during execution of the third function, wherein the third execution budget comprises a maximum duration of time specified by the first execution budget data less a duration of time consumed by the first computing service during a portion of the one or more operations performed during execution of the first function. 
 
     
     
       10. The computer-implemented method of  claim 5 , further comprising:
 determining that execution of a portion of the one or more operations during execution of the first function satisfies a criterion; and 
 generating updated first execution budget data by adding to the first execution budget. 
 
     
     
       11. The computer-implemented method of  claim 5 , further comprising:
 receiving, from the second computing service, remaining budget data representing a remainder of the second execution budget; and 
 generating third execution budget data representing a third execution budget for performing one or more operations during execution of a third function, wherein the third execution budget data is generated based at least partly on the remaining budget data. 
 
     
     
       12. A computing system comprising computer-readable memory and one or more computer processors, wherein the computing system is configured to at least:
 receive a first request to execute a first function of a plurality of functions, wherein the first request is associated with first execution budget data representing a first execution budget of retry attempts for performing one or more operations during execution of the first function, and wherein the first execution budget is different from a maximum permitted quantity of retry attempts associated with execution of the plurality of functions; 
 determine that execution of the first function comprises requesting a second computing system to execute a second function of the plurality of functions; 
 generate second execution budget data representing a second execution budget of retry attempts for performing one or more operations during execution of the second function, wherein the second execution budget data is generated based at least partly on the first execution budget data and at least a portion of the one or more operations performed during execution of the first function, and wherein the second execution budget is different from the maximum permitted quantity of retry attempts associated with execution of the plurality of functions; and 
 generate a second request to the second computing system to execute the second function, wherein the second request is associated with the second execution budget data. 
 
     
     
       13. The computing system of  claim 12 , further configured to update the first execution budget data based at least partly removing, from the first execution budget, a portion of the first execution budget consumed during at least a portion of the one or more operations performed during execution of the first function, wherein the second execution budget data is generated subsequent to updating the first execution budget data. 
     
     
       14. The computing system of  claim 12 , further configured to:
 determine that execution of the first function further comprises requesting a third computing system to execute a third function concurrently with execution of at least a portion of the second function by the second computing system; 
 generate third execution budget data representing a third execution budget for performing one or more operations during execution of the third function; and 
 allocate, among at least the second execution budget and the third execution budget, at least a portion of the first execution budget. 
 
     
     
       15. The computing system of  claim 12 , wherein the second execution budget data further comprises a maximum duration of time for execution of the second function. 
     
     
       16. The computing system of  claim 15 , further configured to:
 determine that execution of the first function further comprises requesting a third computing system to execute a third function concurrently with execution of at least a portion of the second function by the second computing system; and 
 generate third execution budget data representing a third execution budget for performing one or more operations during execution of the third function, wherein the third execution budget comprises a maximum duration of time specified by the first execution budget less a duration of time consumed by the computing system during the portion of the one or more operations performed during execution of the first function. 
 
     
     
       17. The computing system of  claim 12 , further configured to:
 determine that execution of the portion of the one or more operations during execution of the first function satisfies a criterion; and 
 generate updated first execution budget data by adding to the first execution budget. 
 
     
     
       18. The computing system of  claim 12 , further configured to:
 receive, from the second computing system, remaining budget data representing a remainder of the second execution budget; and 
 generate third execution budget data representing a third execution budget for performing one or more operations during execution of a third function, wherein the third execution budget data is generated based at least partly on the remaining budget data.