Vehicle dispatching method and system

Abstract

Vehicle dispatch system includes upper stage unit, lower stage unit and interface communication unit. The upper stage unit, configured to generate vehicle schedules, is communicatively connected to the interface communication unit. The lower stage unit, communicatively connected to the upper stage unit and the interface communication unit, has two storage units and a control unit. The first storage unit stores in a state representation multiple possible states having multiple possible actions. The control unit, which receives the schedule as a state representation, is configured to simulate states during an episode by selecting a state action and determining a reward value. The second storage unit stores the reward value and has a policy linked to one possible action for each state. The interface communication unit, operable to receive and transmit vehicle communications, is configured to access the policy and its associated action and communicate the action to a vehicle.

What is claimed is: 
     
         1 . A vehicle dispatch system comprising:
 an interface communication unit;   an upper stage unit communicatively connected to the interface communication unit and configured to generate, in a state representation, a schedule for a plurality of vehicles in a dynamic work area; and   a lower stage unit communicatively connected to the upper stage unit and the interface communication unit, and having:
 a first storage unit for storing a plurality of possible states in the state representation, each possible state having a plurality of possible actions; 
 a control unit that receives the schedule, the control unit comprising a processor configured to:
 simulate the states during an episode by selecting an action of at least one of the states to produce a result based on proximity to optimum performance of the plurality of vehicles; 
 determine the reward value for the selected action; and 
 propagate the reward value back through simulation of the states with reference to time between the states; and 
 
 a second storage unit, the second storage unit storing the reward value for each of the possible actions and storing a policy linked to the one of the possible actions for each of the states; 
   wherein the interface communication unit is configured to access the policy and the action linked to the policy, the interface communication unit comprising:
 a receiver operable to receive communications from the vehicles; and 
 a transmitter operable to transmit communications to the vehicles to communicate the action to one or more of the vehicles in the dynamic work area to cause the one or more of the vehicles to perform in accordance with the action. 
   
     
     
         2 . The vehicle dispatch system of  claim 1 , wherein the control unit is configured to simulate the states continuously to maximize the reward value of the actions in each state. 
     
     
         3 . The vehicle dispatch system of  claim 1 , wherein the policy is linked to a maximum policy value for an action for each state and the maximum policy value is determined using an elapsed time in the episode. 
     
     
         4 . The vehicle dispatch system of  claim 3 , wherein the maximum policy value is further determined using a subsequent state with a subsequence action in the episode. 
     
     
         5 . The vehicle dispatch system of  claim 3 , wherein the action within each state that results in the maximum possible policy value becomes the policy for that state. 
     
     
         6 . The vehicle dispatch system of  claim 1 , wherein the upper stage unit is configured with linear programming that has an optimization function, an environmental constraint and a vehicle constraint as inputs to generate the schedule. 
     
     
         7 . The vehicle dispatch system of  claim 1 , wherein the policy and reward value are stored in a Q table. 
     
     
         8 . The vehicle dispatch system of  claim 1 , wherein the control unit is configured to continuously simulate the states during the episode to continuously determine the reward value for the selected action. 
     
     
         9 . A non-transitory computer-readable medium comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:
 generating, in a state representation, a schedule for a plurality of vehicles in a dynamic work area;   storing a plurality of possible states in the state representation, each possible state having a plurality of possible actions;   simulating the states during an episode by selecting an action of at least one of the states to produce a result based on proximity to optimum performance of the plurality of vehicles;   determining a reward value for the selected action;   propagating the reward value back through simulation of the states with reference to time between the states;   storing the reward value for each of the possible actions;   storing a policy linked to one of the possible actions for each of the states;   receiving communications from the vehicles; and   transmitting communications to the vehicles to communicate the actions to one or more of the vehicles in the dynamic work area to cause the one or more of the vehicles to perform in accordance with the action.   
     
     
         10 . The non-transitory computer-readable medium of  claim 9 , wherein simulating the states during an episode comprises simulating the states continuously to maximize the reward value of the actions in each state. 
     
     
         11 . The non-transitory computer-readable medium of  claim 9 , further comprising determining a maximum policy value using an elapsed time in the episode;
 wherein storing the policy comprises linking the policy to a maximum policy value for an action for each state.   
     
     
         12 . The non-transitory computer-readable medium of  claim 11 , wherein determining the maximum policy value further comprises using a subsequent state with a subsequent action in the episode. 
     
     
         13 . The non-transitory computer-readable medium of  claim 11 , storing the policy comprises storing the action within each state that results in the maximum possible policy value as the policy for that state. 
     
     
         14 . The non-transitory computer-readable medium of  claim 9 , wherein generating the schedule comprises executing linear programming using an optimization function, an environmental constraint, and a vehicle constraint as inputs. 
     
     
         15 . The non-transitory computer-readable medium of  claim 9 , wherein:
 storing the reward value for each of the possible actions comprises storing the reward value for each of the possible states in a Q table; and   storing the policy comprises storing the policy in the Q table.   
     
     
         16 . The non-transitory computer-readable medium of  claim 9 , simulating the states during an episode comprises continuously simulating the states during the episode to continuously determine the reward value for the selected action.