Architecture for simulation clock-based simulation of distributed systems

Abstract

Systems and methods are provided for the deterministic simulation of distributed systems, such as vehicle-based processing systems. A distributed system may be represented as a plurality of subsystems or “nodelets” executing with a single process of a computing device during a simulation. The nodelets may communicate using in-process communication. A task scheduler can schedule the nodelets to execute separately in serially-occurring frames. A simulated clock may be used to mitigate the variability in timestamped data that may be caused by latency or jitter.

What is claimed is: 
     
       1. A system for deterministic simulation of distributed processing, the system comprising:
 a computer-readable memory; and 
 one or more processors in communication with the computer readable memory and configured to at least:
 load, into a portion of the computer-readable memory allocated to a single process, a computation graph comprising a plurality of executable nodelets,
 wherein the plurality of executable nodelets are configured to execute in a simulation mode in which the plurality of executable nodelets executes serially to process simulation data representing output of one or more sensors, and 
 wherein the plurality of executable nodelets are further configured to execute in a production mode in which two or more nodelets of the plurality of executable nodelets execute concurrently to process sensor data received from the one or more sensors; 
 
 set a simulated clock to a first time based at least partly on a first timestamp associated with a first input data item of the simulation data; 
 schedule execution of the plurality of executable nodelets in the simulation mode based on the simulated clock, wherein the simulated clock maintains the first time during execution of a first nodelet of the plurality of executable nodelets, and wherein the first nodelet uses the simulated clock to generate a second timestamp, representing the first time, for in-process data generated by the first nodelet; 
 establish a channel for in-process communication to a second nodelet of the plurality of executable nodelets; and 
 send the in-process data generated by the first nodelet to the second nodelet using the channel, wherein the channel copies the in-process data from a first location of the portion of the computer-readable memory allocated to the process to a second location of the portion of the computer-readable memory allocated to the process. 
 
 
     
     
       2. The system of  claim 1 , wherein the computation graph comprises at least a portion of a vehicle control system. 
     
     
       3. The system of  claim 1 , wherein the one or more sensors include at least one of: a LiDAR sensor, a RADAR sensor, an inertial sensor, or a camera. 
     
     
       4. The system of  claim 1 , wherein the one or more processors are further configured to at least:
 determine that a new in-process communication is to be sent using the channel; 
 determine, during the production mode, that a queue associated with the channel is full; and 
 overwrite an oldest in-process communication in the queue with the new in-process communication. 
 
     
     
       5. The system of  claim 1 , wherein the one or more processors are further configured to at least:
 determine that a new in-process communication is to be sent using the channel; 
 determine, during the simulation mode, that a queue associated with the channel is full; and 
 delay adding the new in-process communication to the queue until there is space in the queue for the new in-process communication. 
 
     
     
       6. The system of  claim 1 , wherein each nodelet of the plurality of executable nodelets is configured to receive in-process data using at least one channel of a plurality of channels of the computation graph. 
     
     
       7. A computer-implemented method comprising:
 under control of a computing system comprising a computer processor configured to execute specific instructions,
 loading a plurality of subsystems into a portion of computer-readable memory allocated to a single process, wherein the plurality of subsystems are configured to operate in a first operating mode in which the plurality of subsystems executes only serially to process simulated sensor data, and in a second operating mode in which two or more subsystems of the plurality of subsystems execute concurrently to process sensor data; 
 setting a simulated clock to a first time based at least partly on a first timestamp associated with the simulated sensor data; 
 scheduling execution of the plurality of subsystems in the first operating mode based on the simulated clock, wherein the simulated clock maintains the first time during execution of a first subsystem of the plurality of subsystems, and wherein the first subsystem uses the simulated clock to generate a second timestamp, representing a same time as the simulated clock, for in-process data generated by the first subsystem; 
 establishing a channel for communication to a second subsystem of the plurality of subsystems; and 
 sending the in-process data to the second subsystem using the channel, wherein the channel copies the in-process data from a first location of the portion of the computer-readable memory allocated to the process to a second location of the portion of the computer-readable memory allocated to the process. 
 
 
     
     
       8. The computer-implemented method of  claim 7 , further comprising:
 determining that a new in-process communication is to be sent using the channel; 
 determining, during the first operating mode, that a queue associated with the channel is full; and 
 delaying adding the new in-process communication to the queue until there is space in the queue for the new in-process communication. 
 
     
     
       9. The computer-implemented method of  claim 7 , further comprising:
 determining that a new in-process communication is to be sent using the channel; 
 determining, during the second operating mode, that a queue associated with the channel is full; and 
 overwriting an oldest in-process communication in the queue with the new in-process communication. 
 
     
     
       10. The computer-implemented method of  claim 7 , further comprising:
 loading an input data item from the simulated sensor data, wherein the input data item is associated with the first timestamp; and 
 determining the first time at which the simulated clock is to be set based on applying an increment to a second time represented by the first timestamp. 
 
     
     
       11. The computer-implemented method of  claim 10 , further comprising scheduling execution of the second subsystem based at least partly on a third time, wherein the simulated clock maintains the third time during execution of the second subsystem, and wherein the second subsystem uses the simulated clock to generate a third timestamp associated with the in-process data generated by the second subsystem. 
     
     
       12. A system comprising:
 a computer-readable memory; and 
 one or more processors in communication with the computer readable memory and configured to at least:
 load a plurality of subsystems into a portion of the computer-readable memory allocated to a single process, wherein the plurality of subsystems are configured to operate in a first operating mode in which the plurality of subsystems executes only serially to process simulated sensor data, and in a second operating mode in which two or more subsystems of the plurality of subsystems execute concurrently to process sensor data; 
 set a simulated clock to a first time based at least partly on a first timestamp associated with the simulated sensor data; 
 schedule execution of the plurality of subsystems in the first operating mode based on the simulated clock, wherein the simulated clock maintains the first time during execution of a first subsystem of the plurality of subsystems, and wherein the first subsystem uses the simulated clock to generate a second timestamp, representing a same time as the simulated clock, for in-process data generated by the first subsystem; 
 establish a channel for communication to a second subsystem of the plurality of subsystems; and 
 send the in-process data to the second subsystem using the channel, wherein the channel copies the in-process data from a first location of the portion of the computer-readable memory allocated to the process to a second location of the portion of the computer-readable memory allocated to the process. 
 
 
     
     
       13. The system of  claim 12 , wherein the plurality of subsystems comprises at least a portion of a vehicle control system. 
     
     
       14. The system of  claim 12 , wherein the sensor data is generated by at least one of: a LiDAR sensor, a RADAR sensor, an inertial sensor, or a camera. 
     
     
       15. The system of  claim 12 , wherein each subsystem of the plurality of subsystems is configured to receive in-process data using at least one channel of a plurality of channels. 
     
     
       16. The system of  claim 12 , wherein the one or more processors are further configured to at least:
 determine that a new in-process communication is to be sent using the channel; 
 determine, during the first operating mode, that a queue associated with the channel is full; and 
 delay adding the new in-process communication to the queue until there is space in the queue for the new in-process communication. 
 
     
     
       17. The system of  claim 12 , wherein the one or more processors are further configured to at least:
 determine that a new in-process communication is to be sent using the channel; 
 determine, during the second operating mode, that a queue associated with the channel is full; and 
 overwrite an oldest in-process communication in the queue with the new in-process communication. 
 
     
     
       18. The system of  claim 12 , wherein the one or more processors are further configured to at least:
 load an input data item from the simulated sensor data, wherein the input data item is associated with the first timestamp; and 
 determining the first time at which the simulated clock is to be set based on applying an increment to a second time represented by the first timestamp. 
 
     
     
       19. The system of  claim 18 , wherein the one or more processors are further configured to at least schedule execution of the second subsystem based at least partly on a third time, wherein the simulated clock maintains the third time during execution of the second subsystem, and wherein the second subsystem uses the simulated clock to generate a third timestamp associated with the in-process data generated by the second subsystem.