Virtual environment scenarios and observers for autonomous machine applications

Abstract

In various examples, scenarios may be defined using a declarative description—e.g., defining a behavior of interest—that the present system may convert into a procedural description for generating one or more instances and/or variations of a scenario for testing an autonomous or semi-autonomous machine in a virtual environment. The system may execute observers or evaluators for testing the performance and accuracy of the machine and may compute coverage of various elements based on the generated virtual scenarios, and may feed the results back to the system to generate additional instances and/or variations where the coverage or accuracy is below a desired level. As a result, the system may include an end-to-end framework for generating scenarios in virtual environments, testing and validating the scenarios themselves, and/or testing and validating the underlying autonomous or semi-autonomous systems of the machine—all based on a declarative description.

What is claimed is: 
     
         1 . A method comprising:
 receiving first data that defines, at least in part, an observable related to an autonomous machine within a simulated computational environment;   determining, based at least in part on the first data, path information related to one or more defined paths within the simulated environments and dynamic actor information related to one or more dynamic actors within the simulated environment;   based at least in part on the path information and the dynamic actor information, generating second data representative of a plurality of scenarios related to the defined observable within the simulated environment; and   generating the plurality of scenarios based at least in part on the second data, wherein each scenario of the plurality of scenarios includes a set of variables determined based at least in part on the path information and the dynamic actor information.   
     
     
         2 . The method of  claim 1 , wherein the first data defines the observable using first order logic. 
     
     
         3 . The method of  claim 1 , wherein the dynamic actor information includes a description of at least one dynamic actor and an action corresponding thereto. 
     
     
         4 . The method of  claim 1 , wherein the path information includes at least one of a number of lanes, a curvature, locations of wait conditions, or types of wait conditions. 
     
     
         5 . The method of  claim 1 , further comprising:
 testing at least one feature of an autonomous or semi-autonomous machine against the plurality of scenarios; and   updating the at least one feature of the autonomous or semi-autonomous machine based at least in part on the testing.   
     
     
         6 . The method of  claim 5 , wherein the at least one feature includes at least one of a hardware component, a software component, or a sensor model. 
     
     
         7 . The method of  claim 1 , further comprising:
 evaluating the plurality of scenarios in view of the defined observable;   based at least in part on the evaluating, determining that a coverage value corresponding to the plurality of scenarios is below a threshold coverage value; and   generating at least one additional scenario based at least in part on the path information and the dynamic actor information to increase the coverage value.   
     
     
         8 . The method of  claim 7 , further comprising generating at least one additional evaluation mechanism based at least in part on an expected behavior of the autonomous machine, the path information, and the dynamic actor information to increase the coverage value. 
     
     
         9 . The method of  claim 7 , wherein the at least one additional scenario is automatically generated based at least in part on an output of the evaluating the plurality of scenarios. 
     
     
         10 . The method of  claim 1 , further comprising:
 testing a virtual instance of an autonomous or semi-autonomous machine within the plurality of scenarios;   evaluating a condition with respect to the virtual instance during the testing;   determining whether the condition is satisfied based at least in part on the evaluating; and   updating at least one feature of the autonomous or semi-autonomous machine when the condition is not satisfied.   
     
     
         11 . A method comprising:
 receiving a declarative description of a desired observable;   determining, using a domain ontology related to simulated environments for validating autonomous machines and based at least in part on the declarative description, commands for generating a plurality of scenarios related to the observable within a simulation system;   generating, using the simulation system and based at least in part on the commands, the plurality of scenarios;   running simulations of the plurality of scenarios within the simulation system;   determining accuracy of functionality and performance based on the plurality of scenarios in view of the desired observable; and   based at least in part on the accuracy of functionality and performance, determining a coverage value of the plurality of scenarios in view of the desired observable.   
     
     
         12 . The method of  claim 11 , wherein, when the coverage value is below a defined threshold, the method further comprises:
 generating new commands for generating a new plurality of scenarios; and   determining an updated coverage value in view of the new plurality of scenarios.   
     
     
         13 . The method of  claim 11 , further comprising:
 for a subset of scenarios of the plurality of scenarios that do not satisfy the desired observable, analyzing the subset of scenarios to determine criticality of the subset of scenarios to testing the autonomous machines; and   based at least in part on the analyzing, determining to generate new commands for a new plurality of scenarios.   
     
     
         14 . The method of  claim 11 , further comprising:
 testing at least one feature of an autonomous or semi-autonomous machine against the plurality of scenarios; and   updating the at least one feature of the autonomous or semi-autonomous machine based at least in part on the testing.   
     
     
         15 . The method of  claim 14 , wherein the at least one feature includes at least one of a hardware component, a software component, or a sensor model. 
     
     
         16 . The method of  claim 14 , wherein the testing the at least feature is based at least in part on the coverage value exceeding a threshold value. 
     
     
         17 . A system comprising:
 a computing device including one or more processing devices and one or more memory devices communicatively coupled to the one or more processing devices storing programmed instructions thereon, which when executed by the processor causes the instantiation of:
 a domain ontology manager to:
 access a domain ontology related to a simulated environment for testing virtual machines; and 
 select values from the domain ontology based at least in part on a declarative language description of a scene; 
 
 a scenario generator to generate multiple scenarios within a simulation environment that each include an observable represented by the declarative language and a computational engine; 
 a simulation implementer to execute simulations of at least one feature of a virtual machine through the multiple scenarios within the simulation environment; and 
 an analyzer to evaluate performance of the at least one feature during the simulations. 
   
     
     
         18 . The system of  claim 17 , further comprising the at least one feature, wherein the at least one feature includes at least one of a virtual hardware component of the virtual machine, a real-world hardware component of a real-world autonomous machine corresponding to the virtual machine, or a software component of an autonomous driving software stack or a semi-autonomous driving software stack. 
     
     
         19 . The system of  claim 18 , wherein the real-world hardware component includes at least one of a graphic processing unit (GPU), a system on chip (SoC), a field-programmable gate array (FPGA), a central processing unit (CPU), or an electronic control unit (ECU). 
     
     
         20 . The system of  claim 18 , wherein the virtual hardware component includes a virtual sensor. 
     
     
         21 . The system of  claim 18 , wherein the software component includes at least one of a deep neural network (DNN), a machine learning model, an abstract representation model, or a sensor data processor. 
     
     
         22 . The system of  claim 17 , further comprising:
 when the performance is below a desired level, generating a report of a potential cause of the performance being below the desired level; and   populating the report within a graphical user interface (GUI).