Ground surface estimation using stereo imaging for autonomous and semi-autonomous systems and applications

Abstract

Embodiments of the present disclosure relate to surface estimation using stereo imaging and surface disparities. For example, a three-dimensional (3D) surface structure may be modeled as a disparity field, and a surface disparity field representing a surface in the environment (e.g., the ground) may be generated using a constrained nonlinear hierarchical optimization to process stereo image data and iteratively refine estimated surface disparity values based on weights that guide the optimization to expected surface values (e.g., ground, road). The resulting surface (e.g., ground) disparity field may be used for a variety of downstream tasks, such as obstacle detection, segmentation of a navigable space, ego-motion refinement, and/or generation of an estimated surface profile.

What is claimed is: 
     
         1 . One or more processors comprising processing circuitry to:
 generate a surface disparity field representing estimated disparity values of a surface in an environment of an ego-machine based at least on processing a representation of stereo image data corresponding to the environment using a nonlinear hierarchical optimization; and   control one or more operations of the ego-machine based at least on the surface disparity field of the surface.   
     
     
         2 . The one or more processors of  claim 1 , wherein the processing circuitry is further to generate the surface disparity field based at least on one or more weights that encourage the nonlinear hierarchical optimization to converge to smaller disparities. 
     
     
         3 . The one or more processors of  claim 1 , wherein the processing circuitry is further to generate the surface disparity field using one or more weights that emphasize disparity values based at least on proximity to an estimated trajectory of the ego-machine. 
     
     
         4 . The one or more processors of  claim 1 , wherein the processing circuitry is further to generate the surface disparity field using one or more weights that deemphasize disparity values below a detected horizon based at least on proximity to the detected horizon. 
     
     
         5 . The one or more processors of  claim 1 , wherein the processing circuitry is further to generate the surface disparity field based at least on a measurement deviation weight that penalizes deviation between stereo disparity values and the estimated disparity values of the surface. 
     
     
         6 . The one or more processors of  claim 1 , wherein the processing circuitry is further to generate the surface disparity field based at least on deviation between stereo disparity values of a pyramid of stereo disparity layers and the estimated disparity values of the surface. 
     
     
         7 . The one or more processors of  claim 1 , wherein the processing circuitry is further to generate the surface disparity field based at least on one or more weights that emphasize disparity values that correspond to higher intensity gradient consistency in the stereo image data. 
     
     
         8 . The one or more processors of  claim 1 , wherein the processing circuitry is further to generate the surface disparity field based at least on a measurement deviation weight that penalizes deviation between optically refined stereo disparity values and the estimated disparity values of the surface. 
     
     
         9 . The one or more processors of  claim 1 , wherein the processing circuitry is further to generate the surface disparity field based at least on upsampling optically refined stereo disparity values and the estimated disparity values of the surface in the nonlinear hierarchical optimization. 
     
     
         10 . The one or more processors of  claim 1 , wherein the one or more processors are comprised in at least one of:
 a control system for an autonomous or semi-autonomous machine;   a perception system for an autonomous or semi-autonomous machine;   a system for performing simulation operations;   a system for performing digital twin operations;   a system for performing light transport simulation;   a system for performing collaborative content creation for 3D assets;   a system for performing deep learning operations;   a system for performing remote operations;   a system for performing real-time streaming;   a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;   a system implemented using an edge device;   a system implemented using a robot;   a system for performing conversational AI operations;   a system implementing one or more language models;   a system implementing one or more large language models (LLMs);   a system implementing one or more vision language models (VLMs);   a system implementing one or more multi-modal language models;   a system for generating synthetic data;   a system for generating synthetic data using AI;   a system for performing one or more generative AI operations;   a system incorporating one or more virtual machines (VMs);   a system implemented at least partially in a data center; or   a system implemented at least partially using cloud computing resources.   
     
     
         11 . A method comprising:
 generating a ground disparity field representing estimated disparity values of a ground surface in an environment of an ego-machine based at least on processing a representation of stereo image data corresponding to the environment using a nonlinear hierarchical optimization; and   controlling one or more operations of the ego-machine based at least on the ground disparity field.   
     
     
         12 . The method of  claim 11 , further comprising generating the ground disparity field based at least on one or more weights that encourage the nonlinear hierarchical optimization to converge to smaller disparities. 
     
     
         13 . The method of  claim 11 , further comprising generating the ground disparity field using one or more weights that emphasize disparity values based at least on proximity to an estimated trajectory of the ego-machine. 
     
     
         14 . The method of  claim 11 , further comprising generating the ground disparity field using one or more weights that deemphasize disparity values below a detected horizon based at least on proximity to the detected horizon. 
     
     
         15 . The method of  claim 11 , wherein the method is performed by at least one of:
 a control system for an autonomous or semi-autonomous machine;   a perception system for an autonomous or semi-autonomous machine;   a system for performing simulation operations;   a system for performing digital twin operations;   a system for performing light transport simulation;   a system for performing collaborative content creation for 3D assets;   a system for performing deep learning operations;   a system for performing remote operations;   a system for performing real-time streaming;   a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;   a system implemented using an edge device;   a system implemented using a robot;   a system for performing conversational AI operations;   a system implementing one or more language models;   a system implementing one or more large language models (LLMs);   a system implementing one or more vision language models (VLMs);   a system implementing one or more multi-modal language models;   a system for generating synthetic data;   a system for generating synthetic data using AI;   a system for performing one or more generative AI operations;   a system incorporating one or more virtual machines (VMs);   a system implemented at least partially in a data center; or   a system implemented at least partially using cloud computing resources.   
     
     
         16 . A system comprising:
 one or more processors to control, within a simulation rendered using one or more light transport simulation algorithms, one or more operations of an ego-machine in a simulated environment based at least on a ground disparity field representing estimated disparity values of a ground surface in the simulated environment, the ground disparity field generated based at least on processing a representation of stereo image data corresponding to the simulated environment using a nonlinear hierarchical optimization.   
     
     
         17 . The system of  claim 16 , wherein the simulation is generated, at least in part, using one or more content creation applications of a three-dimensional (3D) content collaboration platform for 3D assets. 
     
     
         18 . The system of  claim 17 , wherein the simulated environment is represented in at least one content creation application of the one or more content creation applications using an OpenUSD format. 
     
     
         19 . The system of  claim 16 , wherein the one or more processors are further to generate the ground disparity field based at least on one or more weights that encourage the nonlinear hierarchical optimization to converge to smaller disparities. 
     
     
         20 . The system of  claim 16 , wherein at least one of the processors is implemented in at least one processing node of a plurality of processing nodes of a data center and accessible to one or more remote clients via at least one of an application programming interface (API), or an application plug-in.