Autonomous Control Of Powered Earth-Moving Vehicles To Control Blade Tool Loading Operations

Abstract

Systems and techniques are described for implementing autonomous control of powered earth-moving vehicles, including to automatically control movement of a powered earth-moving vehicle on a job site to control loading of a blade tool attachment, such as to manage tool attachment height and transitions between a pushing/cutting/loading mode and a carrying mode. The automated operations may include generating automated predicted estimates at one or more times of an amount and/or degree of loading of a blade tool attachment on a powered earth-moving vehicle (e.g., on a bulldozer vehicle) being used to move material in a pushing/cutting/loading mode and/or of whether such loading is causing slippage of the vehicle, using an indication of vehicle slippage to initiate raising the blade tool attachment, and using an indication of a fully loaded blade (or a loading degree and/or amount above one or more specified thresholds) to switch to a carrying mode.

What is claimed is: 
     
         1 . An autonomous vehicle steering system using a blade tool attachment, comprising:
 a bulldozer vehicle with a chassis, tracks, a blade tool attachment on a front of the chassis, hydraulic arms between the chassis and the blade tool attachment, one or more first controls for manipulating movement of the tracks, and one or more second controls for manipulating the blade tool attachment via the hydraulic arms;   a microcontroller unit on the bulldozer vehicle that is capable of effecting movement of the first and second controls via piston displacement mechanisms; and   a control system on the bulldozer vehicle that is configured to be in communication with the microcontroller unit and to perform automated operations including:
 determining, while the bulldozer vehicle is in motion and is using the blade tool attachment to push ground material in at least one of a pushing mode or a cutting mode or a loading mode, and by a trained machine learning model using multiple data readings from a plurality of sensors on the bulldozer vehicle, at least one of a predicted degree or a predicted amount of how full the blade tool attachment is with the ground material; 
 determining, based at least in part on the determined at least one of the predicted degree or the predicted amount of how full the blade tool attachment is with the ground material satisfying a defined threshold, to initiate an end to the at least one of the pushing mode or the cutting mode or the loading mode; and 
 initiating, in response to the determining to initiate the end to the at least one of the pushing mode or the cutting mode or the loading mode, autonomous operations of the bulldozer vehicle to at least one of raise the blade tool attachment to end contact with the ground material by using the second controls to manipulate the hydraulic arms via at least one of the piston displacement mechanisms, or stop the motion of the bulldozer vehicle by using the first controls to stop the movement of the tracks via at least one of the piston displacement mechanisms. 
   
     
     
         2 . The autonomous vehicle steering system of  claim 1  wherein the determining to initiate the end to the at least one of the pushing mode or the cutting mode or the loading mode includes determining to switch to a carrying mode to move the ground material in the blade tool attachment to a different location, and wherein the initiated autonomous operations include using the second controls to raise the blade tool attachment to end contact with the ground material, and using the first controls to cause the movement of the tracks toward the different location. 
     
     
         3 . The autonomous vehicle steering system of  claim 1  wherein the determining of the at least one of the predicted degree or the predicted amount of how full the blade tool attachment is with the ground material includes generating a binary output of the blade tool attachment being fully loaded and the defined threshold is that generated binary output, or wherein the determining of the at least one of the predicted degree or the predicted amount of how full the blade tool attachment is with the ground material includes generating a degree of loading of the blade tool attachment relative to being fully loaded and the defined threshold is an indicated degree of loading that is exceeded by the generated degree of loading, or wherein the determining of the at least one of the predicted degree or the predicted amount of how full the blade tool attachment is with the ground material includes generating a predicted estimate of the quantity of the ground material loaded in the blade tool attachment and the defined threshold is an indicated quantity that is exceeded by the generated estimate of the quantity. 
     
     
         4 . The autonomous vehicle steering system of  claim 1  wherein the determining of the at least one of the predicted degree or the predicted amount of how full the blade tool attachment is with the ground material is performed repeatedly during the motion of the bulldozer vehicle using the blade tool attachment to push the ground material in the at least one of the pushing mode or the cutting mode or the loading mode, and wherein the automated operations include determining, before the determining to initiate the end to the at least one of the pushing mode or the cutting mode or the loading mode, to continue the at least one of the pushing mode or the cutting mode or the loading mode in response to one or more prior determinations of the at least one of the predicted degree or the predicted amount of how full the blade tool attachment is with the ground material being below the defined threshold. 
     
     
         5 . The autonomous vehicle steering system of  claim 4  wherein the automated operations further include, as part of one of the one or more prior determinations of the at least one of the predicted degree or the predicted amount of how full the blade tool attachment is with the ground material being below the defined threshold:
 determining, while the bulldozer vehicle is in motion and is using the blade tool attachment to push the ground material in the at least one of the pushing mode or the cutting mode or the loading mode, and by a trained machine learning model using multiple data readings from the plurality of sensors on the bulldozer vehicle, predicted vehicle slippage of the bulldozer vehicle from a reduction in traction based at least in part on the using of the blade tool attachment; and 
 initiating, in response to the determining of the vehicle slippage, autonomous operations of the bulldozer vehicle to use the second controls to manipulate the hydraulic arms via at least one of the piston displacement mechanisms to raise the blade tool attachment while maintaining contact with the ground material. 
 
     
     
         6 . The autonomous vehicle steering system of  claim 1  wherein the trained machine learning model is an LSTM (long short-term memory) recurrent neural network with a classifier output, and wherein the multiple data readings from the plurality of sensors on the bulldozer vehicle provide vehicle status data that includes multiple data types from a group including RPMs (revolutions per minute) of an engine of the bulldozer vehicle, and fuel consumption of the bulldozer vehicle, and engine torque of the bulldozer vehicle, and engine load of the bulldozer vehicle, and speed of the tracks of the bulldozer vehicle, and speed of the chassis of the bulldozer vehicle, and pitch pressure for the blade tool attachment, and a transmission gear ratio in use by the bulldozer vehicle. 
     
     
         7 . The autonomous vehicle steering system of  claim 6  wherein the multiple data readings further include at least one of a measure of an amount of the ground material that is spilling over at least one of a side or a top of the blade tool attachment, or a degree of tilt in a pitch of the bulldozer vehicle. 
     
     
         8 . The autonomous vehicle steering system of  claim 6  wherein the multiple data readings further include one or more data readings from one or more additional sensors external to the bulldozer vehicle, the one or more additional sensors including at least one of an image sensor of a camera or a sensor of a LIDAR component. 
     
     
         9 . The autonomous vehicle steering system of  claim 6  wherein the automated operations further include training the machine learning model before using the trained machine learning model for the determining of the at least one of the predicted degree or the predicted amount of how full the blade tool attachment is with the ground material, including, for each of one or more operational bulldozer vehicles each having a respective blade tool attachment, obtaining multiple camera-based estimates of how full that blade tool attachment is at multiple time points based at least in part on images of that blade tool attachment captured at the multiple time points that are input to a camera-based blade load predictor machine learning model, and supplying the multiple camera-based estimates of how full the blade tool attachment is as training data to the machine learning model along with additional values for the vehicle status data that is captured at the multiple time points for that operational bulldozer vehicle. 
     
     
         10 . The autonomous vehicle steering system of  claim 1  further comprising:
 a LIDAR component that is mounted on the bulldozer vehicle and configured to obtain LiDAR data indicating a plurality of three-dimensional (“3D”) points on surfaces of at least some of a job site on which the bulldozer vehicle is located; 
 one or more GPS antennas mounted at one or more positions on the chassis and capable of receiving GPS signals for use in determining GPS coordinates of at least some of the chassis; 
 one or more inertial navigation system units mounted at one or more positions on the chassis and capable of determining a current direction of the bulldozer vehicle; and 
 one or more first position sensors mounted on the hydraulic arms and configured to detect one or more first angles between the chassis and the hydraulic arms, and one or more second position sensors mounted on the blade tool attachment and configured to detect one or more second angles between the blade tool attachment and at least one of the hydraulic arms, 
 and wherein the automated operations include providing information for display about the determined at least one of the predicted degree or the predicted amount of how full the blade tool attachment is with the ground material. 
 
     
     
         11 . The autonomous vehicle steering system of  claim 1  wherein the control system is configured to implement at least some automated operations of an earth-moving vehicle autonomous operations control system by executing software instructions of the earth-moving vehicle autonomous operations control system, and wherein the determining of the at least one of the predicted degree or the predicted amount of how full the blade tool attachment is with the ground material and the determining to initiate the end to the at least one of the pushing mode or the cutting mode or the loading mode and the initiating of the autonomous operations are performed autonomously without receiving human input and without receiving external signals other than GPS signals and real-time kinematic (RTK) correction signals. 
     
     
         12 . An autonomous vehicle steering system using a blade tool attachment, comprising:
 a bulldozer vehicle with a chassis, tracks, a blade tool attachment on a front of the chassis, hydraulic arms between the chassis and the blade tool attachment, one or more first controls for manipulating movement of the tracks, and one or more second controls for manipulating the blade tool attachment via the hydraulic arms;   a microcontroller unit on the bulldozer vehicle that is capable of effecting movement of the first and second controls via piston displacement mechanisms; and   a control system on the bulldozer vehicle that is configured to be in communication with the microcontroller unit and to perform automated operations including:
 determining, while the bulldozer vehicle is in motion and is using the blade tool attachment to push ground material in at least one of a pushing mode or a cutting mode or a loading mode, and by a trained machine learning model using data readings from a plurality of sensors on the bulldozer vehicle, a predicted reduction in traction of the bulldozer vehicle that is causing vehicle slippage; 
 determining, based at least in part on the determined predicted reduction in traction of the bulldozer vehicle, to initiate a change to the at least one of the pushing mode or the cutting mode or the loading mode; and 
 initiating, in response to the determining to initiate the change to the at least one of the pushing mode or the cutting mode or the loading mode, autonomous operations of the bulldozer vehicle that include raising the blade tool attachment while maintaining contact with the ground material by using the second controls to manipulate the hydraulic arms via at least one of the piston displacement mechanisms. 
   
     
     
         13 . The autonomous vehicle steering system of  claim 12  wherein determining of whether the predicted reduction in traction of the bulldozer vehicle is occurring is performed repeatedly during the motion of the bulldozer vehicle using the blade tool attachment to push the ground material in the at least one of the pushing mode or the cutting mode or the loading mode, and wherein the automated operations include, in response to each of multiple determinations of the predicted reduction in traction of the bulldozer vehicle that is causing vehicle slippage, performing the raising of the blade tool attachment relative to a prior raising of the blade tool attachment and while maintaining the contact with the ground material. 
     
     
         14 . A computer-implemented method, comprising:
 determining, by one or more configured hardware processors on a powered earth-moving vehicle while the powered earth-moving vehicle is in motion and is performing autonomous operations that include positioning at least some of a blade tool attachment of the powered earth-moving vehicle below a ground surface to move ground material in a cutting mode, and using a trained machine learning model and data readings from a plurality of sensors on the powered earth-moving vehicle, at least one of that the powered earth-moving vehicle is predicted to be experiencing vehicle slippage from the vehicle operations, or that a predicted degree of how full the blade tool attachment is with the ground material is above a defined threshold, wherein the powered earth-moving vehicle has a chassis and has at least one of tracks or wheels and has one or more first controls for manipulating movement of the at least one of the tracks or wheels and has one or more second controls for manipulating the blade tool attachment via one or more intervening hydraulic arms; and   initiating, by the one or more configured hardware processors and based at least in part on the determining, further autonomous operations of the powered earth-moving vehicle to implement a change to the cutting mode, wherein the initiated change includes raising the blade tool attachment while maintaining contact with the ground surface and continuing the cutting mode if it is determined that the powered earth-moving vehicle is predicted to be experiencing slipping, and wherein the initiated change is to initiate an end to the cutting mode if it is determined that the predicted degree of how full the blade tool attachment is with the ground material is above the defined threshold, and wherein the further autonomous operations include at least one of using at least one of the second controls to manipulate the blade tool attachment via the one or more intervening hydraulic arms, or using at least one of the first controls to manipulate the at least one of the tracks or wheels.   
     
     
         15 . The computer-implemented method of  claim 14  wherein the powered earth-moving vehicle is a bulldozer vehicle, wherein the determining includes predicting the degree of how full the blade tool attachment is with the ground material and determining that the predicted degree exceeds the defined threshold, and wherein the initiating of the further autonomous operations includes at least one of raising the blade tool attachment to end contact with the ground material by using the second controls to manipulate the hydraulic arms via at least one piston displacement mechanism, or stopping the motion of the bulldozer vehicle by using the first controls to stop the movement of the at least one of the tracks or the wheels via at least one piston displacement mechanism. 
     
     
         16 . The computer-implemented method of  claim 14  wherein the powered earth-moving vehicle is a bulldozer vehicle, wherein the determining includes predicting that the powered earth-moving vehicle is experiencing vehicle slippage from the vehicle operations, and wherein the initiating of the autonomous operations includes raising the blade tool attachment while maintaining contact with the ground material and continuing cutting mode by using the second controls to manipulate the hydraulic arms via at least one piston displacement mechanism. 
     
     
         17 . The computer-implemented method of  claim 14  wherein the powered earth-moving vehicle is one of a bulldozer vehicle or a motorized grader vehicle or a plowing vehicle, wherein at least one of the one or more hardware processors is a low-voltage microcontroller that is located on the powered earth-moving vehicle and is configured to implement at least some automated operations of an earth-moving vehicle autonomous operations control system by executing software instructions of the earth-moving vehicle autonomous operations control system, and wherein the determining and the initiating of the further autonomous operations are performed autonomously without receiving human input and without receiving external signals other than GPS signals and real-time kinematic (RTK) correction signals. 
     
     
         18 . A non-transitory computer-readable medium having stored contents that cause one or more hardware processors to perform automated operations including at least:
 receiving, by the one or more hardware processors, data for a powered earth-moving vehicle with a blade tool attachment, wherein the powered earth-moving vehicle is moving at least one of tracks or wheels and using the blade tool attachment to perform pushing of material in an environment of the powered earth-moving vehicle, the data indicating at least one of engine revolutions per minute, fuel consumption, engine torque, engine load, speed of the at least one of the tracks or wheels, or a transmission gear ratio;   applying, by the one or more hardware processors, a machine learning model to the received data to generate a predicted estimated degree of loading of the blade tool attachment with the material during the pushing of the material;   determining, by the one or more hardware processors and based at least in part on the predicted estimated degree of loading of the blade tool attachment satisfying a defined threshold level, an output indicating at least one of the blade tool attachment being fully loaded or the powered earth-moving vehicle experiencing slippage from the pushing of the material; and   initiating, by the one or more hardware processors and in response to the determining of the output, an automated remedial action.   
     
     
         19 . The non-transitory computer-readable medium of  claim 18  wherein the determined output includes that the blade tool attachment is fully loaded, and wherein the automated remedial action includes at least one of: switching the powered earth-moving vehicle to a carrying mode that includes carrying the material in the fully-loaded blade tool attachment; or ending the pushing of the material by at least one of raising the blade tool attachment or stopping the moving of the at least one of the tracks or wheels. 
     
     
         20 . The non-transitory computer-readable medium of  claim 18  wherein the determined output includes that the powered earth-moving vehicle is experiencing slippage from the pushing of the material, and wherein the automated remedial action includes raising the blade tool attachment to reduce an amount of an amount of resistance to the powered earth-moving vehicle from the pushing of the material. 
     
     
         21 . The non-transitory computer-readable medium of  claim 18  wherein the determined output includes that the blade tool attachment is fully loaded and that the powered earth-moving vehicle is experiencing slippage from the pushing of the material, and wherein the automated operations include informing an operator of the powered earth-moving vehicle of at least one of the blade tool attachment being fully loaded, or the powered earth-moving vehicle experiencing slippage from the pushing of the material.