Transport object specifying device of work machine, work machine, transport object specifying method of work machine, method for producing complementary model, and dataset for learning

Abstract

A transport object specifying device of a work machine includes an image acquisition unit, a drop target specifying unit, a three-dimensional data generation unit, and a surface specifying unit. The image acquisition unit acquires a captured image showing a drop target of the work machine in which a transport object is dropped. The drop target specifying unit specifies a three-dimensional position of at least part of the drop target based on the captured image. The three-dimensional data generation unit generates depth data, which is three-dimensional data representing a depth of the captured image, based on the captured image. The surface specifying unit specifies a three-dimensional position of a surface of the transport object in the drop target by removing, from the depth data, a part corresponding to the drop target based on the three-dimensional position of the at least part of the drop target.

1 . A transport object specifying device of a work machine, the transport object specifying device comprising:
 an image acquisition unit that acquires a captured image showing a drop target of the work machine in which a transport object is dropped;   a drop target specifying unit that specifies a three-dimensional position of at least part of the drop target based on the captured image;   a three-dimensional data generation unit that generates depth data, which is three-dimensional data representing a depth of the captured image, based on the captured image; and   a surface specifying unit that specifies a three-dimensional position of a surface of the transport object in the drop target by removing, from the depth data, a part corresponding to the drop target based on the three-dimensional position of the at least part of the drop target.   
     
     
         2 . The transport object specifying device according to  claim 1 , further comprising:
 a feature point specifying unit that specifies a position of a feature point of the drop target based on the captured image,   the drop target specifying unit specifying the three-dimensional position of the at least part of the drop target based on the position of the feature point.   
     
     
         3 . The transport object specifying device according to  claim 1 , wherein
 the drop target specifying unit specifies the three-dimensional position of the at least part of the drop target based on the captured image and a target model, which is a three-dimensional model indicating a shape of the drop target.   
     
     
         4 . The transport object specifying device according to  claim 1 , wherein
 the surface specifying unit
 extracts, from the depth data, a three-dimensional position in a prismatic area which, is surrounded by a wall portion of the drop target and extends in a height direction of the wall portion, and 
 removes the part corresponding to the drop target in the extracted three-dimensional position to specify the three-dimensional position of the surface of the transport object. 
   
     
     
         5 . The transport object specifying device according to  claim 1 , further comprising:
 a distribution specifying unit that generates distribution information indicating a distribution of an amount of the transport object in the drop target based on
 the three-dimensional position of the surface of the transport object in the drop target and 
 the three-dimensional position of the at least part of the drop target. 
   
     
     
         6 . The transport object specifying device according to  claim 5 , further comprising:
 a distribution estimation unit that estimates a distribution of an amount of the transport object in a shielded part of the distribution information shielded by an obstacle.   
     
     
         7 . The transport object specifying device according to  claim 6 , wherein
 the distribution estimation unit inputs the distribution information generated by the distribution specifying unit to a complementary model to generate distribution information that complements a value of the shielded part, and   the complementary model is a trained model, which, when distribution information with some missing values is input, outputs distribution information that complements the missing values.   
     
     
         8 . The transport object specifying device according to  claim 6 , wherein
 the distribution estimation unit generates distribution information that complements a value of the shielded part based on a rate of change or a pattern of change in a three-dimensional position of the transport object near the shielded part.   
     
     
         9 . The transport object specifying device according to  claim 6 , wherein
 the distribution estimation unit estimates a distribution of an amount of the transport object in the shielded part based on a type of the transport object.   
     
     
         10 . The transport object specifying device according to  claim 1 , wherein
 the captured image is a stereo image including at least a first image and a second image captured by a stereo camera.   
     
     
         11 . A work machine including the transport object specifying device according to  claim 1 , the work machine further comprising:
 work equipment usable to transport the transport object;   an imaging device; and   a display device that displays information about the transport object in the drop target specified by the transport object specifying device.   
     
     
         12 . A transport object specifying method of a work machine, comprising:
 acquiring a captured image showing a drop target of the work machine in which a transport object is dropped;   specifying a three-dimensional position of at least part of the drop target based on the captured image;   generating depth data, which is three-dimensional data representing a depth of the captured image, based on the captured image; and   removing, from the depth data, a part corresponding to the drop target based on the three-dimensional position of the at least part of the drop target to specify a three-dimensional position of a surface of the transport object in the drop target.   
     
     
         13 . A method for producing a complementary model, which, when distribution information with some missing values is input, outputs distribution information that complements the missing values, the method comprising:
 acquiring distribution information indicating a distribution of an amount of a transport object in a drop target of a work machine and incomplete distribution information in which some values of the distribution information are missing, as a dataset for learning; and   training the complementary model using the dataset for learning such that when the incomplete distribution information is used as an input value, the distribution information becomes an output value.   
     
     
         14 . (canceled)