Generation device, generation method, and generation program

Abstract

A generation device (10) includes: a 3D reconstruction unit (12) that reconstructs an original three-dimensional image based on a plurality of images and a plurality of depth images, and acquires information indicating a position, a posture, a shape, and an appearance of a mapping target object that is a target of mapping to a digital space, and position information and posture information of an imaging device that has captured the image and the depth image; a labeling unit (13) that acquires a plurality of two-dimensional images in which labels or categories are associated with all pixels in an image based on the plurality of images; an estimation unit (14) that estimates a material and mass of the mapping target object based on the plurality of two-dimensional images and the position information and the posture information of the imaging device; and a generation unit (16) that integrates the information indicating a position, a posture, a shape, and an appearance of the mapping target object, and the information indicating the material and mass of the mapping target object, and generates digital twin data including position information, posture information, shape information, appearance information, material information, and mass information of the mapping target object.

1 . A generation device comprising:
 a reconstruction unit that reconstructs an original three-dimensional image based on a plurality of images and a plurality of depth images, and acquires information indicating a position, a posture, a shape, and an appearance of a mapping target object that is a target of mapping to a digital space, and position information and posture information of an imaging device that has captured the image and the depth image;   an association unit that acquires a plurality of two-dimensional images in which labels or categories are associated with all pixels in an image based on the plurality of images;   an estimation unit that estimates a material and mass of the mapping target object based on the plurality of two-dimensional images and the position information and the posture information of the imaging device; and   a first generation unit that integrates the information indicating a position, a posture, a shape, and an appearance of the mapping target object acquired by the reconstruction unit, and the information indicating the material and mass of the mapping target object estimated by the estimation unit, and generates digital twin data including position information, posture information, shape information, appearance information, material information, and mass information of the mapping target object.   
     
     
         2 . The generation device according to  claim 1 , wherein
 the estimation unit includes   a second generation unit that generates a plurality of extracted images obtained by extracting the mapping target object based on a plurality of the two-dimensional images,   a first estimation unit that extracts two or more of the extracted images including the same mapping target object from the plurality of extracted images based on position information and posture information of the imaging device, and estimates a material for each of the mapping target objects included in the two or more extracted images,   a determination unit that performs statistical processing on the material information of each mapping target object estimated by the first estimation unit, and determines the material of the mapping target object based on a result of the statistical processing, and   a second estimation unit that estimates the mass of the mapping target object based on a material of the mapping target object and a volume of the mapping target object determined by the determination unit.   
     
     
         3 . The generation device according to  claim 1 or 2 , further comprising:
 an acquisition unit that acquires metadata including a generator, a generation date and time, and a file capacity of the digital twin data as metadata, wherein   the first generation unit assigns the metadata acquired by the acquisition unit to the digital twin data.   
     
     
         4 . A generation method executed by a generation device, the method comprising:
 a step of reconstructing an original three-dimensional image based on a plurality of images and a plurality of depth images, and acquiring information indicating a position, a posture, a shape, and an appearance of a mapping target object that is a target of mapping to a digital space, and position information and posture information of an imaging device that has captured the image and the depth image;   a step of acquiring a plurality of two-dimensional images in which labels or categories are associated with all pixels in an image based on the plurality of images;   a step of estimating a material and mass of the mapping target object based on the plurality of two-dimensional images and the position information and the posture information of the imaging device; and   a step of integrating the information indicating a position, a posture, a shape, and an appearance of the mapping target object, and the information indicating the material and mass of the mapping target object, and generating digital twin data including position information, posture information, shape information, appearance information, material information, and mass information of the mapping target object.   
     
     
         5 . A generation program for causing a computer to function as the generation device according to any one of  claims 1 to 3 .