Method and system for aligning a digital model of a structure with a video stream

Abstract

A method of aligning a digital model of a structure with a displayed portion of the structure within a video stream is disclosed. An approximate position of the camera device in the digital model is determined. A position and an orientation are determined for a plurality of digital surfaces within the digital model visible from the approximate position of the camera device. A position and an orientation of a plurality of object surfaces visible in a video stream are determined. A 3D translation, a 3D scale, and a 3D rotation that maximize an alignment of the position and orientation of the plurality of digital surfaces with the position and orientation of the plurality of object surfaces are determined. The 3D translation, the 3D scale, and the 3D rotation are applied to the digital model and the digital model is displayed contemporaneously with a display of the video stream.

1 . (canceled) 
     
     
         2 . A system comprising:
 one or more computer processors;   one or more computer memories;   a set of instructions incorporated into the one or more computer memories, the set of instructions configuring the one or more computer processors to perform operations, the operations comprising:
 receiving a video stream captured by a camera device, the video stream including a view of a structure, the structure being associated with a digital model; 
 determining an approximate position of a virtual camera in the digital model, wherein the approximate position corresponds with the view; 
 determining a position and an orientation for one or more digital surfaces within the digital model visible from the approximate position of the virtual camera; 
 determining a position and an orientation of one or more object surfaces visible in the video stream; 
 limiting a rotation of the digital model around a vertical axis, aligning the vertical axis of the digital model with a second vertical axis determined from the video stream, and determining a vertical translation of the digital model so that a lowest horizontal digital surface is aligned with a lowest horizontal object surface visible on the video stream; and 
 applying the vertical translation and the limiting of the rotation to the digital model and displaying the digital model contemporaneously with a display of the video stream. 
   
     
     
         3 . The system of  claim 2 , the operations further comprising determining an observation position in the digital model that corresponds to a configurable distance above the lowest horizontal digital surface and associated with the approximate position of the virtual camera. 
     
     
         4 . The system of  claim 3 , the operations further comprising casting a plurality of rays horizontally from the observation position in the digital model and creating a first 2D point cloud including a point for each location whereby a ray from the plurality of rays intersects with a digital surface. 
     
     
         5 . The system of  claim 3 , the operations further comprising determining a second observation position in the video stream that corresponds to the configurable distance above a lowest horizontal object surface in the video stream. 
     
     
         6 . The system of  claim 5 , the operations further comprising casting a plurality of rays horizontally from the second observation position and creating a second 2D point cloud including a point for each location whereby a ray from the plurality of rays intersects an object surface. 
     
     
         7 . The method of  claim 6 , the operations further comprising determining a 2D translation, a 2D scale, and a 2D rotation that maximize an alignment of the first point cloud and the second point cloud and applying the 2D translation, the 2D scale, and the 2D rotation to the digital model. 
     
     
         8 . The system of  claim 2 , wherein the approximate position of the virtual camera in the digital model is estimated by performing one or more of the following:
 receiving input data from an input device, wherein the input data describes the approximate position in the digital model; and   receiving a latitude, a longitude, and an altitude from a global positioning system associated with the camera device and computing the approximate position of the virtual camera based on a comparison with a predetermined latitude, longitude, and altitude of a point within the digital model.   
     
     
         9 . The system of  claim 2 , the operations further comprising:
 digitally projecting a first plurality of rays in a plurality of directions from the approximate position of the virtual camera outwards in the digital model and creating a first 3D point cloud including a point for each location whereby a ray from the first plurality of rays intersects with a digital surface of the one or more digital surfaces;   digitally projecting a second plurality of rays in a plurality of directions from the camera device within the video stream outwards in the video stream data and creating a second 3D point cloud including a point for each location whereby a ray of the second plurality of rays intersects with an object surface within the one or more object surfaces; and   determining a 3D translation, a 3D scale, and a 3D rotation that maximize an alignment of the first point cloud and the second point cloud.   
     
     
         10 . The system of  claim 9 , wherein the computing of the 3D translation, the 3D scale, and the 3D rotation comprises: based on an assumption that a scale of the structure is substantially equal to a scale of the digital model, reducing the computing to a determination of the 3D translation and the 3D rotation only. 
     
     
         11 . A non-transitory computer-readable storage medium storing a set of instructions that, when executed by one or more computer processors, causes the one or more computer processors to perform operations, the operations comprising:
 receiving a video stream captured by a camera device, the video stream including a view of a structure, the structure being associated with a digital model;   determining an approximate position of a virtual camera in the digital model, wherein the approximate position corresponds with the view;   determining a position and an orientation for one or more digital surfaces within the digital model visible from the approximate position of the virtual camera;   determining a position and an orientation of one or more object surfaces visible in the video stream;   limiting a rotation of the digital model around a vertical axis, aligning the vertical axis of the digital model with a second vertical axis determined from the video stream, and determining a vertical translation of the digital model so that a lowest horizontal digital surface is aligned with a lowest horizontal object surface visible on the video stream; and   applying the vertical translation and the limiting of the rotation to the digital model and displaying the digital model contemporaneously with a display of the video stream.   
     
     
         12 . The non-transitory computer-readable storage medium of  claim 11 , the operations further comprising determining an observation position in the digital model that corresponds to a configurable distance above the lowest horizontal digital surface and associated with the approximate position of the virtual camera. 
     
     
         13 . The non-transitory computer-readable storage medium of  claim 12 , the operations further comprising casting a plurality of rays horizontally from the observation position in the digital model and creating a first 2D point cloud including a point for each location whereby a ray from the plurality of rays intersects with a digital surface. 
     
     
         14 . The non-transitory computer-readable storage medium of  claim 12 , the operations further comprising determining a second observation position in the video stream that corresponds to the configurable distance above a lowest horizontal object surface in the video stream. 
     
     
         15 . The non-transitory computer-readable storage medium of  claim 14 , the operations further comprising casting a plurality of rays horizontally from the second observation position and creating a second 2D point cloud including a point for each location whereby a ray from the plurality of rays intersects an object surface. 
     
     
         16 . The non-transitory computer-readable storage medium of  claim 15 , the operations further comprising determining a 2D translation, a 2D scale, and a 2D rotation that maximize an alignment of the first point cloud and the second point cloud and applying the 2D translation, the 2D scale, and the 2D rotation to the digital model. 
     
     
         17 . The non-transitory computer-readable storage medium of  claim 11 , wherein the approximate position of the virtual camera in the digital model is estimated by performing one or more of the following:
 receiving input data from an input device, wherein the input data describes the approximate position in the digital model; and   receiving a latitude, a longitude, and an altitude from a global positioning system associated with the camera device and computing the approximate position of the virtual camera based on a comparison with a predetermined latitude, longitude, and altitude of a point within the digital model.   
     
     
         18 . The non-transitory computer-readable storage medium of  claim 11 , the operations further comprising:
 digitally projecting a first plurality of rays in a plurality of directions from the approximate position of the virtual camera outwards in the digital model and creating a first 3D point cloud including a point for each location whereby a ray from the first plurality of rays intersects with a digital surface of the one or more digital surfaces;   digitally projecting a second plurality of rays in a plurality of directions from the camera device within the video stream outwards in the video stream data and creating a second 3D point cloud including a point for each location whereby a ray of the second plurality of rays intersects with an object surface within the one or more object surfaces; and   determining a 3D translation, a 3D scale, and a 3D rotation that maximize an alignment of the first point cloud and the second point cloud.   
     
     
         19 . The non-transitory computer-readable storage medium of  claim 18 , wherein the computing of the 3D translation, the 3D scale, and the 3D rotation comprises: based on an assumption that a scale of the structure is substantially equal to a scale of the digital model, reducing the computing to a determination of the 3D translation and the 3D rotation only. 
     
     
         20 . A method comprising:
 receiving a video stream captured by a camera device, the video stream including a view of a structure, the structure being associated with a digital model;   determining an approximate position of a virtual camera in the digital model, wherein the approximate position corresponds with the view;   determining a position and an orientation for one or more digital surfaces within the digital model visible from the approximate position of the virtual camera;   determining a position and an orientation of one or more object surfaces visible in the video stream;   limiting a rotation of the digital model around a vertical axis, aligning the vertical axis of the digital model with a second vertical axis determined from the video stream, and determining a vertical translation of the digital model so that a lowest horizontal digital surface is aligned with a lowest horizontal object surface visible on the video stream; and   applying the vertical translation and the limiting of the rotation to the digital model and displaying the digital model contemporaneously with a display of the video stream.   
     
     
         21 . The method of  claim 20 , further comprising:
 digitally projecting a first plurality of rays in a plurality of directions from the approximate position of the virtual camera outwards in the digital model and creating a first 3D point cloud including a point for each location whereby a ray from the first plurality of rays intersects with a digital surface of the one or more digital surfaces;   digitally projecting a second plurality of rays in a plurality of directions from the camera device within the video stream outwards in the video stream data and creating a second 3D point cloud including a point for each location whereby a ray of the second plurality of rays intersects with an object surface within the one or more object surfaces; and   determining a 3D translation, a 3D scale, and a 3D rotation that maximize an alignment of the first point cloud and the second point cloud.