Automated spatial indexing of images to video

Abstract

A spatial indexing system receives a video that is a sequence of frames depicting an environment, such as a floor of a construction site, and performs a spatial indexing process to automatically identify the spatial locations at which each of the images were captured. The spatial indexing system also generates an immersive model of the environment and provides a visualization interface that allows a user to view each of the images at its corresponding location within the model.

What is claimed is: 
     
       1. A method comprising:
 receiving, from a first image capture system, a set of images each comprising an image timestamp, the set of images captured by the first image capture system as the first image capture system is moved through an environment; 
 generating an estimated camera path of a second image capture system representative of movement through the environment, the estimated camera path specifying positions of frames of video captured by the second image capture system relative to one or more reference points; and 
 associating the set of images with locations along the estimated camera path based on the image timestamps of the set of images and timestamps of the frames of video captured by the second image capture system. 
 
     
     
       2. The method of  claim 1 , wherein associating the set of images with locations along the estimated camera path is further based on metadata tags in the frames of video. 
     
     
       3. The method of  claim 1 , wherein associating the set of images with locations along the estimated camera path further comprises:
 performing object detection on the frames of video captured by the second image capture system to identify a presence of the first image capture system in one of the frames; and 
 associating an image of the set of images to one of the locations along the estimated camera path based on the identified presence of the first image capture system in the frame. 
 
     
     
       4. The method of  claim 1 , wherein the first image capture system and the second image capture system are used by a same user such that a movement of the first image capture system is similar to the movement of the second image capture system. 
     
     
       5. The method of  claim 4 , wherein the second image capture system is worn by the user. 
     
     
       6. The method of  claim 1 , wherein the first image capture system is part of a mobile device that contains a mobile application that, when being used, automatically sends the set of images and corresponding timestamps to a spatial indexing system. 
     
     
       7. The method of  claim 1 , wherein the first image capture system is part of a mobile device that contains a mobile application that, when being used, automatically sends metadata associated with the set of images to a spatial indexing system. 
     
     
       8. The method of  claim 1 , wherein the estimated camera path is further based on an obtained floorplan of the environment, the floorplan specifying positions of a plurality of physical features in the environment. 
     
     
       9. The method of  claim 8 , wherein the estimated camera path is used to generate an immersive model of the environment, the immersive model:
 specifying, for each frame of video captured by the second image capture system, a location of the frame within the floorplan and at least one route vector defining a spatial distance between the frame and at least one of the other frames of the video; and 
 specifying, for each image captured by the first image capture system, a location of the image within one of the frames of the video. 
 
     
     
       10. The method of  claim 9 , further comprising:
 receiving a comment associated with one of the images, the comment comprising text authored by a user; and 
 associating the comment with the image in the immersive model. 
 
     
     
       11. The method of  claim 1 , wherein the estimated camera path is generated by performing a simultaneous localization and mapping process on the frames of video. 
     
     
       12. The method of  claim 1 , wherein a camera of the second image capture system is a 360° camera and the frames of the video are 360° frames. 
     
     
       13. The method of  claim 1 , wherein associating an image along the estimated camera path further comprises:
 orienting the image relative to a frame based on a direction of the estimated camera path of the video capture system associated with the frame; and 
 placing the image within the frame based on the orientation. 
 
     
     
       14. The method of  claim 1 , wherein associating an image with a location along the estimated camera path further comprises:
 obtaining inertial measurement unit data from the first image capture system associated with the image timestamp; and 
 placing the image within a frame further based on the obtained inertial measurement unit data. 
 
     
     
       15. A system comprising:
 a processor; and 
 a non-transitory computer readable storage medium comprising computer program instructions that when executed by the processor, cause the processor to:
 receive, from a first image capture system, a set of images each comprising an image timestamp, the set of images captured by the first image capture system as the first image capture system is moved through an environment; 
 generate an estimated camera path of a second image capture system representative of movement through the environment, the estimated camera path specifying positions of frames of video captured by the second image capture system relative to one or more reference points; and 
 associate the set of images with locations along the estimated camera path based on the image timestamps of the set of images and timestamps of the frames of video captured by the second image capture system. 
 
 
     
     
       16. The system of  claim 15 , wherein associating the set of images with locations along the estimated camera path of the second image capture system is further based on metadata tags in the video. 
     
     
       17. The system of  claim 15 , wherein associating the set of images with locations along the estimated camera path further comprises:
 performing object detection on the frames of video to identify a presence of the first image capture system in one of the frames; and 
 associating an image of the set of images to one of the locations along the estimated camera path based on the identified presence of the first image capture system in the frame. 
 
     
     
       18. The system of  claim 15 , wherein the first image capture system and the second image capture system are used by a same user such that a movement of the first image capture system is similar to the movement of the second image capture system. 
     
     
       19. The system of  claim 18 , wherein the second image capture system is worn by the user. 
     
     
       20. The system of  claim 15 , wherein the first image capture system is part of a mobile device that contains a mobile application that, when being used, automatically sends the set of images and corresponding timestamps to a spatial indexing system.