Real-time mapping of projections onto moving 3D objects

Abstract

A “Concurrent Projector-Camera” uses an image projection device in combination with one or more cameras to enable various techniques that provide visually flicker-free projection of images or video, while real-time image or video capture is occurring in that same space. The Concurrent Projector-Camera provides this projection in a manner that eliminates video feedback into the real-time image or video capture. More specifically, the Concurrent Projector-Camera dynamically synchronizes a combination of projector lighting (or light-control points) on-state temporal compression in combination with on-state temporal shifting during each image frame projection to open a “capture time slot” for image capture during which no image is being projected. This capture time slot represents a tradeoff between image capture time and decreased brightness of the projected image. Examples of image projection devices include LED-LCD based projection devices, DLP-based projection devices using LED or laser illumination in combination with micromirror arrays, etc.

What is claimed is: 
     
       1. A computer-implemented process comprising applying a general-purpose computing device to perform process actions for:
 continuously capturing image frames covering a first projection area; 
 responsive to capturing of the image frames, performing an ongoing scene motion analysis including real-time motions of one or more moving 3D objects within the first projection area; 
 further responsive to capturing of the image frames, generating one or more real-time depth maps of the first projection area, the depth maps further comprising one or more of the moving 3D objects and one or more 3D locations within the first projection area, each 3D location corresponding to a current position of one or more of the moving 3D objects; 
 responsive to the ongoing scene motion analysis and generation of the real-time depth maps, matching one or more real-time projections to one or more of the moving 3D objects; and 
 causing one or more projection devices to project frames of one or more of the matched projections onto corresponding moving 3D objects at a current 3D location within the first projection area for the corresponding moving 3D objects. 
 
     
     
       2. The computer-implemented process of  claim 1  further comprising capturing the continuously captured image frames covering the first projection area during periods that avoid corruption of the captured image frames by the projection of any frame of any of the matched projections. 
     
     
       3. The computer-implemented process of  claim 1  further comprising capturing a video covering the first projection area during periods that include the projection of frames of one or more of the matched projections. 
     
     
       4. The computer-implemented process of  claim 3  further comprising projecting the captured video to a second projection area. 
     
     
       5. The computer-implemented process of  claim 3  wherein the captured video is a 3D video. 
     
     
       6. The computer-implemented process of  claim 5  further comprising projecting the captured 3D video to a third projection area. 
     
     
       7. The computer-implemented process of  claim 1  further comprising causing one or more motorized control devices to automatically position one or more of the projection devices to project onto one or more of particular surfaces and particular objects within the first projection area. 
     
     
       8. The computer-implemented process of  claim 1  further comprising projecting frames of one or more real objects into the first projection area. 
     
     
       9. The computer-implemented process of  claim 1  further comprising projecting frames of one or more virtual objects into the first projection area. 
     
     
       10. The computer-implemented process of  claim 1  further comprising transmitting audio to the first projection area via one or more audio output devices. 
     
     
       11. The computer-implemented process of  claim 1  further comprising capturing audio within the first projection area via one or more audio input devices. 
     
     
       12. The computer-implemented process of  claim 1  further comprising projecting at least one structured light pattern onto the first projection area. 
     
     
       13. A computer-implemented process comprising applying a general-purpose computing device to perform process actions for:
 continuously capturing image frames covering a first projection area including one or more moving 3D objects; 
 responsive to capturing of the image frames, generating one or more real-time depth maps of the first projection area, the depth maps further comprising one or more of the moving 3D objects and one or more 3D locations within the first projection area, each 3D location corresponding to a current position of one or more of the moving 3D objects; 
 responsive to the generation of the real-time depth maps, matching one or more real-time projections to one or more of the moving 3D objects; and 
 causing one or more projection devices to project one or more of the matched projections onto corresponding moving 3D objects at a current 3D location within the first projection area for the corresponding moving 3D objects. 
 
     
     
       14. The computer-implemented process of  claim 13  further comprising:
 capturing a video covering the first projection area during periods that include the projection of frames of one or more of the matched projections; and 
 projecting the captured video to a second projection area. 
 
     
     
       15. The computer-implemented process of  claim 13  further comprising causing one or more motorized control devices to automatically position one or more of the projection devices to project onto one or more of particular surfaces and particular objects within the first projection area. 
     
     
       16. The computer-implemented process of  claim 13  further comprising projecting frames of any combination of one or more real objects and one or more virtual objects into the first projection area. 
     
     
       17. The computer-implemented process of  claim 13  further comprising projecting at least one structured light pattern onto the first projection area. 
     
     
       18. A system, comprising:
 one or more image capture devices configured to continuously capture image frames covering a first 3D projection area including a plurality of moving 3D objects; 
 one or more computing devices configured to generate one or more real-time depth maps from the continuously captured image frames, the depth maps further comprising one or more of the moving 3D objects and one or more 3D locations within the first 3D projection area, each 3D location corresponding to a current position of one or more of the moving 3D objects; 
 one or more of the computing devices further configured to match one or more real-time projections from the real-time depth maps to one or more of the moving 3D objects; and 
 one or more projection devices configured to project one or more of the matched projections onto corresponding moving 3D objects at a current 3D location within the first 3D projection area for the corresponding moving 3D objects. 
 
     
     
       19. The system of  claim 18  wherein one or more of the projection devices are further configured project frames of any combination of one or more real objects and one or more virtual objects into the first 3D projection area. 
     
     
       20. The system of  claim 18  further comprising one or more motorized control devices configured to automatically position any combination of the image capture devices and the projection devices.