Augmented reality using projector-camera enabled devices

Abstract

An augmented reality scene may be registered onto an arbitrary surface. A camera may capture an image of the arbitrary surface. The camera may analyze the surface geometry of the arbitrary surface. In some embodiments, a processing computing device may analyze data captured by the camera and an adjacent camera to reconstruct the surface geometry of the arbitrary surface. A scene may be registered to a three dimensional coordinate system corresponding to the arbitrary surface. A projector may project the scene onto the arbitrary surface according to the registration so that the scene may not display as being distorted.

What is claimed is: 
     
       1. A system for projecting an image of a scene onto an arbitrary surface that is non-planar in both horizontal and vertical directions, the system comprising:
 a first un-calibrated camera coupled to un-calibrated in relative camera calibration parameters:  
 a first projector, the first un-calibrated camera being disposed to capture at least a part of at least a first portion of the image projected onto the arbitrary surface by the first projector, wherein the first portion of the image includes a first point; 
 an adjacent un-calibrateda second camera coupled toun-calibrated in relative camera calibration parameters;  
 a second projector, wherein the adjacent un-calibrated second camera is disposed to capture at least a part of at least a second portion of the image projected onto the arbitrary surface by the second projector, wherein the second portion of the image includes the first point; 
 wherein the first camera and the second camera are interchangeably coupled and uncoupled from the first projector and the second projector such that any camera can be coupled to any projector; and 
 aone or more processing computing device coupled to the first camera configured tocoupled to a non-transitory computer readably storage medium that includes computer readable code that, when executed by the one or more processing computing devices, causes the one or more processing computing devices to: 
 perform configuration identification of the projectors and cameras based on an adjacency graph representation in context of overlap in their display space or sensing space,  recognizereconstruct an underlying surface geometry of the arbitrary surface by analyzing data in the at least the part of at least the first portion of the image captured by the first camera and, the at least the part of at least the second portion of the image captured by the adjacentsecond camera, or a combination thereof,    register an overlapping portion of the image of the scene, that is projected by the first projector and projected by the second projector, within a three dimensional coordinate system for display on the arbitrary surface based on an analysis of the data in the at least the part of at least the first portion of the image captured by the first camera, the at least the part of at least the second portion of the image captured by the second camera, or a combination thereof, and   control the at least one of the first projector and the second projector to display the overlapping portion of the image of the registered scene onto the arbitrary surface based on an analysis of the data in the at least first portion of the image captured by the first camera and the at least second portion of the image captured by the adjacent camera.   
 
     
     
       2. The system of  claim 1 , wherein the arbitrary surface is non-planar. 
     
     
       3. The system of  claim 1 , wherein the first or the second camera is uncalibrated prior to capturing the image. 
     
     
       4. The system of claim  3   1 , wherein the first or the second projector is uncalibrated prior to projecting the image. 
     
     
       5. The system of  claim 1 , wherein the first or the second camera captures a field of view larger than a field of view of the first or the second projector respectively. 
     
     
       6. The system of  claim 1 , wherein the first camera is disposed to capture a portion of an image associated with an adjacent the second projector. 
     
     
       7. The system of  claim 1 , wherein the first or the second camera is configured to wirelessly communicate with adjacent other cameras or projectors. 
     
     
       8. The system of  claim 1 , wherein the system comprises a plurality of more than two projectorscoupled to the camera. 
     
     
       9. The system of claim  8   1 , wherein the system comprises a plurality of more than two camerascoupled to the plurality of projectors. 
     
     
       10. A method for projecting an image of a scene onto an arbitrary surface that is non-planar in both horizontal and vertical directions, the method comprising:
 performing configuration identification of projectors and cameras based on an adjacency graph representation in context of overlap in their display space or sensing space, wherein the cameras are interchangeably coupled and uncoupled from the at least two projectors such that any camera can be coupled to any projector;  
 capturing, from at least one un-calibrated camera un-calibrated in relative camera calibration parameters, a set of images of the arbitrary surface while patterns are projected on the arbitrary surface by at least one of at least two projectors, wherein the arbitrary surface is horizontally and vertically irregular, and wherein at least a first portion of a first image projected by a first projector of the at least two projectors overlaps on the arbitrary surface with at least a second portion of a second image projected by a second projector of the at least two projectors; 
 recognizingreconstructing a surface geometry of the arbitrary surface, based on the patterns projected on the arbitrary surface, using multiview stereo reconstruction to determine three dimensional locations of data points within the set of images; and 
 registering the first portion of the first image and the second portion of the second image to project the image of the scene corresponding onto the surface geometry of the arbitrary surface. 
 
     
     
       11. The method of  claim 10 , wherein the step of capturing includes using two adjacent cameras to capture the set of images. 
     
     
       12. The method of  claim 11 , comprising recovering correspondences between the two adjacent cameras using binary structured light patterns (SLP) from at least one of a plurality of projectors. 
     
     
       13. The method of  claim 11 , comprising: analyzing the set of images captured by the two adjacent cameras; and reconstructing the recognized surface geometry into a three dimensional image of the arbitrary surface based on an analysis of the set of images captured by the two cameras. 
     
     
       14. The method of  claim 10 , wherein the step of registration includes associating points in the scene to corresponding three dimensional coordinates along the surface geometry of the arbitrary surface. 
     
     
       15. The method of  claim 10 , comprising projecting a first image in the set of images to overlap an adjacent projected image in the set of images. 
     
     
       16. A non-transitory computer readable storage medium comprising:
 a computer readable code, which when executed by a processing computing device, causes the processing computing device to:
 perform configuration identification of projectors and cameras based on an adjacency graph representation in context of overlap in their display space or sensing space, wherein a first camera and a second camera are interchangeably coupled and uncoupled from a first projector and a second projector such that any camera can be coupled to any projector;  
 capture an image a scene of an arbitrary surface that is non-planar in both horizontal and vertical directions, wherein at least one point within the captured image scene is: 
 (i) within a field of view of an un-calibrateda camera un-calibrated in relative camera calibration parameters, within a first image projected by a first projector and within a second image projected by a second projector, or (ii) within an image projected by a projector and, within a first field of view ofimage captured by a first un-calibrated camera un-calibrated in relative camera calibration parameters and within a second field of view of image captured by a second un-calibrated camera un-calibrated in relative camera calibration parameters; 
 analyze a surface geometry of the arbitrary surface, from the image scene of the arbitrary surface, for surface height variations; and 
 register on the arbitrary surface (a) a first portion of the first image, that is projected by the first projector, with a second portion of the second image, that is projected by the second projector, corresponding to the surface height variations of the arbitrary surface or (b) register a first portion of the first image captured by the first camera with a second portion of the second image captured by the second camera such that first height variations in the first portion of the first image and second height variations in the second portion of the second image are indiscernible from the surface height variations of the arbitrary surface on which the first portion and the second portion are protected projected or captured. 
 
 
     
     
       17. The non-transitory computer readable storage medium of  claim 16 , wherein the computer readable code when executed by a processing computing device, relays data among adjacent cameras capturing images of the arbitrary surface. 
     
     
       18. The non-transitory computer readable storage medium of  claim 16 , wherein the scene is registered by associating points in the scene to correspond to a three dimensional coordinate system associated with corresponding three dimensional coordinates along the surface geometry of the arbitrary surface. 
     
     
       19. The non-transitory computer readable storage medium of  claim 16 , wherein the computer readable code is configured, when executed by a processing computing device, to reconstruct the captured image recover calibration matrices of the projectors using a non-linear optimization on data points representing the captured image arbitrary surface. 
     
     
       20. The non-transitory computer readable storage medium of  claim 16 , wherein the computer readable code is configured, when executed by a processing computing device, to control a projector to display the scene as corresponding to three dimensional coordinates along the surface geometry of the arbitrary surface. 
     
     
       21. The system of  claim 1 , wherein the first or the second camera is configured to wirelessly communicate with a computing device.  
     
     
       22. The system of  claim 1 , wherein the first or the second projector is configured to wirelessly communicate with a computing device.  
     
     
       23. The system of  claim 1 , wherein the computer readable code is configured, when executed by the one or more processing computing devices, to cause the one or more processing computing devices to associate points in the image to three dimensional coordinates along the surface geometry of the arbitrary surface.  
     
     
       24. The system of  claim 1 , wherein the first camera or the second camera comprises fisheye lens.  
     
     
       25. The system of  claim 1 , wherein the computer readable code is configured, when executed by the one or more processing computing devices, to cause the one or more processing computing devices to compute projector calibration parameters.  
     
     
       26. The system of  claim 1 , wherein:
 i. at least one of the first camera, the second camera, the first projector, or the second projector is calibrated with respect to a global coordinate system; or   ii. the non-transitory computer readable storage medium includes computer readable code that, when executed by the one or more processing computing devices, causes the one or more processing computing devices to calibrate at least one of the first camera, the second camera, the first projector, or the second projector to a global coordinate system.    
     
     
       27. The system of  claim 1 , wherein the computer readable code further causes the one or more processing computer devices to:
 register the first portion of the image of the scene, that is projected by the first projector, within the coordinate system for display on the arbitrary surface based on an analysis of the data in the at least the part of at least the first portion of the image captured by the first camera.    
     
     
       28. The system of  claim 1 , wherein the computer readable code further causes the one or more processing computer devices to:
 register the second portion of the image of the scene, that is projected by the second projector, within the coordinate system for display on the arbitrary surface based on an analysis of the data in the at least the part of at least the second portion of the image captured by the second camera.    
     
     
       29. The method of  claim 10 , comprising recovering correspondences between the camera and the first projector or the second projector using structured light pattern (SLP) projected from the first projector or the second projector respectively.  
     
     
       30. The method of  claim 10  further comprising computing projector calibration parameters.  
     
     
       31. The method of  claim 10 , wherein:
 i. the at least one camera or at least one of the projectors is calibrated with respect to a global coordinate system; or   ii. the method further comprises calibrating the at least one camera or at least one of the projectors to a global coordinate system.    
     
     
       32. The method of  claim 10  further comprising:
 registering the first image of the scene to project the image of the scene corresponding to the surface geometry of the arbitrary surface.  
 
     
     
       33. The method of  claim 10  further comprising:
 registering the second image of the scene to project the image of the scene corresponding to the surface geometry of the arbitrary surface.  
 
     
     
       34. The non-transitory computer readable storage medium of  claim 16 , wherein:
 i. at least one of the first camera, the second camera, or the projectors is calibrated with respect to a global coordinate system; or   ii. the computer readable code is configured, when executed by the one or more processing computing devices, to cause the one or more processing computing devices to calibrate at least one of the first camera, the second camera, or the projectors to a global coordinate system.    
     
     
       35. A projection system, comprising:
 a first projector configured to project a first image of a scene onto an arbitrary surface that is non-planar in both horizontal and vertical directions;   a second projector configured to project a second image of the scene onto the arbitrary surface that is non-planar in both horizontal and vertical directions;   a first camera, un-calibrated in relative camera calibration parameters, said first camera configured to capture at least a part of the first image, wherein the part of the first image includes a first point;   a second camera, un-calibrated in relative camera calibration parameters, said second camera configured to capture at least a part of the second image, wherein the part of the second image includes the first point, wherein the first camera and the second camera are interchangeably coupled and uncoupled from the first projector and the second projector such that any camera can be coupled to any projector; and   one or more processing computing devices coupled to a non-transitory computer readable storage medium that includes computer readable code that, when executed by the one or more processing computing devices, causes the one or more processing computing devices to:
 perform configuration identification of the projectors and cameras based on an adjacency graph representation in context of overlap in their display space or sensing space, and 
 perform geometric correction of the projected images such that the projected images are geometrically registered onto the arbitrary surface.  
   
     
     
       36. The projection system of  claim 35 , wherein the computer readable code is configured, when executed by the one or more processing computing devices, to cause the one or more processing computing devices to perform geometric correction of the projected images by reconstructing an underlying surface geometry of the arbitrary surface based on an analysis of data in the first image captured by the first camera or the second image captured by the second camera.  
     
     
       37. The projection system of  claim 35 , wherein the computer readable code is configured, when executed by the one or more processing computing devices, to cause the one or more processing computing devices to control the first projector or the second projector such that overlapping portions of the first image and the second image are displayed seamlessly.  
     
     
       38. The projection system of  claim 37 , wherein the computer readable code is configured, when executed by the one or more processing computing devices, to cause the one or more processing computing devices to associate points in the image to three dimensional coordinates along the surface geometry of the arbitrary surface.  
     
     
       39. The projection system of  claim 35 , wherein the computer readable code is configured, when executed by the one or more processing computing devices, to cause the one or more processing computing devices to reconstruct an underlying surface geometry of the arbitrary surface by analyzing data in the first image captured by the first camera or at least the second image captured by the second camera.  
     
     
       40. The projection system of  claim 39 , wherein the computer readable code is configured, when executed by the one or more processing computing devices, to cause the one or more processing computing devices to associate points in the image to three dimensional coordinates along the surface geometry of the arbitrary surface.  
     
     
       41. The projection system of  claim 35 , wherein the computer readable code is configured, when executed by the one or more processing computing devices, to cause the one or more processing computing devices to associate points in the image to three dimensional coordinates along the surface geometry of the arbitrary surface.  
     
     
       42. The system of  claim 35 , further comprising a plurality of cameras including the first camera and the second camera, wherein said plurality of cameras is un-calibrated in relative camera calibration parameters, wherein the computer readable code is configured, when executed by the one or more processing computing devices, to cause the one or more processing computing devices to identify one or more cameras of the plurality of cameras that are adjacent to each other based on an analysis of images captured by plurality of cameras.  
     
     
       43. The system of  claim 42 , wherein the computer readable code is configured, when executed by the one or more processing computing devices, to cause the one or more processing computing devices to identify one or more projectors associated with the one or more cameras of the plurality of cameras that are adjacent to each other.  
     
     
       44. The system of  claim 35 , further comprising a plurality of projectors including the first projector and the second projector, wherein the computer readable code is configured, when executed by the one or more processing computing devices, to cause the one or more processing computing devices to identify one or more projectors of the plurality of projectors that are adjacent to each other based on an analysis of images projected by the plurality of projectors and captured by one or more cameras.  
     
     
       45. The projection system of  claim 35 , wherein:
 i. at least one of the camera or the projector is calibrated with respect to a global coordinate system; or   ii. the computer readable code is configured, when executed by the one or more processing computing devices, to cause the one or more processing computing devices to calibrate at least one of the camera or the projector to a global coordinate system.    
     
     
       46. A system for projecting an image of a scene onto an arbitrary surface that is non-planar in both horizontal and vertical directions, the system comprising:
 a first camera un-calibrated in relative camera calibration parameters, said first camera configured to capture at least a portion of a first image projected onto the arbitrary surface by a first projector, wherein the portion of the first image includes a first point;   a second camera un-calibrated in relative camera calibration parameters, said second camera configured to capture at least a portion of a second image projected onto the arbitrary surface by a second projector, wherein the first camera and the second camera are interchangeably coupled and uncoupled from the first projector and the second projector such that any camera can be coupled to any projector, wherein the portion of the second image includes the first point; and   one or more processing computing devices coupled to a non-transitory computer readable storage medium that includes computer readable code that, when executed by the one or more processing computing devices, causes the one or more processing computing devices to:
 perform configuration identification of the projectors and cameras based on an adjacency graph representation in context of overlap in their display space or sensing space; and 
 control at least one of the first projector and the second projector such that an overlapping portion of the first and the second images projected by the first projector and the second projector respectively is geometrically registered onto the arbitrary surface, 
 wherein control of at least one of the first projector and the second projector is based on a reconstruction of an underlying surface geometry of the arbitrary surface from an analysis of the data in at least the portion of the first image captured by the first camera, at least the portion of the second image captured by the second camera, or a combination thereof and associating one or more points in the first and the second images to corresponding three dimensional coordinates along the surface geometry of the arbitrary surface.  
   
     
     
       47. The system of  claim 46 , wherein the first or the second camera is uncalibrated prior to capturing the portion of the first image or the portion of the second image.  
     
     
       48. The system of  claim 46 , wherein the first or the second projector is uncalibrated prior to projecting the first or the second image.  
     
     
       49. The system of  claim 46 , wherein the first or the second camera captures a field of view larger than a field of view of the first or the second projector respectively.  
     
     
       50. The system of  claim 46 , wherein the first camera is configured to capture a portion of the second image projected by the second projector or the second camera is configured to capture a portion of the first image projected by the first projector.  
     
     
       51. The system of  claim 46 , wherein the first or the second camera is configured to wirelessly communicate with other cameras or projectors.  
     
     
       52. The system of  claim 46 , wherein the system comprises a plurality of projectors coupled to the first or the second camera.  
     
     
       53. The system of  claim 52 , wherein the system comprises a plurality of cameras coupled to the plurality of projectors, wherein said plurality of cameras is un-calibrated in relative camera calibration parameters.  
     
     
       54. The system of  claim 46 , wherein:
 i. at least one of the first camera, the second camera, the first projector, or the second projector is calibrated with respect to a global coordinate system; or   ii. the non-transitory computer readable storage medium includes computer readable code that, when executed by the one or more processing computing devices, causes the one or more processing computing devices to calibrate at least one of the first camera, the second camera, the first projector, or the second projector to a global coordinate system.