Capturing and relighting images using multiple devices

Abstract

A system and method are provided for coordinating image capture using multiple devices, including for example multiple image capture devices (cameras), multiple lighting devices (flash), and/or the like. In at least one embodiment, the system of the present invention is configured to collect image information from multiple image capture devices, such as cameras, and/or to collect multiple images having different lighting configurations. The collected image data can be processed to generate various effects, such as relighting, parallax, refocusing, and/or three-dimensional effects, and/or to introduce interactivity into the image presentation. In at least one embodiment, the system of the present invention is implemented using any combination of any number of image capture device(s) and/or flash (lighting) device(s), which may be equipped to communicate with one another via any suitable means, such as wirelessly. A communication protocol is established for enabling such communication, such as WiFi, Bluetooth, or the like.

What is claimed is: 
     
         1 . A method for capturing multiple images using multiple devices, comprising:
 at a controller, determining a pattern for capture of multiple images using multiple devices;   establishing communication, via a communication medium, between the controller and at least one image capture device;   establishing communication, via the communication medium, between the controller and at least one illumination device;   transmitting at least one message, via the communication medium, from the controller to the at least one image capture device and to the at least one illumination device to cause the at least one image capture device to capture a plurality of images of a subject, the images having a plurality of illumination configurations;   storing the captured plurality of images at a storage device; and   generating output from at least one of the stored images.   
     
     
         2 . The method of  claim 1 , wherein the communication medium comprises a wireless communication medium. 
     
     
         3 . The method of  claim 1 , wherein establishing communication between the controller and at least one image capture device comprises establishing communication between the controller and a plurality of image capture devices. 
     
     
         4 . The method of  claim 1 , wherein establishing communication between the controller and at least one illumination device comprises establishing communication between the controller and a plurality of illumination devices. 
     
     
         5 . The method of  claim 1 , wherein:
 each image capture device comprises a camera; and   each illumination device comprises a flash unit.   
     
     
         6 . The method of  claim 1 , wherein transmitting at least one message comprises:
 transmitting at least one message to cause a movable illumination device to successively illuminate the subject from each of a plurality of different positions; and   transmitting at least one message to cause at least one image capture device to capture at least one image in synchrony with each illumination of the subject.   
     
     
         7 . The method of  claim 6 , wherein transmitting at least one message further comprises transmitting at least one message to cause the movable illumination device to automatically move from one of the positions to another of the positions. 
     
     
         8 . The method of  claim 6 , wherein transmitting at least one message further comprises transmitting at least one message to cause the movable illumination device to display output instructing a user to move the illumination device from one of the positions to another of the positions. 
     
     
         9 . The method of  claim 1 , wherein transmitting at least one message comprises:
 transmitting at least one message to cause a plurality of illumination devices at different positions with respect to the subject to successively illuminate the subject at each of the positions; and   transmitting at least one message to cause at least one image capture device to capture at least one image in synchrony with each illumination of the subject.   
     
     
         10 . The method of  claim 1 , wherein:
 transmitting at least one message to the at least one image capture device comprises transmitting at least one message indicating at least one of a time and position for image capture; and   transmitting at least one message to the at least one illumination device comprises transmitting at least one message indicating at least one of a time and position for illumination of the subject.   
     
     
         11 . The method of  claim 1 , further comprising generating output to instruct a user to move at least one of an image capture device and an illumination device to a specified position. 
     
     
         12 . The method of  claim 11 , wherein generating output to instruct a user comprises generating at least one of:
 visual output;   audio output; and   haptic output.   
     
     
         13 . The method of  claim 11 , further comprising:
 detecting user movement of at least one of an image capture device and an illumination device;   comparing the detected movement with instructions in the generated output; and   responsive to the comparison, outputting feedback to the user.   
     
     
         14 . The method of  claim 1 , further comprising:
 detecting a new position of an illumination device caused by user movement of the illumination device; and   responsive to the new position being within a predefined tolerance of a predefined position for an image capture event:
 transmitting a message to cause the illumination device to illuminate the subject; and 
 transmitting a message to cause the at least one image capture device to capture an image of the subject in synchrony with the illumination of the subject. 
   
     
     
         15 . The method of  claim 1 , further comprising:
 detecting a new position of an image capture device caused by user movement of the image capture device; and   responsive to the new position being within a predefined tolerance of a predefined position for an image capture event:
 transmitting a message to cause an illumination device to illuminate the subject; and 
 transmitting a message to cause the at least one image capture device to capture an image of the subject in synchrony with the illumination of the subject. 
   
     
     
         16 . The method of  claim 1 , further comprising:
 detecting user activation of an illumination device; and   transmitting a message to cause the at least one image capture device to capture an image of the subject in synchrony with the illumination of the subject.   
     
     
         17 . The method of  claim 16 , further comprising determining a position of the user-activated illumination device. 
     
     
         18 . The method of  claim 1 , wherein determining the pattern for capture of multiple images comprises:
 detecting a series of user movements of at least one of the illumination devices, defining a movement path;   identifying a plurality of positions along the movement path as being suitable for illumination of the subject; and   storing a representation of the positions;   and wherein transmitting the at least one message to cause the at least one image capture device to capture a plurality of images of a subject comprises transmitting at least one message to cause image capture when the illumination device is at one of the identified positions.   
     
     
         19 . The method of  claim 1 , wherein determining the pattern for capture of multiple images comprises:
 detecting a series of user movements of at least one of the image capture devices, defining a movement path;   identifying a plurality of positions along the movement path as being suitable for image capture;   storing a representation of the positions; and   and wherein transmitting the at least one message to cause the at least one image capture device to capture a plurality of images of a subject comprises transmitting at least one message to cause image capture when the image capture device is at one of the identified positions.   
     
     
         20 . The method of  claim 1 , wherein the controller is a separate device from the at least one image capture device and the at least one illumination device. 
     
     
         21 . The method of  claim 1 , wherein each of the controller, the at least one image capture device, and the at least one illumination device comprises at least one selected from the group consisting of:
 a smartphone;   a tablet computer; and   a mobile computing device.   
     
     
         22 . The method of  claim 1 , wherein storing the captured plurality of images at a storage device comprises transmitting the captured image to a remote server for storage at a storage device associated with the remote server. 
     
     
         23 . The method of  claim 1 , wherein transmitting at least one message comprises transmitting at least one message to cause the at least one image capture device to capture a plurality of light-field images. 
     
     
         24 . The method of  claim 1 , further comprising:
 determining a position of the at least one image capture device and the at least one illumination device during an image capture; and   storing the determined position.   
     
     
         25 . The method of  claim 24 , wherein storing the determined position comprises storing the determined position as metadata to accompany at least one stored image. 
     
     
         26 . The method of  claim 1 , wherein generating output from at least one of the stored images comprises:
 generating an image by combining at least two of the stored images; and   outputting the generated image.   
     
     
         27 . The method of  claim 26 , wherein combining at least two of the stored images comprises generating a relighted image from a weighted combination of at least two of the stored images. 
     
     
         28 . The method of  claim 1 , wherein generating output from at least one of the stored images comprises:
 receiving user input to specify a weighting associated with at least two of the stored images;   generating an image by combining at least two of the stored images according to the specified weighting; and   outputting the generated image.   
     
     
         29 . The method of  claim 1 , wherein generating output from at least one of the stored images comprises:
 receiving user input to specify a position for at least one virtual light source;   determining, based on the specified position for at least one virtual light source, a weighting associated with at least two of the stored images; and   generating a relighted image from a combination of at least two of the stored images, wherein the combination is weighted according to the determined weighting.   
     
     
         30 . The method of  claim 29 , wherein receiving user input to specify a position for at least one virtual light source comprises receiving user input interacting with a displayed distribution graph representing weightings and positions for virtual light sources. 
     
     
         31 . The method of  claim 29 , wherein receiving user input to specify a position for at least one virtual light source comprises receiving user input dragging points on a displayed distribution graph representing weightings and positions for virtual light sources. 
     
     
         32 . The method of  claim 29 , wherein receiving user input to specify a position for at least one virtual light source comprises receiving user input to scroll at least one displayed visual element, wherein the position for at least one virtual light source is specified based on the scroll position of the at least one displayed visual element. 
     
     
         33 . The method of  claim 1 , wherein generating output from at least one of the stored images comprises:
 generating a light-field picture by combining at least two of the stored images; and   outputting an image generated from the light-field picture.   
     
     
         34 . The method of  claim 1 , further comprising, prior to establishing communication between the controller and the at least one image capture device, establishing a relationship between the at least one image capture device and the at least one illumination device. 
     
     
         35 . The method of  claim 34 , wherein establishing a relationship between the at least one image capture device and the at least one illumination device comprises pairing the at least one image capture device with the at least one illumination device. 
     
     
         36 . The method of  claim 34 , wherein establishing a relationship between the at least one image capture device and the at least one illumination device comprises:
 receiving user input requesting use of at least one illumination device associated with another user;   transmitting a first signal requesting permission from the other use to establish a relationship with the at least one illumination device;   responsive to receiving a second signal indicating permission has been granted to establish a relationship with the at least one illumination device, pairing the at least one image capture device with the at least one illumination device.   
     
     
         37 . A method for capturing multiple images using multiple devices, comprising:
 at an image capture device, determining a pattern for capture of multiple images using multiple devices;   establishing communication, via the communication medium, between the image capture device and at least one illumination device;   transmitting at least one message, via the communication medium, from the image capture device to the at least one illumination device to cause the at least one illumination device to successively illuminate a subject at a plurality of illumination configurations;   capturing a plurality of images of the subject in synchrony with the successive illuminations;   storing the captured plurality of images at a storage device; and   outputting at least one of the stored images.   
     
     
         38 . A method of interactive display of a relighted image, comprising:
 retrieving, from a storage device, at least two images representing a scene, the at least two images having different lighting characteristics;   receiving user input to specify a position for at least one virtual light source;   determining, based on the specified position for at least one virtual light source, a weighting associated with at least two of the retrieved images; and   generating a relighted image from a combination of at least two of the retrieved images, wherein the combination is weighted according to the determined weighting.   
     
     
         39 . The method of  claim 38 , wherein receiving user input to specify a position for at least one virtual light source comprises receiving user input interacting with a displayed graph representing weightings and positions for virtual light sources. 
     
     
         40 . A non-transitory computer-readable medium for capturing multiple images using multiple devices, comprising instructions stored thereon, that when executed on a processor, perform the steps of:
 determining a pattern for capture of multiple images using multiple devices;   establishing communication, via a communication medium, between the controller and at least one image capture device;   establishing communication, via the communication medium, between the controller and at least one illumination device;   transmitting at least one message, via the communication medium, from the controller to the at least one image capture device and to the at least one illumination device to cause the at least one image capture device to capture a plurality of images of a subject, the images having a plurality of illumination configurations;   causing a storage device to store the captured plurality of images; and   generating output from at least one of the stored images.   
     
     
         41 . The non-transitory computer-readable medium of  claim 40 , wherein transmitting at least one message comprises:
 transmitting at least one message to cause a movable illumination device to successively illuminate the subject from each of a plurality of different positions; and   transmitting at least one message to cause at least one image capture device to capture at least one image in synchrony with each illumination of the subject.   
     
     
         42 . The non-transitory computer-readable medium of  claim 40 , wherein transmitting at least one message comprises:
 transmitting at least one message to cause a plurality of illumination devices at different positions with respect to the subject to successively illuminate the subject at each of the positions; and   transmitting at least one message to cause at least one image capture device to capture at least one image in synchrony with each illumination of the subject.   
     
     
         43 . The non-transitory computer-readable medium of  claim 40 , further comprising instructions stored thereon, that when executed on a processor, perform the step of:
 causing an output device to generate output to instruct a user to move at least one of an image capture device and an illumination device to a specified position.   
     
     
         44 . The non-transitory computer-readable medium of  claim 40 , further comprising instructions stored thereon, that when executed on a processor, perform the steps of:
 detecting a new position of an illumination device caused by user movement of the illumination device; and   responsive to the new position being within a predefined tolerance of a predefined position for an image capture event:
 transmitting a message to cause the illumination device to illuminate the subject; and 
 transmitting a message to cause the at least one image capture device to capture an image of the subject in synchrony with the illumination of the subject. 
   
     
     
         45 . The non-transitory computer-readable medium of  claim 40 , further comprising instructions stored thereon, that when executed on a processor, perform the steps of:
 detecting a new position of an image capture device caused by user movement of the image capture device; and   responsive to the new position being within a predefined tolerance of a predefined position for an image capture event:
 transmitting a message to cause an illumination device to illuminate the subject; and 
 transmitting a message to cause the at least one image capture device to capture an image of the subject in synchrony with the illumination of the subject. 
   
     
     
         46 . The non-transitory computer-readable medium of  claim 40 , wherein transmitting at least one message comprises transmitting at least one message to cause the at least one image capture device to capture a plurality of light-field images. 
     
     
         47 . The non-transitory computer-readable medium of  claim 40 , wherein generating output from at least one of the stored images comprises:
 generating an image by combining at least two of the stored images; and   causing an output device to output the generated image.   
     
     
         48 . The non-transitory computer-readable medium of  claim 47 , wherein combining at least two of the stored images comprises generating a relighted image from a weighted combination of at least two of the stored images. 
     
     
         49 . The non-transitory computer-readable medium of  claim 40 , wherein generating output from at least one of the stored images comprises:
 receiving, from an input device, user input to specify a position for at least one virtual light source;   determining, based on the specified position for at least one virtual light source, a weighting associated with at least two of the stored images; and   generating a relighted image from a combination of at least two of the stored images, wherein the combination is weighted according to the determined weighting.   
     
     
         50 . The non-transitory computer-readable medium of  claim 40 , wherein generating output from at least one of the stored images comprises:
 generating a light-field picture by combining at least two of the stored images; and   causing at least one output device to output an image generated from the light-field picture.   
     
     
         51 . A system for capturing multiple images using multiple devices, comprising:
 a controller, configured to:
 determine a pattern for capture of multiple images using multiple devices; 
 establish communication, via a communication medium, between the controller and at least one image capture device; 
 establish communication, via the communication medium, between the controller and at least one illumination device; and 
 transmit at least one message, via the communication medium, to the at least one image capture device and to the at least one illumination device to cause the at least one image capture device to capture a plurality of images of a subject, the images having a plurality of illumination configurations; 
   a storage device, communicatively coupled to the controller, configured to store the captured plurality of images; and   an output device, communicatively coupled to the storage device, configured to generate output from at least one of the stored images.   
     
     
         52 . The system of  claim 51 , wherein the communication medium comprises a wireless communication medium. 
     
     
         53 . The system of  claim 51 , wherein the controller is configured to establish communication between the controller and a plurality of image capture devices. 
     
     
         54 . The system of  claim 51 , wherein the controller is configured to establish communication between the controller and a plurality of illumination devices. 
     
     
         55 . The system of  claim 51 , wherein:
 each image capture device comprises a camera; and   each illumination device comprises a flash unit.   
     
     
         56 . The system of  claim 51 , wherein the controller is configured to:
 transmit at least one message to cause a movable illumination device to successively illuminate the subject from each of a plurality of different positions; and   transmit at least one message to cause at least one image capture device to capture at least one image in synchrony with each illumination of the subject.   
     
     
         57 . The system of  claim 51 , wherein the controller is configured to:
 transmit at least one message to cause a plurality of illumination devices at different positions with respect to the subject to successively illuminate the subject at each of the positions; and   transmit at least one message to cause at least one image capture device to capture at least one image in synchrony with each illumination of the subject.   
     
     
         58 . The system of  claim 51 , wherein the controller is configured to cause at least one output device to generate output to instruct a user to move at least one of an image capture device and an illumination device to a specified position. 
     
     
         59 . The system of  claim 51 , wherein the controller is configured to:
 detect a new position of an illumination device caused by user movement of the illumination device; and   responsive to the new position being within a predefined tolerance of a predefined position for an image capture event:
 transmit a message to cause the illumination device to illuminate the subject; and 
 transmit a message to cause the at least one image capture device to capture an image of the subject in synchrony with the illumination of the subject. 
   
     
     
         60 . The system of  claim 51 , wherein the controller is configured to:
 detect a new position of an image capture device caused by user movement of the image capture device; and   responsive to the new position being within a predefined tolerance of a predefined position for an image capture event:
 transmit a message to cause an illumination device to illuminate the subject; and 
 transmit a message to cause the at least one image capture device to capture an image of the subject in synchrony with the illumination of the subject. 
   
     
     
         61 . The system of  claim 51 , wherein the controller is configured to transmit at least one message by transmitting at least one message to cause the at least one image capture device to capture a plurality of light-field images. 
     
     
         62 . The system of  claim 51 , further comprising an image combining module, communicatively coupled to the storage device, configured to combine at least two of the stored images to generate an image, and wherein the output device is configured to output the generated image. 
     
     
         63 . The system of  claim 62 , wherein the image combining module is configured to combine at least two of the stored images by generating a relighted image from a weighted combination of at least two of the stored images. 
     
     
         64 . The system of  claim 51 , further comprising:
 at least one input device, communicatively coupled to the controller, configured to be receptive to user input to specify a position for at least one virtual light source;   wherein:   the image combining module is configured to determine, based on the specified position for at least one virtual light source, a weighting associated with at least two of the stored images, and to generate a relighted image from a combination of at least two of the stored images, wherein the combination is weighted according to the determined weighting;   and wherein the output device is configured to output the generated relighted image.   
     
     
         65 . The system of  claim 51 , further comprising an image combining module, communicatively coupled to the storage device, configured to generate a light-field picture by combining at least two of the stored images;
 and wherein the output device is configured to output an image generated from the light-field picture.