Systems and methods to facilitate interactions with virtual content

Abstract

According to some embodiments, a graphics platform may receive a video signal, including an image of a person, from a video camera. The graphics platform may then add a virtual object to the video signal to create a viewer or broadcast signal. 3 D information associated with a spatial relationship between the person and the virtual object is determined. The graphics platform may then create a supplemental signal based on the 3 D information, wherein the supplemental signal includes sufficient information to enable the person to interact with the virtual object as if such object was ‘seen’ or sensed from the person's perspective. The supplemental signal may comprise video, audio and/or pressure all as necessary to enable the person to interact with the virtual object as if he/she were physically present with the virtual object.

1 . A method comprising:
 receiving, at a graphics platform, 3D information about a virtual object;   determining 3D information associated with a person in a scene; and   creating, by the graphics platform: (i) a viewer signal of the scene and (ii) a supplemental signal, wherein the viewer signal and the supplemental signal are from different perspectives.   
     
     
         2 . The method of  claim 1 , wherein the viewer signal is a viewer video signal generated from a perspective of a video camera filming the scene and wherein the viewer video signal includes the virtual object and a representation of the person. 
     
     
         3 . The method of  claim 1 , wherein and the supplemental signal is a view of the virtual object from the perspective of the person. 
     
     
         4 . The method of  claim 1 , wherein the supplemental signal is a sensory signal indicative of the location of the virtual object relative to the person. 
     
     
         5 . The method of  claim 4 , wherein the sensory signal is associated with at least one of video information, audio information, pressure information, or thermal information. 
     
     
         6 . The method of  claim 1 , wherein the 3D information associated with the person is received from one or more sensors. 
     
     
         7 . The method of  claim 1 , wherein the 3D information associated with the person is determined by analyzing at least one of (i) video from at least one video camera or (ii) telemetry data from one or more sensors. 
     
     
         8 . The method of  claim 7 , wherein a plurality of video signals are received and analyzed by the graphics platform to determine at least one of: (i) a three dimensional location of the person, (ii) a distance between the person and virtual object, (iii) one or more angles associated with the person and virtual object, or (iv) an orientation of the person. 
     
     
         9 . The method of  claim 1 , wherein the view of the virtual object within the supplemental signal is associated with at least one of: (i) a marker, (ii) a lower or different resolution image as compared with the viewer signal, (iii) a lower or different frame rate image as compared with the viewer signal, or (iv) a dynamically generated occlusion zone. 
     
     
         10 . The method of  claim 9 , further comprising:
 transmitting the supplemental signal to a display device worn by the person.   
     
     
         11 . The method of  claim 10 , wherein the display device comprises at least one of: (i) an eyeglasses display, (ii) a retinal display, (iii) a contact lens display, or (iv) a hearing aid or other in-“ear” device. 
     
     
         12 . The method of  claim 10 , wherein the supplemental signal is wirelessly transmitted to the display device. 
     
     
         13 . The method of  claim 1 , wherein the supplemental signal is further based on an orientation of the person's line of sight and/or body position. 
     
     
         14 . The method of  claim 1 , wherein multiple people and virtual objects are able to interact, whether or not any or all such images are included in the final scene presented to the viewing audience, and further comprising:
 creating a supplemental signal for each person, wherein each supplemental signal includes a view or a means for perceiving the location of the virtual objects as would be seen or perceived from that person's perspective.   
     
     
         15 . The method of  claim 1 , further comprising:
 detecting a command from an entity; and   responsive to said detection, adjusting the virtual object.   
     
     
         16 . The method of  claim 15 , wherein the command comprises at least one of: (i) an audible command, (ii) a gesture command, or (iii) a third party or second virtual object command. 
     
     
         17 . The method of  claim 1 , wherein the virtual object is associated with at least one of: (i) a virtual human, (ii) a video game, or (iii) an avatar. 
     
     
         18 . A system, comprising:
 a device to capture a feed including an image or location data of a person;   a platform to receive the feed from the device and to render a supplemental feed in substantially real-time, based on 3D information relating to the person and 3D information relating to a virtual object, wherein the supplemental feed includes sufficient information to provide the person with a means of perceiving a location of the virtual object as would be perceived from the person's perspective; and   a device worn by the person to receive and present to the person the supplemental feed.   
     
     
         19 . The system of  claim 18 , wherein the platform includes at least one of: (i) a computer generated image database, (ii) a control system, (iii) a video processor, (iv) an audio processor, (v) a three dimensional graphics engine, (vi) a mixer, or (vii) a location sensor to detect a location of the person. 
     
     
         20 . The system of  claim 18 , wherein the device comprises a light detection and ranging camera. 
     
     
         21 . The system of  claim 18 , wherein the platform is further to render a viewer feed including a information of the virtual object as would be necessary to perceive the virtual object from a camera's perspective. 
     
     
         22 . The system of  claim 18 , wherein the platform receives two feeds that include images of the human and performs stereoscopic matching of pixels within the feeds to determine a three dimensional location of the person. 
     
     
         23 . The system of  claim 18 , wherein the platform uses a depth map to create a binary occlusion mask for the supplemental feed. 
     
     
         24 . A non-transitory, computer-readable medium storing instructions adapted to be executed by a processor to perform a method, the method comprising:
 receiving a signal from a camera, the signal including an image of a person;   inserting a virtual object into the signal to create a viewer signal, wherein the viewer signal includes a view of the virtual object as would be seen from the camera's perspective; and   creating a supplemental video signal, wherein the supplemental video signal includes a view of the virtual object as would be seen from the person's perspective.   
     
     
         25 . The medium of  claim 24 , wherein the method further comprises:
 outputting the supplemental feed to a device to be perceived by the person.   
     
     
         26 . The medium of  claim 24 , wherein the method further comprises:
 modeling the person based on the received video signal; and   using the model to adjust image information associated with the person in the viewer signal.