Augmented reality system for viewing an event with mode based on crowd sourced images

Abstract

Augmented reality systems provide graphics over views from a mobile device for both in-venue and remote viewing of a sporting or other event. A server system can provide a transformation between the coordinate system of a mobile device (smart phone, tablet computer, head mounted display) and a real world coordinate system. Requested graphics for the event are displayed over a view of an event.

We claim: 
     
         1 . A method, comprising:
 sending, from a server system to each of a plurality of mobile devices at a venue, instructions to generate image data and corresponding image metadata of an event at the venue to use to build a model of the venue, each of the mobile devices independently maintaining a corresponding internal coordinate system and the corresponding image metadata including information about a location and an orientation of the corresponding mobile device within the venue in the internal coordinate system of the corresponding mobile device when capturing the image data;   receiving, at the server system from each of the plurality of mobile devices, the corresponding image data and corresponding image metadata generated in response to the instructions;   building by the server system of the model of the venue in a real world coordinate system from the image data and image metadata from the plurality of mobile devices, the model comprising a reference map including location data of a set reference features in the real world coordinate system;   generating by the server system of a corresponding transformation between each of the mobile device's internal coordinate system and the real world coordinate system from the corresponding image data and image metadata of each of the mobile devices and the model of the venue in the real world coordinate system built from the image data and image metadata from the plurality of mobile devices; and   transmitting from the server system to each of the mobile devices the corresponding transformation between the mobile device's internal coordinate system and the real world coordinate system.   
     
     
         2 . The method of  claim 1 , further comprising:
 retrieving point features of the venue in a first coordinate system and locations of a set of fiducials features for the venue in the real world coordinate system by the server system from one or more databases,   wherein the location of each of the fiducial features is at corresponding point of the venue in the real world coordinate system as determined in a survey of the venue, and   wherein the server system further builds the model of the venue in the real world coordinate system from the point features of the venue in the first coordinate system and the locations of the set of fiducials features for the venue in the real world coordinate system.   
     
     
         3 . The method of  claim 2 , further comprising:
 building the model of the venue in the real world coordinate system from the point features of the venue in the first coordinate system and the locations of the set of fiducials features for the venue in the real world coordinate system prior to the event; and   during the event, revising the model of the venue in the real world coordinate system using the image data of the venue for the event and image metadata as received from the plurality of mobile devices during the event.   
     
     
         4 . The method of  claim 2 , wherein building the model of the venue in the real world coordinate system includes:
 building the model of the venue in the real world coordinate system from the point features of the venue in the first coordinate system, the locations of the set of fiducials features for the venue in the real world coordinate system, and the image data of the venue for the event and image metadata as received from the plurality of mobile devices prior to the event.   
     
     
         5 . The method of  claim 2 , wherein generating the corresponding transformation between each mobile device's internal coordinate system and the real world coordinate system includes:
 determining a transformation between the image data from the mobile device and the point features of the venue from the image data and image metadata from the mobile device and from the point features of the venue;   locating the set of fiducials features in the real world coordinate system within the image data from the mobile device; and   determining, from the transformation between the image data from the mobile device and the point features of the venue and from locating of features in the real world coordinate system within the image data from the mobile device, a pose for the image from the mobile device in the real world coordinate system.   
     
     
         6 . The method of  claim 1 , wherein building the model of the venue in the real world coordinate system includes:
 processing the image data and image metadata from the plurality of mobile devices to generate point features of the venue and the locations of a set of fiducials features in the world coordinate system, wherein the transformation between each of the mobile device's internal coordinate system and the real world coordinate system is generated by comparing the image data and image metadata from the mobile device with the point features of the venue.   
     
     
         7 . The method of  claim 6 , wherein building the model of the venue in the real world coordinate system further includes:
 extracting one or more of the set of fiducial features from the image data and image metadata from the plurality of mobile devices.   
     
     
         8 . The method of  claim 1 , further comprising:
 receiving, at the server system for one or more of the mobile devices, an independent request for graphics to be displayed by the mobile device over a view of the venue as specified by location and orientation in the real world coordinate system; and   transmitting from the server system to each of the mobile devices the requested graphics.   
     
     
         9 . A system, comprising:
 one or more servers configured to access data from one or more databases, to receive data from and transmit data to a plurality of mobile devices, and to:
 send, to each of a plurality of mobile devices at a venue, instructions to generate image data and corresponding image metadata of an event at the venue to use to build a model of the venue, each of the mobile devices independently maintaining a corresponding internal coordinate system and the corresponding image metadata including information about a location and an orientation of the corresponding mobile device within the venue in the internal coordinate system of the corresponding mobile device when capturing the image data; 
 receive, from each of the plurality of mobile devices, the corresponding image data and corresponding image metadata generated in response to the instructions; 
 build the model of the venue in a real world coordinate system from the image data and image metadata from the plurality of mobile devices, the model comprising a reference map including location data of a set reference features in the real world coordinate system; 
 generate a corresponding transformation between each of the mobile device's internal coordinate system and the real world coordinate system from the corresponding image data and image metadata of each of the mobile devices and the model of the venue in the real world coordinate system built from the image data and image metadata from the plurality of mobile devices; and 
 transmit to each of the mobile devices the corresponding transformation between the mobile device's internal coordinate system and the real world coordinate system. 
   
     
     
         10 . The system of  claim 9 , wherein the one or more servers are further configured to:
 retrieve point features of the venue in a first coordinate system and locations of a set of fiducials features for the venue in the real world coordinate system from one or more databases, wherein the location of each of the fiducial features is at corresponding point of the venue in the real world coordinate system as determined in a survey of the venue, and wherein the system of one or more serversfurther builds the model of the venue in the real world coordinate system from the point features of the venue in the first coordinate system and the locations of the set of fiducials features for the venue in the real world coordinate system.   
     
     
         11 . The system of  claim 10 , wherein the one or more servers are further configured to:
 build the model of the venue in the real world coordinate system from the point features of the venue in the first coordinate system and the locations of the set of fiducials features for the venue in the real world coordinate system prior to the event; and   during the event, revise the model of the venue in the real world coordinate system using the image data of the venue for the event and image metadata as received from the plurality of mobile devices during the event.   
     
     
         12 . The system of  claim 10 , wherein, to build the model of the venue in the real world coordinate system, the one or more servers are further configured to:
 build the model of the venue in the real world coordinate system from the point features of the venue in the first coordinate system, the locations of the set of fiducials features for the venue in the real world coordinate system, and the image data of the venue for the event and image metadata as received from the plurality of mobile devices prior to the event.   
     
     
         13 . The system of  claim 10 , wherein, to generate the corresponding transformation between each mobile device's internal coordinate system and the real world coordinate system the one or more servers are further configured to:
 determine a transformation between the image data from the mobile device and the point features of the venue from the image data and image metadata from the mobile device and from the point features of the venue;   locate the set of fiducials features in the real world coordinate system within the image data from the mobile device; and   determine, from the transformation between the image data from the mobile device and the point features of the venue and from locating of features in the real world coordinate system within the image data from the mobile device, a pose for the image from the mobile device in the real world coordinate system.   
     
     
         14 . The system of  claim 9 , wherein, to build the model of the venue in the real world coordinate system, the one or more servers are further configured to:
 process the image data and image metadata from the plurality of mobile devices to generate point features of the venue and the locations of a set of fiducials features in the world coordinate system, wherein the transformation between each of the mobile device's internal coordinate system and the real world coordinate system is generated by comparing the image data and image metadata from the mobile device with the point features of the venue.   
     
     
         15 . The system of  claim 14 , wherein, to build the model of the venue in the real world coordinate system, the one or more servers are further configured to:
 extract one or more of the set of fiducial features from the image data and image metadata from the plurality of mobile devices.   
     
     
         16 . The system of  claim 9 , wherein the one or more servers are further configured to:
 receive, for one or more of the mobile devices, an independent request for graphics to be displayed by the mobile device over a view of the venue as specified by location and orientation in the real world coordinate system; and   transmit to each of the mobile devices the requested graphics.   
     
     
         17 . The system of  claim 9 , wherein the one or more servers are further configured to:
 receive by the one or more servers of a request from a first mobile device to supply specified content with specified graphics to a second mobile device; and   transmit the specified content with specified graphics to the second mobile device.   
     
     
         18 . The system of  claim 17 , wherein: the image data from the plurality of mobile devices comprise crowd sourced images that are used to achieve a cumulative result in a form of the model, wherein identity and/or number of the plurality of mobile devices are not known in advance prior to the event at the venue. 
     
     
         19 . A method, comprising:
 independently maintaining by each of a plurality of mobile devices at a venue a corresponding internal coordinate system;   receiving, at each of the of mobile devices at the venue, instructions from a server system to generate image data and corresponding image metadata of an event at the venue to use to build a model of the venue, the corresponding image metadata including information about a location and an orientation of the corresponding mobile device within the venue in the internal coordinate system of the corresponding mobile device when capturing the image data;   receiving, at the server system from each of the plurality of mobile devices, the corresponding image data and corresponding image metadata generated in response to the instructions;   building by the server system of the model of the venue in a real world coordinate system from the image data and image metadata from the plurality of mobile devices, the model comprising a reference map including location data of a set reference features in the real world coordinate system;   generating by the server system of a corresponding transformation between each of the mobile device's internal coordinate system and the real world coordinate system from the corresponding image data and image metadata of each of the mobile devices and the model of the venue in the real world coordinate system built from the image data and image metadata from the plurality of mobile devices;   transmitting from the server system to each of the mobile devices the corresponding transformation between the mobile device's internal coordinate system and the real world coordinate system; and   receiving at each of the mobile devices the corresponding transformation between the mobile device's internal coordinate system and the real world coordinate system.   
     
     
         20 . The method of  claim 19 , further comprising:
 receiving point features of the venue and locations of a set of fiducials features for the venue in the real world coordinate system by the server system from one or more databases, wherein building the model of the venue in the real world coordinate system includes:
 building the model of the venue in the real world coordinate system from the point features of the venue, the locations of the set of fiducials features for the venue in the real world coordinate system, and the image data of the venue for the event and image metadata as received from the plurality of mobile devices prior to the event.