Live streaming architecture with server-side stream mixing

Abstract

A live media streaming architecture facilitates live interaction among multiple participants, such as viewers and stage performers for a show. A cluster of computers is responsible for generating, transmitting, and displaying a composed live stream, such as a grid, of one group of individuals, by mixing live streams from those individuals, that will allow them to interact with a stage performer or other contributors or other individuals during an event. The architecture can support receiving the live streams at a physical location of an event or other locations for centralized redistribution.

What is claimed is: 
     
       1. A system for delivering live streaming media, comprising:
 a plurality of origin devices, wherein each origin device is configured to receive and forward a respective stream of live streaming media received from a respective broadcaster device associated with a broadcaster; 
 a plurality of edge devices, wherein each edge device is configured to receive one or more respective streams forwarded from one or more respective origin devices from among the plurality of origin devices and to provide the received streams to at least one respective client device configured as a subscriber device; 
 a computer network interconnecting the origin devices and the edge devices for transmitting streams between the origin devices and the edge devices; 
 one or more mixers, each mixer configured as a respective subscriber device to receive streams of live streaming media from client devices and generate a respective mixed stream of live streaming media based on the received streams, and each mixer is configured as a respective origin device to forward the respective mixed stream as live streaming media to one or more edge devices; 
 a stream manager comprising a computer programmed to:
 i. assign, for each stream of live streaming media being delivered to one or more subscriber devices, a respective relationship between an origin device forwarding the stream of live streaming media and one or more edge devices receiving the stream; 
 ii. in response to a request from a client device to access a stream of live streaming media, assign the client device as a subscriber device for the stream to a respective edge device among the edge devices assigned for the stream; 
 iii. in response to a request from a client device to provide a stream of live streaming media as an input for a mixed stream, assign a mixer as a subscriber device to receive the input from the client device; 
 
 wherein the stream manager makes available one or more mixed streams generated by the one or more mixers to subscriber devices through edge devices as one or more streams of live streaming media. 
 
     
     
       2. The system of  claim 1 , wherein a first device of a first individual is connected over a broadcaster network connection to provide a first stream of live streaming media including media data of the first individual to a respective origin device of the plurality of origin devices, and the first device of the first individual is connected over a subscriber network connection to receive a second stream of live streaming media from a respective edge device among the plurality of edge devices. 
     
     
       3. The system of  claim 2 , wherein the first device of the first individual is a performer computer, wherein the first stream provided by the performer computer includes media data of the first individual, and the second stream received by the performer computer includes the respective mixed stream generated by a respective mixer among the one or more mixers. 
     
     
       4. The system of  claim 3 , wherein a second device of a second individual is a viewer computer connected over a broadcaster network connection to provide a third stream of live streaming media including media data of the second individual to a respective mixer among the one or more mixers, and the second device of the second individual is connected over a subscriber network connection to receive a fourth stream of live streaming media including a media stream based on the first stream provided by the performer computer. 
     
     
       5. The system of  claim 1 , further comprising a first device of a first individual comprising at least one performer computer providing live streaming media data of a performer and a plurality of viewer computers providing live streaming media data of a plurality of respective viewers,
 wherein the at least one performer computer is configured as a broadcaster device to provide the live streaming media data of the performer to a respective origin device among the plurality of origin devices; 
 wherein the plurality of viewer computers are configured as broadcaster devices to provide the live streaming media data of the plurality of respective viewers to respective origin devices of the plurality of origin devices; 
 wherein a first mixer among the one or more mixers is configured as a subscriber device to receive the live streaming media data of the plurality of respective viewers through the plurality of edge devices; and 
 wherein the first mixer is further configured as a broadcaster device to provide the mixed stream to a respective origin device among the plurality of origin devices. 
 
     
     
       6. The system of  claim 5 , wherein the at least one performer computer is further configured as a subscriber device to receive live streaming media data based on the mixed stream through an edge device. 
     
     
       7. The system of  claim 6 , wherein the plurality of viewer computers further include viewer computers configured as subscriber devices to receive a live streaming media stream based on the live streaming media data of the performer through an edge device. 
     
     
       8. The system of  claim 7 , wherein the plurality of viewer computers further include viewer computers configured as subscriber devices to receive live streaming media data based on the mixed stream through an edge device. 
     
     
       9. The system of  claim 5 , wherein the plurality of viewer computers are further configured as subscriber devices to receive a live streaming media stream based on the live streaming media data of the performer through an edge device. 
     
     
       10. The system of  claim 9 , wherein the plurality of viewer computers further include viewer computers configured as subscriber devices to receive live streaming media data based on the mixed stream through an edge device. 
     
     
       11. The system of  claim 5 , wherein the plurality of viewer computers further include viewer computers configured as subscriber devices to receive live streaming media data based on the mixed stream through an edge device. 
     
     
       12. The system of  claim 1 , wherein a plurality of user devices for a plurality of users are configured such that each user has a respective subscriber device and a respective broadcaster device, and wherein a first mixer among the one or more mixers is configured to receive live streaming media data of at least some of the users from the plurality of user devices and the mixer is further configured as a broadcaster device to provide a first mixed stream of live streaming media data to a respective origin device among the plurality of origin devices, and wherein at least one of the user devices is configured as a subscriber device to receive the first mixed stream from the first mixer. 
     
     
       13. The system of  claim 12 , wherein the mixer is configured as a subscriber device to receive the live streaming media data of at least some of the users from the plurality of user devices through the plurality of edge devices. 
     
     
       14. The system of  claim 1 , wherein a mixer among the one or more mixers is configured as a subscriber device to receive live streaming media data from any available broadcaster device through the plurality of edge devices, and wherein the mixer is further configured as a broadcaster device to provide a mixed stream as live streaming media data to the at least one origin device. 
     
     
       15. The system of  claim 14 , wherein a plurality of user devices are configured as broadcaster devices, and the mixer is configured to receive live streaming media data from at least some of the plurality of user devices, and at least one of the user devices is configured as a subscriber device to receive the mixed stream as live streaming media data from the mixer. 
     
     
       16. The system of  claim 14 , further comprising a producer computer having access to parameters used by the mixer to generate the mixed stream, the producer computer allowing a user to specify or modify the parameters. 
     
     
       17. The system of  claim 14 , wherein the mixer has parameters to generate the mixed stream, wherein the parameters can be modified to select different streams to be mixed. 
     
     
       18. The system of  claim 14 , wherein the mixed stream comprises live streaming media data and metadata about the live streaming media data in the mixed stream. 
     
     
       19. The system of  claim 1 , wherein the streams of live streaming media provided by the one or more mixers allows participants to publish and subscribe to live streaming media data with latencies below 500 milliseconds to enable live video interaction among the participants. 
     
     
       20. The system of  claim 1 , wherein the stream manager is further configured to, in response to requests from a producer computer, associate origin devices providing selected streams of live streaming data to be mixed using the one or more mixers.