Neural network-based image stream modification

Abstract

Systems, devices, media, and methods are presented for object detection and inserting graphical elements into an image stream in response to detecting the object. The systems and methods detect an object of interest in received frames of a video stream. The systems and methods identify a bounding box for the object of interest and estimate a three-dimensional position of the object of interest based on a scale of the object of interest. The systems and methods generate one or more graphical elements having a size based on the scale of the object of interest and a position based on the three-dimensional position estimated for the object of interest. The one or more graphical elements are generated within the video stream to form a modified video stream. The systems and methods cause presentation of the modified video stream including the object of interest and the one or more graphical elements.

What is claimed is: 
     
       1. A method comprising:
 receiving, by one or more processors of a mobile computing device, one or more frames of a video stream; 
 detecting, using a neural network model, an object of interest within at least a portion of the one or more frames of the video stream; 
 identifying, using a trained detector, a bounding box for the object of interest, the bounding box encompassing at least a portion of the object of interest, the trained detector being trained to identify the bounding box by selecting a default box from a plurality of default boxes, that vary over location, aspect ratio and scale, that corresponds to a ground-truth box for a training image and minimizing loss between a predicted box and the ground-truth box; 
 estimating a three-dimensional position of the object of interest based on a scale of the object of interest; 
 generating one or more graphical elements having a size based on the scale of the object of interest and a position based on the three-dimensional position, the one or more graphical elements generated within the video stream to form a modified video stream; and 
 causing presentation of the modified video stream including the object of interest and the one or more graphical elements. 
 
     
     
       2. The method of  claim 1 , wherein the three-dimensional position of the object of interest is a first position, and generating the one or more graphical elements further comprises:
 tracking the object of interest within the modified video stream to identify a position change of the object of interest within the modified video stream, the position change reflecting movement from the first position to a second position; and 
 generating a modified position for the one or more graphical elements in response to the position change of the object of interest, the modified position corresponding to the second position of the object of interest. 
 
     
     
       3. The method of  claim 2 , wherein the object of interest corresponds to a first size at the first position and a second size at the second position, and generating the one or more graphical elements further comprises:
 tracking the object of interest within the modified video stream to identify a size change of the object of interest as depicted within the modified video stream, the size change corresponding to the position change; and 
 generating a modified size for the one or more graphical elements in response to the size change of the object of interest, the modified size corresponding to the second size of the object of interest at the second position. 
 
     
     
       4. The method of  claim 1 , wherein identifying the bounding box further comprises:
 determining an object type for the object of interest; 
 selecting a bounding box type associated with the neural network model and corresponding to the object type; and 
 defining the bounding box for the object of interest corresponding to the bounding box type and at least partially encompassing the object of interest. 
 
     
     
       5. The method of  claim 1  further comprising:
 identifying one or more metadata elements corresponding to the object of interest; 
 generating a set of metadata tags corresponding to the one or more metadata elements identified for the object of interest; and 
 generating a modified bounding box by associating the set of metadata tags with the bounding box. 
 
     
     
       6. The method of  claim 1 , wherein the one or more graphical elements includes an image animated to fill the bounding box. 
     
     
       7. The method of  claim 5 , wherein the one or more graphical elements modify a portion of the object of interest, and wherein generating the one or more graphical elements further comprises:
 generating the one or more graphical elements to have an element context corresponding to at least one metadata tag of the set of metadata tags. 
 
     
     
       8. A system comprising:
 one or more processors; and 
 a non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: 
 receiving, by the one or more processors, one or more frames of a video stream; 
 detecting, using a neural network model, an object of interest within at least a portion of the one or more frames of the video stream; 
 identifying, using a trained detector, a bounding box for the object of interest, the bounding box encompassing at least a portion of the object of interest, the trained detector being trained to identify the bounding box by selecting a default box from a plurality of default boxes, that vary over location, aspect ratio and scale, that corresponds to a ground-truth box for a training image and minimizing loss between a predicted box and the ground-truth box; 
 estimating a three-dimensional position of the object of interest based on a scale of the object of interest; 
 generating one or more graphical elements having a size based on the scale of the object of interest and a position based on the three-dimensional position, the one or more graphical elements generated within the video stream to form a modified video stream; and 
 causing presentation of the modified video stream including the object of interest and the one or more graphical elements. 
 
     
     
       9. The system of  claim 8 , wherein the three-dimensional position of the object of interest is a first position, and generating the one or more graphical elements further comprises:
 tracking the object of interest within the modified video stream to identify a position change of the object of interest within the modified video stream, the position change reflecting movement from the first position to a second position; and 
 generating a modified position for the one or more graphical elements in response to the position change of the object of interest, the modified position corresponding to the second position of the object of interest. 
 
     
     
       10. The system of  claim 9 , wherein the object of interest corresponds to a first size at the first position and a second size at the second position, and generating the one or more graphical elements further comprises:
 tracking the object of interest within the modified video stream to identify a size change of the object of interest as depicted within the modified video stream, the size change corresponding to the position change; and 
 generating a modified size for the one or more graphical elements in response to the size change of the object of interest, the modified size corresponding to the second size of the object of interest at the second position. 
 
     
     
       11. The system of  claim 8 , wherein identifying the bounding box further comprises:
 determining an object type for the object of interest; 
 selecting a bounding box type associated with the neural network model and corresponding to the object type; and 
 defining the bounding box for the object of interest corresponding to the bounding box type and at least partially encompassing the object of interest. 
 
     
     
       12. The system of  claim 8 , wherein the operations further comprise:
 identifying one or more metadata elements corresponding to the object of interest; 
 generating a set of metadata tags corresponding to the one or more metadata elements identified for the object of interest; and 
 generating a modified bounding box by associating the set of metadata tags with the bounding box. 
 
     
     
       13. The system of  claim 8 , wherein the one or more graphical elements includes an image animated to fill the bounding box. 
     
     
       14. The system of  claim 12 , wherein the one or more graphical elements modify a portion of the object of interest, and wherein generating the one or more graphical elements further comprises:
 generating the one or more graphical elements to have an element context corresponding to at least one metadata tag of the set of metadata tags. 
 
     
     
       15. A non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:
 receiving one or more frames of a video stream; 
 detecting, using a neural network model, an object of interest within at least a portion of the one or more frames of the video stream; 
 identifying, using a trained detector, a bounding box for the object of interest, the bounding box encompassing at least a portion of the object of interest, the trained detector being trained to identify the bounding box by selecting a default box from a plurality of default boxes, that vary over location, aspect ratio and scale, that corresponds to a ground-truth box for a training image and minimizing loss between a predicted box and the ground-truth box; 
 estimating a three-dimensional position of the object of interest based on a scale of the object of interest; 
 generating one or more graphical elements having a size based on the scale of the object of interest and a position based on the three-dimensional position, the one or more graphical elements generated within the video stream to form a modified video stream; and 
 causing presentation of the modified video stream including the object of interest and the one or more graphical elements. 
 
     
     
       16. The non-transitory processor-readable storage medium of  claim 15 , wherein the three-dimensional position of the object of interest is a first position, and generating the one or more graphical elements further comprises:
 tracking the object of interest within the modified video stream to identify a position change of the object of interest within the modified video stream, the position change reflecting movement from the first position to a second position; and 
 generating a modified position for the one or more graphical elements in response to the position change of the object of interest, the modified position corresponding to the second position of the object of interest. 
 
     
     
       17. The non-transitory processor-readable storage medium of  claim 16 , wherein the object of interest corresponds to a first size at the first position and a second size at the second position, and generating the one or more graphical elements further comprises:
 tracking the object of interest within the modified video stream to identify a size change of the object of interest as depicted within the modified video stream, the size change corresponding to the position change; and 
 generating a modified size for the one or more graphical elements in response to the size change of the object of interest, the modified size corresponding to the second size of the object of interest at the second position. 
 
     
     
       18. The non-transitory processor-readable storage medium of  claim 15 , wherein identifying the bounding box further comprises:
 determining an object type for the object of interest; 
 selecting a bounding box type associated with the neural network model and corresponding to the object type; and 
 defining the bounding box for the object of interest corresponding to the bounding box type and at least partially encompassing the object of interest. 
 
     
     
       19. The non-transitory processor-readable storage medium of  claim 15 , wherein the operations further comprise:
 identifying one or more metadata elements corresponding to the object of interest; 
 generating a set of metadata tags corresponding to the one or more metadata elements identified for the object of interest; and 
 generating a modified bounding box by associating the set of metadata tags with the bounding box. 
 
     
     
       20. The non-transitory processor-readable storage medium of  claim 19 , wherein the one or more graphical elements modify a portion of the object of interest, and wherein generating the one or more graphical elements further comprises:
 generating the one or more graphical elements to have an element context corresponding to at least one metadata tag of the set of metadata tags.