Customized passthrough

Abstract

System and methods for customizing passthrough feed are disclosed. Expected VR content and predicted user movements in a real-world environment are determined based on recently rendered VR content, user activity data, and real-world scenes. When passthrough feed is enabled, it may be customized based on the expected VR content and predicted user movements. Customization techniques comprising custom style filters, style transfer neural networks, and adding VR content elements to the passthrough feed display window may be used to customize the passthrough feed to match the style of the VR content. The use of the customization techniques may be based on one of or a combination of a duration during which the passthrough feed is activated, the color intensity difference between the realities, an event or purpose which triggered activation of the passthrough feed, or a size of a display window of the passthrough feed relative to the VR content.

1 . (canceled) 
     
     
         2 . A method comprising:
 rendering, on a virtual reality (VR) device, VR content;   predicting a next segment of VR content based on the rendered VR content;   activating a passthrough feed;   generating a custom style filter for customizing the passthrough feed, wherein the custom style filter is based on low-level features extracted from at least an image frame of the rendered VR content;   customizing the passthrough feed based on the predicted next segment of VR content and the custom style filter; and   displaying the customized passthrough feed simultaneously with the VR content on the VR device.   
     
     
         3 . The method of  claim 2 , wherein the low-level features are extracted from a sequence of recently rendered or predicted frames of VR content. 
     
     
         4 . The method of  claim 3 , wherein the low-level features include at least one of: color attributes, patterns, and textures from the recently rendered or predicted frames of VR content. 
     
     
         5 . The method of  claim 2 , wherein a degree of blending by the custom style filter of the passthrough feed and the VR content corresponds to a level of color intensity difference between the passthrough feed and the VR content. 
     
     
         6 . The method of  claim 2 , wherein the custom style filter affects specific image properties and comprises a pixel-manipulation-based filter. 
     
     
         7 . The method of  claim 6 , wherein the specific image properties include at least one of:
 color, brightness, contrast, saturation, and gamma correction.   
     
     
         8 . The method of  claim 2 , further comprising:
 creating a style transfer neural network for customizing the passthrough feed, wherein the style transfer neural network is based on low-level features extracted from at least an image frame of the rendered VR content.   
     
     
         9 . The method of  claim 8 , wherein a degree of blending by the style transfer neural network of the passthrough feed and the VR content corresponds to a level of color intensity difference between the passthrough feed and the VR content. 
     
     
         10 . The method of  claim 8 , further comprising:
 estimating a number of layers of the style transfer neural network, the number of layers based on at least one of: a duration during which the passthrough feed is activated, a level of color intensity difference between the VR content and the passthrough feed, an event or purpose which triggered activation of the passthrough feed, or a size of a display window of the passthrough feed relative to a size of a display window of the VR content.   
     
     
         11 . The method of  claim 2 , wherein customizing the passthrough feed further comprises:
 selecting an object of interest from the VR content; and   overlaying the object of interest on a portion of the passthrough feed.   
     
     
         12 . A system comprising control circuitry configured to:
 render, on a virtual reality (VR) device, VR content;   predict a next segment of VR content based on the rendered VR content;   activate a passthrough feed;   generate a custom style filter for customizing the passthrough feed, wherein the custom style filter is based on low-level features extracted from at least an image frame of the rendered VR content;   customize the passthrough feed based on the predicted next segment of VR content and the custom style filter; and   display the customized passthrough feed simultaneously with the VR content on the VR device.   
     
     
         13 . The system of  claim 12 , wherein the low-level features are extracted from a sequence of recently rendered or predicted frames of VR content. 
     
     
         14 . The system of  claim 13 , wherein the low-level features includes at least one of: color attributes, patterns, and textures from the recently rendered or predicted frames of VR content. 
     
     
         15 . The system of  claim 12 , wherein a degree of blending by the custom style filter of the passthrough feed and the VR content corresponds to a level of color intensity difference between the passthrough feed and the VR content. 
     
     
         16 . The system of  claim 12 , wherein the custom style filter affects specific image properties and comprises a pixel-manipulation-based filter. 
     
     
         17 . The system of  claim 16 , wherein the specific image properties include at least one of:
 color, brightness, contrast, saturation, and gamma correction.   
     
     
         18 . The system of  claim 12 , wherein the control circuitry is further configured to:
 create a style transfer neural network for customizing the passthrough feed, wherein the style transfer neural network is based on low-level features extracted from at least an image frame of the rendered VR content.   
     
     
         19 . The system of  claim 18 , wherein a degree of blending by the style transfer neural network of the passthrough feed and the VR content corresponds to a level of color intensity difference between the passthrough feed and the VR content. 
     
     
         20 . The system of  claim 18 , wherein the control circuitry is further configured to:
 estimate a number of layers of the style transfer neural network, the number of layers based on at least one of: a duration during which the passthrough feed is activated, a level of color intensity difference between the VR content and the passthrough feed, an event or purpose which triggered activation of the passthrough feed, or a size of a display window of the passthrough feed relative to a size of a display window of the VR content.   
     
     
         21 . The system of  claim 12 , wherein customizing the passthrough feed further comprises:
 selecting an object of interest from the VR content; and   overlaying the object of interest on a portion of the passthrough feed.