System and method of representations of user interfaces of an electronic device

Abstract

Some examples of the disclosure are directed to systems and methods for displaying and interacting with a representation of a user interface of an electronic device in a three-dimensional environment. While presenting a first user interface, a first electronic device presents a representation of a second user interface of a second electronic device based on display data representing the second user interface. In response to detecting a respective event corresponding to user input, if a gaze of a user of the first electronic device is directed to the representation of the second user interface, the first electronic device causes the second electronic device to perform a first operation directed to the second user interface based on the respective event. Alternatively, if the gaze is directed to the first user interface, the first electronic device performs a second operation directed to the first user interface based on the respective event.

What is claimed is: 
     
       1. A method comprising:
 at a first electronic device in communication with a display and one or more input devices of the first electronic device:
 capturing, via the one or more input devices, at least a portion of a real-world environment including a second electronic device with a display configured to display a first representation of a first user interface of the second electronic device; 
 presenting, via the display, a computer-generated environment including a representation of the captured portion of the real-world environment and a first affordance associated with a representation of the second electronic device; 
 while presenting the computer-generated environment including the representation of the captured portion of the real-world environment and the first affordance, receiving, via the one or more input devices, an input corresponding to a selection of the first affordance; 
 in response to receiving the input, presenting, in the computer-generated environment, a second representation of a second user interface of the second electronic device different than the first representation of the first user interface; 
 while presenting the computer-generated environment that includes the second representation of the second user interface, detecting a first indication of first respective input received via one or more input devices of the second electronic device; and 
 in response to detecting the first indication:
 in accordance with a determination that a gaze of a user of the first electronic device is directed to the first representation of the first user interface, causing the second electronic device to perform a first operation directed to the first user interface in accordance with the first respective input; and 
 in accordance with a determination that the gaze is directed to the second representation of the second user interface, performing a second operation directed to the second representation of the second user interface in accordance with the first respective input. 
 
 
 
     
     
       2. The method of  claim 1 , wherein the first affordance is displayed in a predetermined region of the display that is separate from the representation of the second electronic device in the computer-generated environment. 
     
     
       3. The method of  claim 1 , further comprising:
 in response to detecting the first indication:
 in accordance with a determination that the gaze of the user is directed to a respective location in the computer-generated environment that does not include the first representation of the first user interface or the second representation of the second user interface, forgoing performing any operation involving the first representation of the first user interface or the second representation of the second user interface. 
 
 
     
     
       4. The method of  claim 1 , further comprising:
 while presenting the computer-generated environment that includes the second representation of the second user interface, detecting a second indication of second respective input received via the one or more input devices of the second electronic device, wherein, after detecting the first indication, an indication of input was not detected before detecting the second indication; and 
 in response to detecting the second indication and in accordance with a determination that the gaze of the user is directed to a respective location in the computer-generated environment that does not include the first representation of the first user interface or the second representation of the second user interface:
 in accordance with a determination that the gaze was directed to the first representation of the first user interface when the first indication was detected, causing the second electronic device to perform a third operation directed to the first user interface in accordance with the second respective input; and 
 in accordance with a determination that the gaze was directed to the second representation of the second user interface when the first indication was detected, performing a fourth operation directed to the second representation of the second user interface in accordance with the second respective input. 
 
 
     
     
       5. The method of  claim 1 , further comprising:
 while presenting the computer-generated environment that includes the second representation of the second user interface, detecting, via the one or more input devices of the first electronic device, a respective input; and 
 in response to detecting the respective input:
 in accordance with a determination that the gaze is directed to the first representation of the first user interface, causing the second electronic device to perform a third operation directed to the first user interface in accordance with the respective input; and 
 in accordance with a determination that the gaze is directed to the second representation of the second user interface, performing a fourth operation directed to the second representation of the second user interface in accordance with the respective input. 
 
 
     
     
       6. The method of  claim 5 , wherein the first representation of the first user interface includes a cursor controllable via one or more of the one or more input devices of the second electronic device. 
     
     
       7. The method of  claim 6 , wherein:
 the first respective input corresponds to movement of the cursor; and 
 in response to detecting the first indication, in accordance with the determination that the gaze is directed to the second representation of the second user interface, performing the fourth operation directed to the second representation of the second user interface includes:
 moving the cursor from the first representation of the first user interface to the second representation of the second user interface in accordance with the movement of the cursor. 
 
 
     
     
       8. The method of  claim 7 , wherein:
 the one or more of the one or more input devices includes a first input device having a touch-sensitive surface; and 
 the first respective input includes movement of a contact on the touch-sensitive surface that satisfies one or more criteria. 
 
     
     
       9. The method of  claim 8 , wherein the one or more criteria include a criterion that is satisfied when the movement of the contact includes lift-off of the contact from the touch-sensitive surface. 
     
     
       10. The method of  claim 8 , wherein the one or more criteria include a criterion that is satisfied when the movement of the contact on the touch-sensitive surface exceeds a threshold movement. 
     
     
       11. The method of  claim 1 , wherein:
 the real-world environment further includes a third electronic device; and 
 in response to detecting the first indication:
 forgoing performing any operation involving the third electronic device. 
 
 
     
     
       12. The method of  claim 1 , further comprising:
 before displaying the first affordance associated with the representation of the second electronic device in the computer-generated environment, pairing the first electronic device and the second electronic device, such that the first electronic device is in communication with the second electronic device. 
 
     
     
       13. The method of  claim 12 , wherein pairing the first electronic device and the second electronic device includes:
 scanning, via the one or more input devices of the first electronic device, an image that is displayed via the display of the second electronic device that causes the first electronic device to be in communication with the second electronic device. 
 
     
     
       14. The method of  claim 12 , wherein pairing the first electronic device and the second electronic device includes:
 detecting, via the one or more input devices of the first electronic device, respective audio that is output from the second electronic device that causes the first electronic device to be in communication with the second electronic device. 
 
     
     
       15. The method of  claim 12 , wherein pairing the first electronic device and the second electronic device includes:
 detecting an indication of input received via the one or more input devices of the second electronic device that causes the first electronic device to be in communication with the second electronic device. 
 
     
     
       16. The method of  claim 1 , further comprising:
 while presenting the computer-generated environment that includes the second representation of the second user interface, detecting disassociation of the first electronic device from the user of the first electronic device; and 
 in response to detecting the disassociation of the first electronic device from the user:
 transmitting, to the second electronic device, an indication of the disassociation, including data indicative of a state of the second representation of the second user interface, wherein the indication causes the second electronic device to concurrently display the first user interface and the second user interface. 
 
 
     
     
       17. The method of  claim 16 , further comprising:
 detecting reassociation of the first electronic device with the user of the first electronic device; 
 in response to detecting the reassociation of the first electronic device with the user:
 presenting the computer-generated environment; and 
 transmitting, to the second electronic device, an indication of the reassociation; 
 
 while presenting the computer-generated environment, receiving, from the second electronic device, data indicative of a current state of the second user interface; and 
 in response to receiving the data:
 redisplaying the second representation of the second user interface in the computer-generated environment based on the current state of the second user interface. 
 
 
     
     
       18. The method of  claim 1 , wherein:
 the first electronic device includes a head-mounted display; and 
 the second electronic device is a laptop computer, a desktop computer, or a tablet computer. 
 
     
     
       19. The method of  claim 1 , further comprising:
 while presenting the computer-generated environment, receiving an invitation to join a communication session with a user of a third electronic device; and 
 in response to receiving the invitation:
 in accordance with a determination that the invitation is accepted by the user of the first electronic device, entering the communication session with the user of the third electronic device, including concurrently displaying an avatar corresponding to the user of the third electronic device and the first representation of the first user interface. 
 
 
     
     
       20. The method of  claim 19 , further comprising:
 while presenting the computer-generated environment that includes the avatar corresponding to the user of the third electronic device and the first representation of the first user interface, receiving an indication corresponding to a request from the third electronic device to share content with the first electronic device; and 
 in response to receiving the indication:
 in accordance with a determination that the request is accepted by the user of the first electronic device, displaying a third representation of a third user interface corresponding to the content in the computer-generated environment, wherein the third user interface is configured to be displayed on the third electronic device. 
 
 
     
     
       21. The method of  claim 20 , further comprising:
 while presenting the computer-generated environment that includes the third representation of the third user interface, detecting, via the one or more input devices of the first electronic device, an input directed to the third representation of the third user interface; and 
 in response to detecting the input:
 forgoing performing an operation directed to the third representation of the third user interface in accordance with the input. 
 
 
     
     
       22. A first electronic device comprising:
 one or more processors; 
 memory; and 
 one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing a method comprising:
 capturing, via one or more input devices, at least a portion of a real-world environment including a second electronic device with a display configured to display a first representation of a first user interface of the second electronic device; 
 presenting, via a display, a computer-generated environment including a representation of the captured portion of the real-world environment and a first affordance associated with a representation of the second electronic device; 
 while presenting the computer-generated environment including the representation of the captured portion of the real-world environment and the first affordance, receiving, via the one or more input devices, an input corresponding to a selection of the first affordance; 
 in response to receiving the input, presenting, in the computer-generated environment, a second representation of a second user interface of the second electronic device different than the first representation of the first user interface; 
 while presenting the computer-generated environment that includes the second representation of the second user interface, detecting a first indication of first respective input received via one or more input devices of the second electronic device; and 
 in response to detecting the first indication:
 in accordance with a determination that a gaze of a user of the first electronic device is directed to the first representation of the first user interface, causing the second electronic device to perform a first operation directed to the first user interface in accordance with the first respective input; and 
 in accordance with a determination that the gaze is directed to the second representation of the second user interface, performing a second operation directed to the second representation of the second user interface in accordance with the first respective input. 
 
 
 
     
     
       23. The first electronic device of  claim 22 , wherein the first affordance is displayed in a predetermined region of the display that is separate from the representation of the second electronic device in the computer-generated environment. 
     
     
       24. The first electronic device of  claim 22 , wherein the method further comprises:
 in response to detecting the first indication:
 in accordance with a determination that the gaze of the user is directed to a respective location in the computer-generated environment that does not include the first representation of the first user interface or the second representation of the second user interface, forgoing performing any operation involving the first representation of the first user interface or the second representation of the second user interface. 
 
 
     
     
       25. The first electronic device of  claim 22 , wherein the method further comprises:
 while presenting the computer-generated environment that includes the second representation of the second user interface, detecting a second indication of second respective input received via the one or more input devices of the second electronic device, wherein, after detecting the first indication, an indication of input was not detected before detecting the second indication; and 
 in response to detecting the second indication and in accordance with a determination that the gaze of the user is directed to a respective location in the computer-generated environment that does not include the first representation of the first user interface or the second representation of the second user interface:
 in accordance with a determination that the gaze was directed to the first representation of the first user interface when the first indication was detected, causing the second electronic device to perform a third operation directed to the first user interface in accordance with the second respective input; and 
 in accordance with a determination that the gaze was directed to the second representation of the second user interface when the first indication was detected, performing a fourth operation directed to the second representation of the second user interface in accordance with the second respective input. 
 
 
     
     
       26. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of a first electronic device, cause the first electronic device to perform a method comprising:
 capturing, via one or more input devices, at least a portion of a real-world environment including a second electronic device with a display configured to display a first representation of a first user interface of the second electronic device; 
 presenting, via a display, a computer-generated environment including a representation of the captured portion of the real-world environment and a first affordance associated with a representation of the second electronic device; 
 while presenting the computer-generated environment including the representation of the captured portion of the real-world environment and the first affordance, receiving, via the one or more input devices, an input corresponding to a selection of the first affordance; 
 in response to receiving the input, presenting, in the computer-generated environment, a second representation of a second user interface of the second electronic device different than the first representation of the first user interface; 
 while presenting the computer-generated environment that includes the second representation of the second user interface, detecting a first indication of first respective input received via one or more input devices of the second electronic device; and 
 in response to detecting the first indication:
 in accordance with a determination that a gaze of a user of the first electronic device is directed to the first representation of the first user interface, causing the second electronic device to perform a first operation directed to the first user interface in accordance with the first respective input; and 
 in accordance with a determination that the gaze is directed to the second representation of the second user interface, performing a second operation directed to the second representation of the second user interface in accordance with the first respective input. 
 
 
     
     
       27. The non-transitory computer readable storage medium of  claim 26 , wherein the method further comprises:
 while presenting the computer-generated environment that includes the second representation of the second user interface, detecting, via the one or more input devices of the first electronic device, a respective input; and 
 in response to detecting the respective input:
 in accordance with a determination that the gaze is directed to the first representation of the first user interface, causing the second electronic device to perform a third operation directed to the first user interface in accordance with the respective input; and 
 in accordance with a determination that the gaze is directed to the second representation of the second user interface, performing a fourth operation directed to the second representation of the second user interface in accordance with the respective input. 
 
 
     
     
       28. The non-transitory computer readable storage medium of  claim 27 , wherein the first representation of the first user interface includes a cursor controllable via one or more of the one or more input devices of the second electronic device. 
     
     
       29. The non-transitory computer readable storage medium of  claim 28 , wherein:
 the first respective input corresponds to movement of the cursor; and 
 in response to detecting the first indication, in accordance with the determination that the gaze is directed to the second representation of the second user interface, performing the fourth operation directed to the second representation of the second user interface includes:
 moving the cursor from the first representation of the first user interface to the second representation of the second user interface in accordance with the movement of the cursor. 
 
 
     
     
       30. The non-transitory computer readable storage medium of  claim 29 , wherein:
 the one or more of the one or more input devices includes a first input device having a touch-sensitive surface; and 
 the first respective input includes movement of a contact on the touch-sensitive surface that satisfies one or more criteria.