Modifying video call data

Abstract

A method comprising: displaying a UI for display of received video; detecting selection of a UI button; receiving a plurality of drawing inputs while the button is selected, each drawing input defining image data to be applied at a facial position on a first side of a face of a user displayed in the received video; for each drawing input, determining a further facial position on a second side of the face, that is symmetrically opposite to the facial position; and for each received video frame, the method comprises: for each drawing input (i) determining a position of the face in the frame by executing an algorithm to determine the locations of the facial position and determined further facial position on the face in the frame; and (ii) applying the image data to the facial position and the determined further facial position; and displaying the modified frame in the UI.

The invention claimed is: 
     
       1. A method implemented at a user terminal during a video call conducted with at least one further user terminal over a communications network, the method comprising:
 displaying a user interface on a display of the user terminal for display of received video frames; 
 detecting selection of a selectable button displayed in said user interface; 
 receiving a plurality of drawing inputs while the selectable button is in a selected state, each drawing input defining image data to be applied at a facial position on a first side of a face of a user displayed in the received video frames; 
 for each of the plurality of drawing inputs, determining a further facial position on a second side of the face, the further facial position on the second side of the face being symmetrically opposite to the facial position on the first side of said face; 
 modifying each received video frame by applying the image data to the facial position on the first side of the face of the user displayed in the received video frame and applying the image data to the further facial position on the second side of the face of the user displayed in the received video frame; and 
 displaying the modified video frame in the user interface. 
 
     
     
       2. The method of  claim 1 , further comprising, for each of the received video frames, determining the position of the face in the received video frame by identifying facial feature points of said face in two-dimensions and fitting a three-dimensional triangulated mesh to the tracked facial feature points, wherein for each of the plurality of drawing inputs the facial position corresponds to a position on said three-dimensional triangulated mesh, and the determined further facial position corresponds to a further position on said three-dimensional triangulated mesh. 
     
     
       3. The method of  claim 2 , wherein the modifying each received video frame further comprises mapping textures of each triangle of the three-dimensional triangulated mesh obtained onto a two-dimensional representation of the three-dimensional triangulated mesh; and
 for each received drawing input:
 determining a position and further position in the two-dimensional representation of the three-dimensional triangulated mesh that corresponds to the position and further position on said three-dimensional triangulated mesh; 
 applying the image data to the determined position and determined further position in the two-dimensional representation of the three-dimensional triangulated mesh; and 
 mapping textures of each triangle of the two dimensional representation of the three-dimensional triangulated mesh onto the three-dimensional triangulated mesh. 
 
 
     
     
       4. The method of  claim 2 , wherein the modifying each received video frame further comprises mapping textures of each triangle of the three-dimensional triangulated mesh onto a first two-dimensional representation of the three-dimensional triangulated mesh; and
 for each received drawing input:
 determining a position and further position in a second two-dimensional representation of the three-dimensional triangulated mesh that corresponds to the position and further position on said three-dimensional triangulated mesh; 
 applying the image data to the determined position and determined further position in the second two-dimensional representation of the three-dimensional triangulated mesh; 
 blending the second two-dimensional representation of the three-dimensional triangulated mesh onto the first two-dimensional representation of the three-dimensional triangulated mesh to create a blended two-dimensional representation of the three-dimensional triangulated mesh; and 
 mapping textures of each triangle of the blended two-dimensional representation of the three-dimensional triangulated mesh onto the three-dimensional triangulated mesh. 
 
 
     
     
       5. The method of  claim 1 , wherein the method further comprises displaying at least one selectable drawing tool button in the user interface. 
     
     
       6. The method of  claim 5 , wherein the at least one selectable drawing tool button comprises one or any combination of: a pen drawing tool button, a lipstick drawing tool button, or a brush drawing tool button. 
     
     
       7. The method of  claim 5 , wherein the method further comprises in response to detecting selection of one of the at least one selectable drawing tool buttons, displaying a plurality of selectable colours associated with the selected drawing tool button. 
     
     
       8. The method of  claim 5 , wherein the image data is based on the selected drawing tool button and a selected colour of the plurality of selectable colours associated with the selected drawing tool button. 
     
     
       9. The method of  claim 8 , wherein the image data comprises a colour modification factor to be applied to at least one pixel at the facial position on the first side of said face, the colour modification factor based on the selected colour. 
     
     
       10. The method of  claim 9 , wherein the colour modification factor comprises: (i) a multiplicative term and an additive term that is to be applied to an original red colour value of each of the at least one pixel; (ii) a multiplicative term and an additive term that is to be applied to an original green colour value of each of the at least one pixel; and (iii) a multiplicative term and an additive term that is to be applied to an original blue colour value of each of the at least one pixel. 
     
     
       11. The method of  claim 8 , wherein the image data comprises a colour value to be applied to at least one pixel at the facial position on the first side of said face, the colour value based on the selected colour. 
     
     
       12. The method of  claim 1 , further comprising for each of the plurality of drawing inputs, storing a drawing command in a memory of said user terminal, the drawing command comprising the image data associated with said drawing input, an indication of the facial position on the first side of said face, and an indication of the determined further facial position on the second side of said face. 
     
     
       13. The method of  claim 12 , wherein modifying the video frame comprises retrieving all of the drawing commands stored in said memory, and modifying the video frame in accordance with the retrieved drawing commands. 
     
     
       14. The method of  claim 1 , wherein video frames are received from a camera of said user terminal. 
     
     
       15. The method of  claim 14 , the method further comprising transmitting each modified video frame over the communications network to the at least one further user terminal. 
     
     
       16. The method of  claim 1 , wherein video frames are received over the communications network from said at least one further user terminal. 
     
     
       17. The method of  claim 1 , wherein the plurality of drawing inputs are received from a user associated with the user terminal. 
     
     
       18. The method of  claim 1 , wherein the communications network is a packet-based network. 
     
     
       19. A user terminal comprising:
 a display; 
 an input device; 
 a network interface configured to transmit and receive video data between the user terminal and a communication network during a video call between the user terminal and at least one further user terminal; 
 a processor configured to run an application operable during said video call to:
 display a user interface on the display of the user terminal for display of received video frames; 
 detect selection of a selectable button displayed in said user interface via the input device; 
 receive a plurality of drawing inputs while the selectable button is in a selected state, each drawing input defining image data to be applied at a facial position on a first side of a face of a user displayed in the received video frames; 
 modifying each received video frame by applying the image data to the facial position on the first side of the face of the user displayed in the received video frame and applying the image data to the further facial position on the second side of the face of the user displayed in the received video frame; and 
 displaying the modified video frame in the user interface. 
 
 
     
     
       20. A computer-readable storage medium comprising instructions stored thereon that, responsive to execution by a processor of a user terminal during a video call between the user terminal and at least one further user terminal, is configured to:
 display a user interface on a display of the user terminal for display of received video frames; 
 detect selection of a selectable button displayed in said user interface; 
 receive a plurality of drawing inputs while the selectable button is in a selected state, each drawing input defining image data to be applied at a facial position on a first side of a face of a user displayed in the received video frames; 
 for each of the plurality of drawing inputs, determine a further facial position on a second side of the face, the further facial position on the second side of the face being symmetrically opposite to the facial position on the first side of said face; and 
 modify each received video frame by applying the image data to the facial position on the first side of the face of the user displayed in the received video frame and applying the image data to the further facial position on the second side of the face of the user displayed in the received video frame; and 
 display the modified video frame in the user interface.