Remote support service with smart whiteboard

Abstract

Mobile devices, such as a smartphone, tablet or wearable, are leveraged to facilitate remote support between a user of the device, and a remote technician. In this approach, the mobile device is used as a tool on behalf of the remote technician. In particular, and in one embodiment, the mobile device is used to transfer images to the remote technician's desktop. The remote technician desktop is provisioned with a whiteboard functionality by which the remote technical draws on the camera image that the remote technician is then viewing. The resulting annotation(s) are then transferred back over the network and drawn on the end user's camera view. In this manner, the technician highlights any area of the camera image, and those highlights are visible on the end user mobile device regardless of whether the local end user rotates the view or moves the mobile device camera during the support session.

Having described our invention, what we now claim is as follows: 
     
         1 . Apparatus having a display interface, comprising:
 a processor;   computer memory holding computer program instructions executed by the processor during a remote support session established with a user of a mobile device, the computer program instructions operative to:
 receive and render on the display interface a first image frame, the first image frame having been generated at the mobile device; 
 receive accelerometer or gyroscope data generated at the mobile device; 
 capture data representing an annotation drawn over the first image frame being displayed on the display interface; 
 output the captured data to the mobile device to enable a representation of the annotation to be reproduced on a camera display of the mobile device; 
 as a second image frame distinct from the first image frame is received from the mobile device, compute transformation data representing a movement of the mobile device relative to a prior image frame generated at the mobile device, wherein the transformation data is computed at least in part on the accelerometer or gyroscope data received; and 
 output the adjusted transformation data to the mobile device to enable the at last one annotation to be re-drawn. 
   
     
     
         2 . The apparatus as described in  claim 1  wherein the computer program instructions are further operative to:
 store the annotation as a sequence of connected points, each point having an X,Y coordinate normalized with respect to a coordinate system whose origin is at a center of an image depicted in the first image frame. 
 
     
     
         3 . The apparatus as described in  claim 2  wherein normalized values within the coordinate system have maximum values at an outer perimeter of the image depicted in the first image frame. 
     
     
         4 . The apparatus as described in  claim 1  wherein the computer program instructions generate the transformation data as a transformation matrix having a set of elements whose values represent an action that is one of: a move operation, a scaling operation, a rotate around origin operation, and combinations thereof. 
     
     
         5 . The apparatus as described in  claim 1  wherein the accelerometer or gyroscope data generated at the mobile device is received with the first and second image frames over a WebRTC data flow. 
     
     
         6 . The apparatus as described in  claim 1  wherein the mobile device is one of: a smartphone, a tablet, and a wearable. 
     
     
         7 . The apparatus as described in  claim 1  wherein the image frames comprise a live camera feed generated by the user of the mobile device. 
     
     
         8 . The apparatus as described in  claim 7  wherein the remote support session is implemented over a peer-to-peer communication channel. 
     
     
         9 . Mobile device apparatus having a camera, comprising:
 a processor;   computer memory holding computer program instructions executed by the processor during a remote support session established with a support computing entity, the computer program instructions configured to:
 output accelerometer or gyroscope data; 
 render on a display interface at last a first image frame, the first image frame captured by the camera; 
 receive data from the support computing entity, the data being a representation of an annotation, the annotation having been drawn over the first image frame as the first image frame is displayed at the support computing entity; 
 render on the display interface the representation of the annotation; 
 output to the support computing entity a second image frame captured by the camera; 
 receive transformation data from the support computing entity, the transformation data having been generated at the support computing entity to represent a movement of the mobile device relative to a prior image frame captured by the camera, wherein the transformation data is computed at the support computing entity at least in part on the accelerometer or gyroscope data output; and 
 based on the transformation data, re-render on the display interface the representation of the annotation. 
   
     
     
         10 . The mobile device apparatus as described in  claim 9  wherein the transformation data is a transformation matrix having a set of elements whose values represent an action that is one of: a move operation, a scaling operation, a rotate around origin operation, and combinations thereof. 
     
     
         11 . The mobile device apparatus as described in  claim 9 , being one of: a smartphone, a tablet, and a wearable. 
     
     
         12 . The mobile device apparatus as described in  claim 9  wherein the image frames comprise a live camera feed generated by the mobile device camera. 
     
     
         13 . The mobile device apparatus as described in  claim 12  wherein the remote support session is implemented over a peer-to-peer communication channel. 
     
     
         14 . A method of remote support executed during a support session established between a first computing entity and a second computing entity, comprising:
 receiving at the second computing entity (i) an image feed generated at the first computing entity, the image feed associated with a target object having a support issue, together with (ii) accelerometer or gyroscope data generated at the first computing entity;   capturing one or more annotations generated at the second computing entity;   forwarding information representing the one or more annotations from the second computing entity to the first computing entity to enable rendering of the one or more annotations at the first computing entity in association with the image feed; and   providing transformation data from the second computing entity to the first computing entity, the transformation data adapted to be applied at the first computing entity to maintain position and orientation of the one or more annotations relative to the image feed irrespective of movement of the mobile device camera relative to the target object, wherein the transformation data is computed at the second computing entity at least in part on the accelerometer or gyroscope data.   
     
     
         15 . The method as described in  claim 14  wherein the first computing entity is one of: a smartphone, a tablet, and a wearable. 
     
     
         16 . The method as described in  claim 14  wherein the second computing entity is a technician support computer. 
     
     
         17 . The method as described in  claim 14  wherein the transformation data is a transformation matrix having a set of elements whose values represent an action that is one of: a move operation, a scaling operation, a rotate around origin operation, and combinations thereof.