Gaze-based workflow adaptation

Abstract

A method for gaze-based workflow adaptation includes identifying a workflow in which a user is engaged, where the workflow comprises a plurality of tasks that collectively achieves a desired goal, creating a trigger for a given task of the plurality of tasks, wherein the trigger specifies an action to be automatically taken in response to a gaze of the user meeting a defined criterion, monitoring a progress of the workflow, monitoring the gaze of the user, and sending a signal to a remote device in response to the gaze of the user meeting the defined criterion, wherein the signal instructs the remote device to take the action.

What is claimed is: 
     
         1 . A method comprising:
 identifying, by a processor, a workflow being performed by a user in a physical environment, wherein the workflow comprises a plurality of tasks that collectively achieves a desired goal with respect to an object in the physical environment, wherein a first trigger is stored for a first task of the plurality of tasks, wherein the first trigger specifies a first action to be automatically taken in response to an observed appearance of the object deviating from an expected appearance of the object during a same time that a gaze of the user is focused on the object during the first task of the workflow, wherein the expected appearance of the object is defined by a visual model of the object, and the first trigger specifies that the first action is to be taken when the observed appearance deviates from the visual model by more than a threshold, wherein a second trigger is stored for a second task of the plurality of tasks, wherein the second trigger specifies a second action to be automatically taken in response to the observed appearance of the object deviating from the expected appearance of the object during a same time that the gaze of the user is focused on the object during the second task of the workflow, and wherein the second trigger specifies that the second action is to be taken when the observed appearance deviates from the visual model by more than the threshold;   monitoring, by the processor, a progress of the workflow to determine a current task of the workflow that is being performed by the user;   monitoring, by the processor, the gaze of the user; and   sending, by the processor, a signal to a remote device in response to the observed appearance of the object deviating from the expected appearance of the object during a same time that the gaze of the user is focused on the object, wherein the signal instructs the remote device to take the first action in response to determining that the current task is the first task and to take the second action in response to determining that the current task is the second task.   
     
     
         2 . The method of  claim 1 , wherein the identifying comprises:
 analyzing, by the processor, data provided by a sensor that is positioned in a location of the user.   
     
     
         3 . The method of  claim 2 , wherein the sensor is integrated into a device that is worn by the user. 
     
     
         4 . The method of  claim 3 , wherein the device that is worn by the user comprises a head mounted display. 
     
     
         5 . The method of  claim 1 , wherein the identifying comprises:
 retrieving, by the processor, a service request identifying an action to be performed by the user.   
     
     
         6 . The method of  claim 1 , wherein the workflow is customized to accommodate a preference of the user with respect to at least one task of the plurality of tasks. 
     
     
         7 . The method of  claim 1 , further comprising:
 suggesting, by the processor, a modification to the workflow, based on an observation of another user who has performed the workflow.   
     
     
         8 . The method of  claim 7 , wherein the modification alters a way in which at least one task of the plurality of tasks is performed. 
     
     
         9 . The method of  claim 1 , further comprising:
 detecting, by the processor, that the workflow has been completed; and   performing, by the processor, a verification process to verify that the workflow has been performed correctly.   
     
     
         10 . The method of  claim 1 , further comprising:
 storing the workflow for future use in achieving the desired goal.   
     
     
         11 . The method of  claim 1 , wherein the user is a technician who has been dispatched on behalf of a service provider. 
     
     
         12 . The method of  claim 1 , wherein the user is an unmanned vehicle, and the gaze of the user comprises a focus of a camera integrated into the unmanned vehicle. 
     
     
         13 . The method of  claim 1 , wherein the workflow comprises a training simulation for the user. 
     
     
         14 . The method of  claim 11 , wherein the object comprises a piece of telecommunications equipment being serviced by the technician. 
     
     
         15 . The method of  claim 14 , wherein the method is performed by an extended reality-based application. 
     
     
         16 . The method of  claim 15 , wherein the extended reality-based application allows the technician to initiate an interactive dialog with a remote technician who is not present at the piece of telecommunications equipment. 
     
     
         17 . The method of  claim 12 , wherein the object comprises a piece of telecommunications equipment being serviced by the unmanned vehicle. 
     
     
         18 . The method of  claim 1 , wherein the object is a physical object and the observed appearance of the object comprises a real-time appearance of the physical object. 
     
     
         19 . A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform operations, the operations comprising:
 identifying a workflow being performed by a user in a physical environment, wherein the workflow comprises a plurality of tasks that collectively achieves a desired goal with respect to an object in the physical environment, wherein a first trigger is stored for a first task of the plurality of tasks, wherein the first trigger specifies a first action to be automatically taken in response to an observed appearance of the object deviating from an expected appearance of the object during a same time that a gaze of the user is focused on the object during the first task of the workflow, wherein the expected appearance of the object is defined by a visual model of the object, and the first trigger specifies that the first action is to be taken when the observed appearance deviates from the visual model by more than a threshold, wherein a second trigger is stored for a second task of the plurality of tasks, wherein the second trigger specifies a second action to be automatically taken in response to the observed appearance of the object deviating from the expected appearance of the object during a same time that the gaze of the user is focused on the object during the second task of the workflow, and wherein the second trigger specifies that the second action is to be taken when the observed appearance deviates from the visual model by more than the threshold;   monitoring a progress of the workflow to determine a current task of the workflow that is being performed by the user;   monitoring the gaze of the user; and   sending a signal to a remote device in response to the observed appearance of the object deviating from the expected appearance of the object during a same time that the gaze of the user is focused on the object, wherein the signal instructs the remote device to take the first action in response to determining that the current task is the first task and to take the second action in response to determining that the current task is the second task.   
     
     
         20 . A system comprising:
 a processor deployed in a telecommunication service provider network; and   a non-transitory computer-readable medium storing instructions which, when executed by the processor, cause the processor to perform operations, the operations comprising:
 identifying a workflow being performed by a user in a physical environment, wherein the workflow comprises a plurality of tasks that collectively achieves a desired goal with respect to an object in the physical environment, wherein a first trigger is stored for a first task of the plurality of tasks, wherein the first trigger specifies a first action to be automatically taken in response to an observed appearance of the object deviating from an expected appearance of the object during a same time that a gaze of the user is focused on the object during the first task of the workflow, wherein the expected appearance of the object is defined by a visual model of the object, and the first trigger specifies that the first action is to be taken when the observed appearance deviates from the visual model by more than a threshold, wherein a second trigger is stored for a second task of the plurality of tasks, wherein the second trigger specifies a second action to be automatically taken in response to the observed appearance of the object deviating from the expected appearance of the object during a same time that the gaze of the user is focused on the object during the second task of the workflow, and wherein the second trigger specifies that the second action is to be taken when the observed appearance deviates from the visual model by more than the threshold; 
 monitoring a progress of the workflow to determine a current task of the workflow that is being performed by the user; 
 monitoring the gaze of the user; and 
 sending a signal to a remote device in response to the observed appearance of the object deviating from the expected appearance of the object during a same time that the gaze of the user is focused on the object, wherein the signal instructs the remote device to take the first action in response to determining that the current task is the first task and to take the second action in response to determining that the current task is the second task.