Visibility metrics in multi-view medical activity recognition systems and methods

Abstract

Visibility metrics in multi-view medical activity recognition systems and methods are described herein. In certain illustrative examples, a system access imagery of a scene of a medical session captured by a plurality of sensors from a plurality of viewpoints, the imagery including first imagery captured by a first sensor of the plurality of sensors from a first viewpoint of the plurality of viewpoints. The system determines, during the medical session and based on the first imagery, a value of an activity visibility metric for the first sensor. The system facilitates, based on the value of the activity visibility metric, adjusting the first viewpoint of the first sensor.

1 - 32 . (canceled) 
     
     
         33 . A system comprising:
 a memory storing instructions;   a processor communicatively coupled to the memory and configured to execute the instructions to:
 access imagery of a scene of a medical session captured by a plurality of sensors from a plurality of viewpoints, the imagery including first imagery captured by a first sensor of the plurality of sensors from a first viewpoint of the plurality of viewpoints; 
 determine, during the medical session and based on the first imagery, a value of an activity visibility metric for the first sensor; and 
 facilitate, based on the value of the activity visibility metric, adjusting the first viewpoint of the first sensor. 
   
     
     
         34 . The system of  claim 33 , wherein the facilitating adjusting the first viewpoint of the first sensor comprises providing an output to a robotic system to instruct the robotic system to change a pose of the first sensor. 
     
     
         35 . The system of  claim 33 , wherein the facilitating adjusting the first viewpoint of the first sensor comprises providing an output to a user to instruct the user to change a pose of the first sensor. 
     
     
         36 . The system of  claim 33 , wherein:
 the instructions comprise a machine learning model trained based on training imagery labeled with an activity of scenes captured in the training imagery; and   the determining the value of the activity visibility metric for the first sensor comprises using the machine learning model.   
     
     
         37 . The system of  claim 33 , wherein the processor is further configured to execute the instructions to:
 access additional imagery of the scene of the medical session captured by the plurality of sensors from another plurality of viewpoints, the additional imagery including second imagery captured by the first sensor from a second viewpoint different from the first viewpoint; and   determine, based on the additional imagery, an additional value of the activity visibility metric that is higher than the value of the activity visibility metric.   
     
     
         38 . The system of  claim 33 , wherein:
 the imagery of the scene includes:
 second imagery captured by a second sensor of the plurality of sensors from a second viewpoint of the plurality of viewpoints, and 
 third imagery captured by a third sensor of the plurality of sensors from a third viewpoint of the plurality of viewpoints; and 
   the processor is further configured to execute the instructions to:
 determine that the value of the activity visibility metric for the first sensor is below a threshold value of the activity visibility metric, and 
 use, based on the determining that the value of the activity visibility metric for the first sensor is below the threshold value of the activity visibility metric, a generative model to produce generated imagery based on the second imagery and the third imagery. 
   
     
     
         39 . The system of  claim 38 , wherein the generated imagery comprises imagery generated based on the first viewpoint, the generated imagery having a generated value of the activity visibility metric that is higher than the value of the activity visibility metric for the first sensor. 
     
     
         40 . The system of  claim 38 , wherein:
 the generated imagery comprises imagery generated based on a fourth viewpoint, the generated imagery having a generated value of the activity visibility metric that is higher than the value of the activity visibility metric for the first sensor; and   the facilitating adjusting the first viewpoint of the first sensor comprises providing an output comprising an instruction to change a pose of the first sensor to capture additional imagery of the scene from the fourth viewpoint.   
     
     
         41 . The system of  claim 38 , wherein:
 the processor is further configured to execute the instructions to:
 determine, based on the second imagery, a value of the activity visibility metric for the second sensor, 
 determine, based on the third imagery, a value of the activity visibility metric for the third sensor; and 
   the using the generative model to produce generated imagery based on the second imagery and the third imagery is further based on the values of the activity visibility metric for the second sensor and the third sensor being at least the threshold value of the activity visibility metric.   
     
     
         42 . The system of  claim 33 , wherein:
 the imagery of the scene includes second imagery captured by a second sensor of the plurality of sensors from a second viewpoint of the plurality of viewpoints;   the processor is further configured to execute the instructions to determine, based on the first imagery and the second imagery, an overall value of the activity visibility metric for the plurality of sensors; and   the facilitating adjusting the first viewpoint of the first sensor comprises adjusting the first viewpoint to improve the overall value of the activity visibility metric.   
     
     
         43 . The system of  claim 42 , wherein the facilitating adjusting the first viewpoint results in a lower value of the activity visibility metric for the first sensor and a higher overall value of the activity visibility metric for the plurality of sensors. 
     
     
         44 . The system of  claim 33 , wherein the value of the activity visibility metric represents a rating of how visible an activity of the scene is in the first imagery. 
     
     
         45 . A system comprising:
 a memory storing instructions;   a processor communicatively coupled to the memory and configured to execute the instructions to:
 access imagery of a scene of a medical session captured by a plurality of sensors from a plurality of viewpoints, the imagery including first imagery captured by a first sensor of the plurality of sensors from a first viewpoint of the plurality of viewpoints; 
 determine, based on the first imagery, a first classification of an activity of the scene; 
 determine, during the medical session and based on the first imagery, a value of an activity visibility metric for the first sensor; 
 determine that the value of the activity visibility metric for the first sensor is below a threshold value of the activity visibility metric; and 
 lower, based on the determining that the value of the activity visibility metric for the first sensor is below the threshold value of the activity visibility metric, a weighting of the first classification of the activity of the scene for determining an overall classification of the activity of the scene based on the imagery of the scene. 
   
     
     
         46 . The system of  claim 45 , wherein the processor is further configured to execute the instructions to facilitate, based on the determining that the value of the activity visibility metric for the first sensor is below the threshold value of the activity visibility metric, adjusting the first viewpoint of the first sensor. 
     
     
         47 . A method comprising:
 accessing, by a processor, imagery of a scene of a medical session captured by a plurality of sensors from a plurality of viewpoints, the imagery including first imagery captured by a first sensor of the plurality of sensors from a first viewpoint of the plurality of viewpoints;   determining, by the processor, during the medical session and based on the first imagery, a value of an activity visibility metric for the first sensor; and   facilitating, by the processor, based on the value of the activity visibility metric, adjusting the first viewpoint of the first sensor.   
     
     
         48 . The method of  claim 47 , wherein the facilitating adjusting the first viewpoint of the first sensor comprises providing an output to a robotic system to instruct the robotic system to change a pose of the first sensor. 
     
     
         49 . The method of  claim 47 , wherein the facilitating adjusting the first viewpoint of the first sensor comprises providing an output to a user to instruct the user to change a pose of the first sensor. 
     
     
         50 . The method of  claim 47 , wherein the determining the value of the activity visibility metric for the first sensor comprises using a machine learning model trained based on training imagery labeled with an activity of scenes captured in the training imagery. 
     
     
         51 . The method of  claim 47 , further comprising:
 accessing, by the processor, additional imagery of the scene of the medical session captured by the plurality of sensors from another plurality of viewpoints, the additional imagery including second imagery captured by the first sensor from a second viewpoint different from the first viewpoint; and   determining, by the processor, based on the additional imagery, an additional value of the activity visibility metric that is higher than the value of the activity visibility metric.   
     
     
         52 . The method of  claim 47 , wherein:
 the imagery of the scene includes:
 second imagery captured by a second sensor of the plurality of sensors from a second viewpoint of the plurality of viewpoints, and 
 third imagery captured by a third sensor of the plurality of sensors from a third viewpoint of the plurality of viewpoints; and 
   the method further comprises:
 determining, by the processor, that the value of the activity visibility metric for the first sensor is below a threshold value of the activity visibility metric, and 
 using, by the processor, based on the determining that the value of the activity visibility metric for the first sensor is below the threshold value of the activity visibility metric, a generative model to produce generated imagery based on the second imagery and the third imagery.