Human action recognition and assistance to AR device

Abstract

A method for operating a monitoring unit ( 100 ), configured to monitor a manipulation of at least one object ( 40, 45 ) by a user, the method comprising: —receiving, via a cellular network ( 60 ), actual user position data comprising an actual position of at least one portion of user used to manipulate the at least one object, and actual object position data comprising an actual position of the at least one object, —matching the actual user position data to predefined user position data provided at the monitoring unit ( 100 ), the predefined user position data indicating a correct position of the at least one portion of the user for manipulating the at least one object, and matching the actual object position data to predefined object position data provided at the monitoring unit, the predefined object position data indicating a correct position of the at least one object, —determining, based on the matching, whether the manipulation of the at least one object by the at least one portion of the user is correct or not.

The invention claimed is: 
     
         1 . A method for operating a monitoring unit configured to monitor a manipulation of at least a first object by a user, the method comprising:
 receiving, via a cellular network, i) actual user position data comprising an actual position of at least a first portion of the user, wherein the first portion of the user is used to manipulate the first object, and ii) actual object position data comprising an actual position of the first object;   using i) the actual user position data and the actual object position data and ii) a task list and information indicating the last task performed, selecting a position set from a database of position sets and retrieving the selected position set, wherein the retrieved position set comprises:
 a) predefined user position data indicating a correct position of the first portion of the user for manipulating the first object, and 
 b) predefined object position data indicating a correct position of the first object; and 
   determining, based on the predefined user position data and the predefined object position data included in the retrieved position set, whether the position of the first portion of the user relative to the object to be manipulated is correct.   
     
     
         2 . The method of  claim 1 , wherein determining whether the manipulation is correct or not comprises determining that the manipulation of the first object is not correct when the actual user position data differ from the predefined user position data by more than a threshold, wherein a notification is transmitted to the user over the network that the manipulation is not correct. 
     
     
         3 . The method of  claim 1 , further comprising determining in a sequence of steps to be carried out by the user a next step to be carried out by user, a next position of the first portion of the user in the next step, wherein the selecting comprises determining whether the actual user position data are in agreement with the next position. 
     
     
         4 . The method of  claim 1 , further comprising determining, based on the retrieved position set, an instruction for the user how to manipulate the first object, wherein the instruction is sent to the user. 
     
     
         5 . The method of  claim 4 , further comprising dividing an environment reachable by the user into different sections, the different sections comprising a preferred section in which the correct position of the first portion of the user is located, wherein information about the preferred section is transmitted to the user such that the information about the preferred section can be displayed to the user as augmented reality. 
     
     
         6 . The method of  claim 1 , further comprising determining more detailed information about the first object describing the first object in more detail, wherein the more detailed information is sent to the user. 
     
     
         7 . The method of  claim 1 , wherein the selecting comprises:
 determining, in a sequence of steps to be carried out by the user the next step to be carried out by user, a next position of the first portion of the user in the next step; and   determining whether the next position is within a threshold distance to a best matching position of the portion of the user.   
     
     
         8 . The method of  claim 7 , further comprising indicating to the user the next position of the first portion of the user. 
     
     
         9 . The method of  claim 1 , wherein the selecting comprises at least one of a nearest neighbor clustering and a k-means clustering. 
     
     
         10 . The method of  claim 1 , further comprising collecting the predefined object position data and the predefined user position data including monitoring a plurality of user manipulating the first object in a sequence of steps. 
     
     
         11 . The method of  claim 1 , further comprising:
 receiving first image data generated by first image sensor comprising 2 dimensional images of the user and its direct environment,   receiving second image data generated by a second image sensor different from the first image sensor, the second image data comprising further images comprising an additional depth information,   generating fused image data based on the first image data and the second image data in a common coordinate system, the fused image data comprising the actual position of the first portion of the user and the actual position of the first object in the common coordinate system, wherein receiving the actual user position data and the actual object position data comprises receiving the fused image data.   
     
     
         12 . The method of  claim 11 , wherein the actual position of the first portion is determined based on the second image data, wherein the actual position of the first object is determined based on the first image data. 
     
     
         13 . The method of  claim 11 , wherein the fused image data are 2 dimensional image data. 
     
     
         14 . A non-transitory computer readable medium storing a computer program comprising program code to be executed by at least one processing unit of a monitoring unit, wherein execution of the program code causes the monitoring unit to perform the method of  claim 1 . 
     
     
         15 . A monitoring unit configured to monitor a manipulation of at least a first object by a user, the monitoring unit comprising:
 a memory; and   at least one processing unit, the memory containing instructions executable by the at least one processing unit, wherein the monitoring unit is operative to perform a method comprising:   receiving, via a cellular network, i) actual user position data comprising an actual position of at least a first portion of the user, wherein the first portion of the user is used to manipulate the first object, and ii) actual object position data comprising an actual position of the first object;   using i) the actual user position data and the actual object position data and ii) a task list and information indicating the last task performed, selecting a position set from a database of position sets and retrieving the selected position set, wherein the retrieved position set comprises:
 a) predefined user position data indicating a correct position of the first portion of the user for manipulating the first object, and 
 b) predefined object position data indicating a correct position of the first object; and 
   determining, based on the predefined user position data and the predefined object position data included in the retrieved position set, whether the position of the first portion of the user relative to the object to be manipulated is correct.   
     
     
         16 . The monitoring unit of  claim 15 , further being operative, for determining whether the manipulation is correct or not, to determine that the manipulation of the first object is not correct when the actual user position data differ from the predefined user position data by more than a threshold, and to transmit a notification to the user over the network that the manipulation is not correct. 
     
     
         17 . The monitoring unit of  claim 15 , further being operative to determine in a sequence of steps to be carried out by the user a next step to be carried out by user, a next position of the first portion of the user in the next step, and to determine whether the actual user position data are in agreement with the next position. 
     
     
         18 . The monitoring unit of  claim 15 , further being operative, to determine, based on the retrieved position set, an instruction for the user how to manipulate the first object, and to send the instruction to the user. 
     
     
         19 . The monitoring unit  claim 15 , comprising a first monitoring module and a second monitoring module, wherein the first monitoring module is operative to receive the actual user position data and the actual object position data, and to carry out the selecting, wherein the second monitoring module is operative to receive the first and second image data and to generate the fused image data.