Image analysis using neural networks for pose and action identification

Abstract

An apparatus for performing image analysis to identify human actions represented in an image, comprising: a joint-determination module configured to analyse an image depicting one or more people using a first computational neural network to determine a set of joint candidates for the one or more people depicted in the image; a pose- estimation module configured to derive pose estimates from the set of joint candidates that estimate a body configuration for the one or more people depicted in the image; and an action-identification module configured to analyse a region of interest within the image identified from the derived pose estimates using a second computational neural network to identify an action performed by a person depicted in the image.

What is claimed is: 
     
         1 . An apparatus for performing image analysis to identify human actions represented in an image, comprising:
 a joint-determination module configured to analyse an image depicting one or more people using a first computational neural network to determine a set of joint candidates for the one or more people depicted in the image;   a pose-estimation module configured to derive pose estimates from the set of joint candidates that estimate a body configuration for the one or more people depicted in the image; and   an action-identification module configured to analyse a region of interest within the image identified from the derived pose estimates using a second computational neural network to identify an action performed by a person depicted in the image.   
     
     
         2 . An apparatus as claimed in  claim 1 , wherein the region of interest defines a sub-region of the image, and action-identification module is configured to analyse only the sub-region of the image using the second computational neural network. 
     
     
         3 . An apparatus as claimed in  claim 1 , wherein the action-identification module is configured to analyse the region of interest to identify objects of a specified object class, and to identify the action in response to detecting an object of the specified class in the region of interest. 
     
     
         4 . An apparatus as claimed in  claim 1 , wherein the apparatus further comprises an image-region module configured to identify the region of interest within the image from the derived pose estimates. 
     
     
         5 . An apparatus as claimed in  claim 4 , wherein the region of interest bounds a specified subset of joints of a derived pose estimate. 
     
     
         6 . An apparatus as claimed in  claim 4 , wherein the region of interest bounds one or more derived pose estimates. 
     
     
         7 . An apparatus as claimed in  claim 4 , wherein the image-region module is configured to identify a region of interest that bounds terminal ends of a derived pose estimate, and the action-identification module is configured to analyse the identified region of interest using the second computational neural network to identify whether the person depicted in the image is holding an object of a specified class or not. 
     
     
         8 . An apparatus as claimed in  claim 1 , wherein the apparatus is configured to receive the image from a 2-D camera. 
     
     
         9 . An apparatus as claimed in  claim 1 , wherein the action-identification module is configured to identify the action from a class of actions including: scanning an item at a point-of-sale; and selecting an item for purchase. 
     
     
         10 . An apparatus as claimed in  claim 1 , wherein the first network is a convolutional neural network. 
     
     
         11 . An apparatus as claimed in  claim 1 , wherein the second network is a convolutional neural network. 
     
     
         12 . A method of image analysis to identify human actions represented in an image, comprising:
 analysing an image depicting one or more people using a first computational neural network to determine a set of joint candidates for the one or more people depicted in the image;   deriving pose estimates from the set of joint candidates that estimate a body configuration for the one or more people depicted in the image; and   identifying a region of interest within the image from the derived pose estimates and analysing the identified region of interest using a second computational neural network to identify an action performed by a person depicted in the image.   
     
     
         13 - 21 . (canceled) 
     
     
         22 . An apparatus for performing image analysis to identify human actions from one or more images, comprising:
 a joint-determination module configured to analyse one or more images each depicting one or more people using a first computational neural network to determine for each image a set of joint candidates for the one or more people depicted in the image;   a pose-estimation module configured to derive for each image pose estimates from the set of joint candidates that estimate a body configuration for the one or more people depicted in the image; and   an action-identification module configured to analyse the derived pose estimates for the one or more images using a second computational neural network and to identify an action performed by a person depicted in the one or more images.   
     
     
         23 . An apparatus as claimed in  claim 22 , the apparatus further comprising an extractor module configured to extract from each derived pose estimate values for a set of one or more parameters characterising the pose estimate, wherein the action-identification module is configured to use the second computational neural network to identify an action performed by a person depicted in the one or more images in dependence on the extracted parameter values. 
     
     
         24 . An apparatus as claimed in  claim 23 , wherein the set of one or more parameters relate to a specified subset of joints of the pose estimate. 
     
     
         25 . An apparatus as claimed in  claim 23 , wherein the set of one or more parameters comprises at least one of: joint position for specified joints; joint angles between specified connected joints; joint velocity for specified joints; and the distance between specified pairs of joints. 
     
     
         26 . An apparatus as claimed in  claim 22 , wherein the one or more images is a series of multiple images, and the action-identification module is configured to identify the action performed by the person depicted in the series of images from changes in their derived pose estimate over the series of images. 
     
     
         27 . An apparatus as claimed in  claim 23 , wherein the one or more images is a series of multiple images, and the action-identification module is configured to identify the action performed by the person depicted in the series of images from changes in their derived pose estimate over the series of images and wherein the action-identification module is configured to use the second computational neural network to identify an action performed by the person from the change in the extracted parameter values characterising their derived pose estimate over the series of images. 
     
     
         28 . An apparatus as claimed in  claim 22 , wherein each image of the one or more images depicts a plurality of people, and the action-identification module is configured to analyse the derived pose estimates for each of the plurality of people for the one or more images using the second computational neural network and to identify an action performed by each person depicted in the one or more images. 
     
     
         29 . An apparatus as claimed in  claim 22 , wherein the pose-estimate module is configured to derive the pose estimates from the set of joint candidates and further from imposed anatomical constraints on the joints. 
     
     
         30 - 45 . (canceled)