Automatically classifying animal behavior

Abstract

Systems and methods are disclosed to objectively identify sub-second behavioral modules in the three-dimensional (3D) video data that represents the motion of a subject. Defining behavioral modules based upon structure in the 3D video data itself— rather than using a priori definitions for what should constitute a measurable unit of action— identifies a previously-unexplored sub-second regularity that defines a timescale upon which behavior is organized, yields important information about the components and structure of behavior, offers insight into the nature of behavioral change in the subject, and enables objective discovery of subtle alterations in patterned action. The systems and methods of the invention can be applied to drug or gene therapy classification, drug or gene therapy screening, disease study including early detection of the onset of a disease, toxicology research, side-effect study, learning and memory process study, anxiety study, and analysis in consumer behavior.

The invention claimed is: 
     
       1. A method for analyzing motion of a subject, the method comprising:
 processing three-dimensional video image data frames representing the motion of the subject using a model-free algorithm to output a first and a second set of modules, wherein each module in the first set of modules exhibits pose dynamics for 200-900 milliseconds, and each module in the second set of modules exhibits pose dynamics for 200-900 milliseconds; and 
 identifying repeated modules of the subject's behavior using a template matching procedure for matching motifs of the subject's motion. 
 
     
     
       2. The method of  claim 1 , wherein each module in the first set of modules and the second set of modules exhibits pose dynamics satisfying a predetermined similarity threshold. 
     
     
       3. The method of  claim 2 , wherein the processing further comprises isolating the subject from a background in each frame of the three-dimensional video image data. 
     
     
       4. The method of  claim 3 , wherein the processing further comprises identifying an orientation of a feature of the subject on a set of frames of the three-dimensional video image data with respect to a coordinate system common to each frame. 
     
     
       5. The method of  claim 4 , wherein the processing further comprises modifying the orientation of the subject in at least a subset of the set of frames so that the feature is oriented in a same direction with respect to the coordinate system to output a set of aligned frames. 
     
     
       6. The method of  claim 5 , wherein the processing further comprises reducing dimensionality of the three-dimensional video image data frames using a principal component analysis (PCA) to output pose dynamics data representing a pose of the subject through principal component space. 
     
     
       7. The method of  claim 5 , wherein the processing further comprises reducing dimensionality of the three-dimensional video image data frames using a random projections technique. 
     
     
       8. The method of  claim 1 , wherein the processing further comprises identifying a time period of a behavior module of the subject by computing an auto-correlogram to evaluate timescale over which the subject's behavior is self-repeating. 
     
     
       9. The method of  claim 1 , wherein the processing further comprises performing a power-spectral density (PSD) analysis on the subject's behavioral data to further analyze its time domain structure. 
     
     
       10. The method of  claim 1 , wherein the processing further comprises automatically locating changepoints for transition periods between modules using a filtered derivative algorithm. 
     
     
       11. A non-transitory computer-readable medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method for analyzing motion of a subject, the method comprising:
 processing three-dimensional video image data frames representing the motion of the subject using a model-free algorithm to output a first and a second set of modules, wherein each module in the first set of modules exhibits pose dynamics for 200-900 milliseconds, and each module in the second set of modules exhibits pose dynamics for 200-900 milliseconds; and 
 identifying repeated modules of the subject's behavior using a template matching procedure for matching motifs of the subject's motion. 
 
     
     
       12. A system for recording motion of a subject and parsing recorded image data into sets of frames that represent different behaviors of the subject, the system comprising:
 a three-dimensional video camera configured to output video image data representing the motion of the subject; 
 a memory in communication with the three-dimensional video camera and having a non-transitory machine-readable storage medium with a machine-executable instruction set stored thereon; and 
 a control system comprising one or more processors coupled to the memory, the one or more processors configured to execute the machine-executable instruction set to cause the control system to:
 process frames obtained using the video image data, the frames representing the motion of the subject, using a model-free algorithm to output a first and a second set of modules, wherein each module in the first set of modules exhibits pose dynamics for 200-900 milliseconds, and each module in the second set of modules exhibits pose dynamics for 200-900 milliseconds; and 
 identify repeated modules of the subject's behavior using a template matching procedure for matching motifs of the subject's motion. 
 
 
     
     
       13. The system of  claim 12 , wherein the control system is further configured to:
 pre-process the video image data to isolate the subject from a background of the video image data; and 
 identify an orientation of a feature of the subject on a set of frames of the video image data with respect to a coordinate system common to each frame. 
 
     
     
       14. The system of  claim 13 , wherein the control system is further configured to modify the orientation of the feature of the subject in at least a subset of the set of frames so that the feature is oriented in a same direction with respect to the coordinate system to output a set of aligned frames. 
     
     
       15. The system of  claim 14 , wherein the control system is further configured to:
 process the set of aligned frames using a principal component analysis to output pose dynamics data for each frame of the set of aligned frames, wherein the pose dynamics data represents a pose of the subject for each aligned frame through principal component space; and 
 process, using the control system, the set of aligned frames to temporally segment the pose dynamics data into a plurality of sets of modules, wherein each module within a set of modules exhibits similar pose dynamics satisfying a predetermined similarity threshold and comprises 200-900 milliseconds. 
 
     
     
       16. The system of  claim 15 , wherein the control system is further configured to store, in a database, each frame in the set of aligned frames referenced to its module. 
     
     
       17. The system of  claim 12 , wherein the control system is further configured to send, to a display, a representation of a sub-set of the first and second sets of modules that occur above a predetermined similarity threshold. 
     
     
       18. The system of  claim 12 , wherein the control system is further configured to automatically apply a behavior tag to video image data outputted from the three-dimensional video camera representing motion of a second subject. 
     
     
       19. The system of  claim 12 , wherein the video image data represents motion of the subject before and after administration of an agent to the subject, and wherein the control system is further configured to:
 determine a quantity of modules in the first and second sets of modules before administration of the agent to the subject; 
 determine a quantity of modules in the first and second sets of modules after administration of the agent to the subject; and 
 output an indication of a change in frequency of expression of the quantity of modules in each of the first and second sets of modules before and after administration of the agent to the subject. 
 
     
     
       20. The system of  claim 19 , wherein the agent is (i) a pharmaceutically-active compound, (ii) a visual stimulus, (iii) an auditory stimulus, or (iv) an odorant.