Detecting temporally related components of multi-modal signals

Abstract

A method detects objects in a scene over time. Sets of time-aligned features are extracted from multiple signals representing a scene over time; each signal is acquired using a different modality. Each set of time-aligned features is arranged as a vector in a matrix to which a first transform is applied to produce a compressed matrix. A second transform is applied to the compressed matrix to extract spatio-temporal profiles of objects occurring in the scene.

1. A method for detecting objects in a scene over time, comprising:
 extracting sets of time-aligned features from a plurality of signals representing a scene over time, each signal acquired using a different modality; 
 arranging each set of time-aligned features as a vector in a matrix; 
 applying a first transform to the matrix to produce a compressed matrix; and 
 applying a second transform to the compressed matrix to extract spatio-temporal profiles of objects occurring in the scene. 
 
   
   
     2. The method of  claim 1  in which the first transform uses principal component analysis. 
   
   
     3. The method of  claim 1  in which the second transform uses independent component analysis. 
   
   
     4. The method of  claim 1  in which each vector is arranged as a temporally ordered column in the matrix. 
   
   
     5. The method of  claim 3  in which the independent component analysis is applied to a convolutive mixture of features in each vector. 
   
   
     6. The method of  claim 1  in which the plurality of signals includes an audio signal and a video signal.