Methods, architecture, and apparatus for implementing machine intelligence and hierarchical memory systems

Abstract

Sophisticated memory systems and intelligent machines may be constructed by creating an active memory system with a hierarchical architecture. Specifically, a system may comprise a plurality of individual cortical processing units arranged into a hierarchical structure. Each individual cortical processing unit receives a sequence of patterns as input. Each cortical processing unit processes the received input sequence of patterns using a memory containing previously encountered sequences with structure and outputs another pattern. As several input sequences are processed by a cortical processing unit, it will therefore generate a sequence of patterns on its output. The sequence of patterns on its output may be passed as an input to one or more cortical processing units in next higher layer of the hierarchy. A lowest layer of cortical processing units may receive sensory input from the outside world. The sensory input also comprises a sequence of patterns.

1 . A memory system, said memory system comprising: 
 a hierarchy comprising a plurality of cortical processing units, each of said cortical processing units comprising 
 a first output, said first output for outputting a first set of pattern information,  
 a first input for receiving a second set of pattern information, said first input coupled to a sensory unit or to a first output from another cortical processing unit,  
 a memory, said memory storing information about patterns that may appear in said second set of pattern information on said first input, and  
 a processing method, said processing method comparing said second set of pattern information on said first input with said information about patterns stored in said memory.  
   
   
   
       2 . The memory system as claimed in  claim 1 , said memory system further comprising: 
 sensory units, said sensory units generating a third set of pattern information.    
   
   
       3 . The memory system as claimed in  claim 2  wherein a lowest layer of said hierarchy of cortical processing units is coupled to said sensory units.  
   
   
       4 . The memory system as claimed in  claim 1  wherein said pattern information comprises a sequence of patterns.  
   
   
       5 . The memory system as claimed in  claim 1  wherein said memory containing information about recognized patterns comprises a set of sequences of patterns that have been repeated on said input.  
   
   
       6 . The memory system as claimed in  claim 1 , said memory system further comprising: 
 a second input, said second input for receiving a prediction from a cortical processing unit in a higher layer of said hierarchy.    
   
   
       7 . The memory system as claimed in  claim 6  wherein said prediction comprises information about patterns in said memory.  
   
   
       8 . The memory system as claimed in  claim 6  wherein said prediction comprises an identifier of a next sequence patterns expected.  
   
   
       9 . The memory system as claimed in  claim 1 , said memory system further comprising: 
 a second output, said second output comprising prediction information.    
   
   
       10 . The memory system as claimed in  claim 9  wherein said second output is coupled to a cortical processing unit in a lower layer of said hierarchy.  
   
   
       11 . (canceled)  
   
   
       12 . A system, comprising: 
 a first level of modules, the modules each arranged to store a sequence of patterns; and    a second level of at least one module, the at least one module arranged to store a sequence of patterns, wherein the first level of modules and the second level of at least one module form at least part of a hierarchical network structure,    wherein, at least partially dependent on the sequences of patterns stored in the first level of modules and in the second level of at least one module, an object causing at least one pattern is determinable by information passing through the hierarchical network structure.    
   
   
       13 . The system of  claim 12 , wherein the stored sequence in each module in the first level was determinable over time, and wherein the stored sequence in the at least one module in the second level was determinable over time.  
   
   
       14 . The system of  claim 12 , wherein the object is determinable at least partially dependent on inference passing from a module in the first level to the at least one module in the second level.  
   
   
       15 . The system of  claim 14 , wherein the inference passing is at least partially dependent on a probability distribution generated in the module in the first level.  
   
   
       16 . The system of  claim 12 , wherein the object is determinable at least partially dependent on prediction passing from the at least one module in the second level to a module in the first level.  
   
   
       17 . The system of  claim 12 , wherein sequences stored in the modules in the first level and the at least one module in the second level each form at least a portion of an invariant representation of the object.  
   
   
       18 . The system of  claim 12 , wherein the hierarchical network structure comprises a Bayesian network structure.  
   
   
       19 . The system of  claim 12 , wherein any of the modules are at least one of at least partially represented in software and at least partially implemented in hardware.  
   
   
       20 . A method, comprising: 
 accessing a first module having a stored sequence of patterns;    accessing a second module having a stored sequence of patterns, wherein the first module and the second module form at least part of a hierarchical network; and    determining an object causing a pattern, the determining comprising at least one of: 
 passing first information from the first module to the second module dependent on the sequence stored in the first module, wherein the first information is indicative of at least one possible cause of the pattern, and  
 passing second information from the second module to the first module dependent on the sequence stored in the second module, wherein the second information is indicative of an expectation of a next pattern caused by the object.  
   
   
   
       21 . The method of  claim 20 , further comprising: 
 learning and storing the sequence of patterns in the first module; and    learning and storing the sequence of patterns in the second module.    
   
   
       22 . The method of  claim 21 , wherein any of the learning occurs over time.  
   
   
       23 . The method of  claim 20 , wherein the stored sequence of patterns in the first module and the stored sequence of patterns in the second module each form at least part of an invariant representation of the object.  
   
   
       24 . The method of  claim 20 , wherein the first module and the second module form part of a hierarchy of modules each arranged to perform a function substantially identical to the first module and the second module.  
   
   
       25 . The method of  claim 20 , wherein the hierarchical network comprises a Bayesian network.  
   
   
       26 . The method of  claim 20 , the determining further comprising: 
 passing third information from the second module to the first module, wherein the third information is arranged to modify a probability distribution in the first module.    
   
   
       27 . A method, comprising: 
 inputting a first sensed pattern to a first processing unit;    inputting a second sensed pattern to a second processing unit;    in response to inputting the first sensed pattern, determining a first distribution having at least one possible cause of the first sensed pattern dependent on at least one sequence of patterns stored in the first processing unit;    in response to inputting the second sensed pattern, determining a second distribution having at least one possible cause of the second sensed pattern dependent on at least one sequence of patterns stored in the second processing unit;    passing the first distribution and the second distribution to a third processing unit; and    determining a cause of the first sensed pattern and the second sensed pattern at least partially dependent on at least one operation performed on the first distribution and the second distribution by the third processing unit.    
   
   
       28 . The method of  claim 27 , wherein at least one of the first processing unit, the second processing unit, and the third processing unit is at least one of at least partially implemented in hardware and at least partially represented in software.  
   
   
       29 . The method of  claim 27 , further comprising: 
 determining a third distribution having at least one possible cause of the first sensed pattern and the second sensed pattern dependent on at least one sequence of patterns stored in the third processing unit;    passing the third distribution to the first processing unit and the second processing unit; and    modifying the first distribution and the second distribution dependent on the third distribution.    
   
   
       30 . The method of  claim 27 , further comprising: 
 inputting a first plurality of sequences of sensed patterns to the first processing unit;    inputting a second plurality of sequences of sensed patterns to the second processing unit;    the third processing unit determining at least one coincidence between the first plurality of sequences and the second plurality of sequences;    storing the at least one sequence of patterns in the first processing unit dependent on the at least one coincidence; and    storing the at least one sequence of patterns in the second processing unit dependent on the at least one coincidence.    
   
   
       31 . A system, comprising: 
 a first processing unit arranged to input a sensed pattern and further arranged to generate an inference as to at least one possible cause of the sensed pattern dependent on a plurality of sequences stored in the first processing unit, wherein the inference comprises a probability distribution; and    a second processing unit arranged to, dependent on the inference, generate a prediction as to a next inputted sensed pattern dependent on a plurality of sequences stored in the second processing unit,    wherein a subsequent inference by the first processing unit is dependent on the prediction.    
   
   
       32 . The system of  claim 31 , wherein the prediction is arranged to serve as a context for the subsequent inference.  
   
   
       33 . The system of  claim 31 , wherein at least one of the first processing unit and the second processing unit is at least one of at least partially implemented in hardware and at least partially represented in software.  
   
   
       34 . An apparatus, comprising: 
 a representation of a lower level of processing units, wherein the lower level processing units are arranged to output information regarding possible causes of sequences of sensed input patterns dependent on stored sequences previously learned by the lower level processing units; and    a representation of a higher level of at least one processing unit, wherein the at least one higher level processing unit is arranged to input the output information from the lower level processing units and output information regarding at least one possible cause of the sequences dependent on stored sequences previously learned by the higher level processing unit,    wherein the lower level of processing units and the higher level of at least one processing unit collectively represent at least part of a hierarchical network.    
   
   
       35 . The apparatus of  claim 34 , wherein the output information from the lower level processing units is dependent on output information from the at least one higher level processing unit.  
   
   
       36 . The apparatus of  claim 34 , wherein the hierarchical network comprises a Bayesian network.  
   
   
       37 . The apparatus of  claim 34 , wherein at least one of the representation of the lower level of processing units and the representation of the higher level of at least one processing unit is in at least one of hardware and software.  
   
   
       38 . A method, comprising: 
 storing sequences associated with an invariant structure of an object in a hierarchy of modules forming a hierarchical network;    receiving a sensed pattern;    first comparing the sensed pattern with a first part of the stored sequences;    determining at least one possible cause of the sensed pattern at least partially dependent on the comparing;    second comparing information associated with the at least one possible cause with a second part of the stored sequences, the second part representing a larger part of the invariant structure than the first part; and    determining the object as a cause of the sensed pattern at least partially dependent on the second comparing.    
   
   
       39 . The method of  claim 38 , wherein the hierarchical network comprises a Bayesian network.  
   
   
       40 . A system, comprising: 
 a hierarchical network of processing units, the hierarchical network having: 
 a first level of processing units each arranged to store at least one sequence of patterns associated with an object, and  
 a second level of at least one processing unit arranged to store at least one sequence of patterns associated with the object,  
   wherein a cause of input data received by the hierarchical network is determinable at least partially dependent on belief propagation through the hierarchical network, the belief propagation being at least partially dependent on the sequences stored in the first level of processing units and the second level of at least one processing unit.    
   
   
       41 . The system of  claim 40 , wherein the hierarchical network comprises a Bayesian network.  
   
   
       42 . The system of  claim 40 , wherein at least a portion of the hierarchical network is at least one of implemented in hardware and represented in software.  
   
   
       43 . The system of  claim 40 , wherein the cause comprises the object.