Methods and systems for implementing dynamic neural networks

Abstract

A method is described for designing systems that provide efficient implementations of feed-forward, recurrent, and deep networks that process dynamic signals using temporal filters and static or time-varying nonlinearities. A system design methodology is described that provides an engineered architecture. This architecture defines a core set of network components and operations for efficient computation of dynamic signals using temporal filters and static or time-varying nonlinearities. These methods apply to a wide variety of connected nonlinearities that include temporal filters in the connections. Here we apply the methods to synaptic models coupled with spiking and/or non-spiking neurons whose connection parameters are determined using a variety of methods of optimization.

1 - 24 . (canceled) 
     
     
         25 . A computer implemented method for implementing an artificial neural network on a computing system, the artificial neural network comprising a plurality of nodes, each node having a node input and a node output, the method comprising:
 determining a desired dynamic function ƒ to be computed using the artificial neural network;   defining at least one node response function for the plurality of nodes of the artificial neural network, wherein the at least one node response function is defined to receive varying inputs over time at each node input and produce varying outputs over time at each node output;   associating at least one synaptic filter with each node in the plurality of nodes, wherein at least one particular synaptic filter in the at least one synaptic filter is associated with each node in the plurality of nodes;   determining an approximated dynamic function by approximating the desired dynamic function using Padé approximants;
 determining connection weights between the plurality of nodes in the artificial neural network using the at least one node response function and the at least one linear synaptic filter, wherein each connection weight is coupled to at least one of a corresponding node input or a corresponding node output and each connection weight is configured to be multiplied by at least one of the input received by the corresponding node input or the output produced by the corresponding node output; and 
   operating the artificial neural network on the computing system using the determined connected weights to compute the approximated dynamic function.   
     
     
         26 . The computer implemented method of  claim 25 , wherein the desired dynamic function ƒ includes at least one desired delay. 
     
     
         27 . The computer implemented method of  claim 26 , wherein the at least one desired delay is a controllable delay. 
     
     
         28 . The computer implemented method of  claim 26 , wherein the at least one desired delay comprises a plurality of delays arranged into a delay bank. 
     
     
         29 . The computer implemented method of  claim 28 , further comprising, for each delay in the plurality of delays, associating a delay weighting with that delay and weighting the output from that delay using the associated delay weighting. 
     
     
         30 . The computer implemented method of  claim 27 , wherein the at least one desired delay comprises a plurality of delays arranged into a delay bank. 
     
     
         31 . The computer implemented method of  claim 25 , wherein the desired dynamic function ƒ is defined to process multidimensional vector inputs. 
     
     
         32 . The computer implemented method of  claim 25 , wherein the desired dynamic function ƒ is defined to process auditory inputs and/or video inputs. 
     
     
         33 . The computer implemented method of  claim 25 , wherein the desired dynamic function ƒ is defined to implement an acausal filter. 
     
     
         34 . The computer implemented method of  claim 25 , wherein the computing system has a system-specific synaptic filter and determining the approximated dynamic function comprises:
 determining a modified dynamic function ƒ′ based on the desired dynamic function ƒ, wherein the modified dynamic function ƒ′ is determined to account for the system-specific synaptic filter; and   determining the Padé approximants from the modified dynamic function.   
     
     
         35 . A data processing system comprising:
 a non-transitory computer readable medium storing computer readable instructions and a data structure configured to compute a desired dynamic function, wherein the data structure comprises a plurality of nodes, each node having a node input and a node output, and the plurality of nodes being arranged into a plurality of layers of nodes including at least one input layer and at least one output layer; and   a computer processor operable to execute the computer readable instructions stored on the computer readable medium using the data structure to compute the desired dynamic function;   wherein the data structure is defined by
 determining the desired dynamic function ƒ to be computed by the plurality of nodes; 
 defining at least one node response function for the plurality of nodes, wherein the at least one node response function is defined to receive varying inputs over time at each node input and produce varying outputs over time at each node output; 
 associating at least one synaptic filter with each node in the plurality of nodes, wherein at least one particular synaptic filter in the at least one synaptic filter is associated with each node in the plurality of nodes; 
 determining an approximated dynamic function by approximating the desired dynamic function using Padé approximants; and 
 determining connection weights between the plurality of nodes in the artificial neural network using the at least one node response function and the at least one linear synaptic filter, wherein each connection weight is coupled to at least one of a corresponding node input or a corresponding node output and each connection weight is configured to be multiplied by at least one of the input received by the corresponding node input or the output produced by the corresponding node output, whereby the connection weights enable the plurality of nodes to compute the desired dynamic function ƒ based on input received at the at least one input layer. 
   
     
     
         36 . The data processing system of  claim 35 , wherein the desired dynamic function ƒ includes at least one desired delay. 
     
     
         37 . The data processing system of  claim 36 , wherein the at least one desired delay is a controllable delay. 
     
     
         38 . The data processing system of  claim 36 , wherein the at least one desired delay comprises a plurality of delays arranged into a delay bank. 
     
     
         39 . The data processing system of  claim 38 , wherein, for each delay in the plurality of delays:
 a delay weighting is associated with that delay; and   the output from that delay is weighted by the associated delay weighting.   
     
     
         40 . The data processing system of  claim 37 , wherein the at least one desired delay comprises a plurality of delays arranged into a delay bank. 
     
     
         41 . The data processing system of  claim 38 , wherein the desired dynamic function ƒ is defined to process multidimensional vector inputs. 
     
     
         42 . The data processing system of  claim 35 , wherein the desired dynamic function ƒ is defined to process auditory inputs and/or video inputs. 
     
     
         43 . The data processing system of  claim 35 , wherein the desired dynamic function ƒ is defined to implement an acausal filter. 
     
     
         44 . The data processing system of  claim 35 , wherein the processor has a system-specific synaptic filter and the approximated dynamic function is determined by:
 determining a modified dynamic function ƒ′ based on the desired dynamic function ƒ, wherein the modified dynamic function ƒ′ is determined to account for the system-specific synaptic filter; and   determining the Padé approximants from the modified dynamic function.