Reservoir computing neural networks based on synaptic connectivity graphs

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for implementing a reservoir computing neural network. In one aspect there is provided a reservoir computing neural network comprising: (i) a brain emulation sub-network, and (ii) a prediction sub-network. The brain emulation sub-network is configured to process the network input in accordance with values of a plurality of brain emulation sub-network parameters to generate an alternative representation of the network input. The prediction sub-network is configured to process the alternative representation of the network input in accordance with values of a plurality of prediction sub-network parameters to generate the network output. The values of the brain emulation sub-network parameters are determined before the reservoir computing neural network is trained and are not adjusting during training of the reservoir computing neural network.

1 . (canceled) 
     
     
         2 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement:
 a reservoir computing neural network configured to receive a network input and to generate a network output from the network input, the reservoir computing neural network comprising: (i) a brain emulation sub-network, and (ii) a prediction sub-network, wherein:
 the brain emulation sub-network is configured to process the network input in accordance with values of a plurality of brain emulation sub-network parameters to generate an alternative representation of the network input, comprising, for each time step after a first time step in a sequence of multiple time steps:
 receiving a brain emulation sub-network input for the time step that comprises: (i) the network input, and (ii) a brain emulation sub-network output generated by the brain emulation sub-network at a preceding time step in the sequence of time steps; 
 processing the brain emulation sub-network input for the time step using the brain emulation sub-network to generate a brain emulation sub-network output for the time step; 
 wherein the alternative representation of the network input comprises a brain emulation sub-network outputs generated at a last time step in the sequence of time steps; 
 
 the prediction sub-network is configured to process the alternative representation of the network input in accordance with values of a plurality of prediction sub-network parameters to generate the network output; 
 the values of the brain emulation sub-network parameters are derived from a synaptic resolution image of at least a portion of a brain of a biological organism and are not adjusted during training of the reservoir computing neural network; 
 the values of the prediction sub-network parameters are adjusted during training of the reservoir computing neural network. 
   
     
     
         3 . The system of  claim 2 , wherein the values of the brain emulation sub-network parameters are derived from the synaptic resolution image of the brain of biological organism by operations comprising:
 processing the synaptic resolution image of the brain to identify a plurality of synaptic connections depicted in the synaptic resolution image of the brain; and   processing the synaptic resolution image of the brain to generate a respective brain emulation sub-network parameter value for each of the plurality of synaptic connections depicted in the synaptic resolution image of the brain.   
     
     
         4 . The system of  claim 3 , wherein each of the plurality of synaptic connections is between a respective first neuron and a respective second neuron in the brain of the biological organism, and
 wherein processing the synaptic resolution image of the brain to generate a respective brain emulation sub-network parameter value for each of the plurality of synaptic connections depicted in the synaptic resolution image of the brain comprises, for each of the plurality of synaptic connections:
 determining: (i) a first tolerance region in the synaptic resolution image around a first neuron corresponding to the synaptic connection, and (ii) a second tolerance region in the synaptic resolution image around a second neuron corresponding to the synaptic connection; and 
 determining the brain emulation sub-network parameter value for the synaptic connection based on an area of overlap between the first tolerance region and the second tolerance region. 
   
     
     
         5 . The system of  claim 2 , wherein the synaptic resolution image of the brain of the biological organism is an electron microscope image. 
     
     
         6 . The system of  claim 2 , wherein the values of the brain emulation sub-network parameters are derived from a region of the synaptic resolution image of the brain that is predicted to correspond to a region of the brain having a particular biological function. 
     
     
         7 . The system of  claim 6 , wherein the particular biological function is a visual data processing function, an audio data processing function, or an odor data processing function. 
     
     
         8 . The system of  claim 2 , wherein the values of the prediction sub-network parameters are adjusted during training of the reservoir computing neural network to optimize an objective function. 
     
     
         9 . The system of  claim 8 , wherein the objective function includes a term characterizing a prediction accuracy of the reservoir computing neural network. 
     
     
         10 . The system of  claim 9 , wherein the term characterizing the prediction accuracy of the reservoir computing neural network comprises a cross-entropy loss term. 
     
     
         11 . The system of  claim 8 , wherein the objective function includes a term characterizing a magnitude of the values of the prediction sub-network parameters. 
     
     
         12 . The system of  claim 2 , wherein dropout regularization is applied to the brain emulation sub-network parameters during training of the reservoir computing neural network. 
     
     
         13 . The system of  claim 2 , wherein the reservoir computing neural network is configured to process a network input comprising image data, video data, audio data, odor data, point cloud data, magnetic field data, or a combination thereof. 
     
     
         14 . The system of  claim 2 , wherein the reservoir computing neural network is configured to generate a classification output that comprises a respective score for each of a plurality of classes. 
     
     
         15 . The system of  claim 2 , wherein a neural network architecture of the prediction sub-network is less complex than the neural network architecture of the brain emulation sub-network. 
     
     
         16 . The system of  claim 15 , wherein the prediction sub-network comprises only a single neural network layer. 
     
     
         17 . The system of  claim 2 , wherein the biological organism is an animal. 
     
     
         18 . The system of  claim 17 , wherein the biological organism is a fly. 
     
     
         19 . The system of  claim 2 , wherein the values of the brain emulation sub-network parameters are determined based on weight values associated with synaptic connections between neurons in the brain of the biological organism. 
     
     
         20 . A method performed by one or more data processing apparatus, the method comprising:
 processing a network input using a reservoir computing neural network to generate a network output, wherein the reservoir computing neural network comprises: (i) a brain emulation sub-network, and (ii) a prediction sub-network, wherein:
 the brain emulation sub-network is configured to process the network input in accordance with values of a plurality of brain emulation sub-network parameters to generate an alternative representation of the network input, comprising, for each time step after a first time step in a sequence of multiple time steps:
 receiving a brain emulation sub-network input for the time step that comprises: (i) the network input, and (ii) a brain emulation sub-network output generated by the brain emulation sub-network at a preceding time step in the sequence of time steps; 
 processing the brain emulation sub-network input for the time step using the brain emulation sub-network to generate a brain emulation sub-network output for the time step; 
 wherein the alternative representation of the network input comprises a brain emulation sub-network outputs generated at a last time step in the sequence of time steps; 
 
 the prediction sub-network is configured to process the alternative representation of the network input in accordance with values of a plurality of prediction sub-network parameters to generate the network output; 
 the values of the brain emulation sub-network parameters are derived from a synaptic resolution image of at least a portion of a brain of a biological organism and are not adjusted during training of the reservoir computing neural network; 
 the values of the prediction sub-network parameters are adjusted during training of the reservoir computing neural network. 
   
     
     
         21 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers perform operations comprising:
 processing a network input using a reservoir computing neural network to generate a network output, wherein the reservoir computing neural network comprises: (i) a brain emulation sub-network, and (ii) a prediction sub-network, wherein:
 the brain emulation sub-network is configured to process the network input in accordance with values of a plurality of brain emulation sub-network parameters to generate an alternative representation of the network input, comprising, for each time step after a first time step in a sequence of multiple time steps:
 receiving a brain emulation sub-network input for the time step that comprises: (i) the network input, and (ii) a brain emulation sub-network output generated by the brain emulation sub-network at a preceding time step in the sequence of time steps; 
 processing the brain emulation sub-network input for the time step using the brain emulation sub-network to generate a brain emulation sub-network output for the time step; 
 wherein the alternative representation of the network input comprises a brain emulation sub-network outputs generated at a last time step in the sequence of time steps; 
 
 the prediction sub-network is configured to process the alternative representation of the network input in accordance with values of a plurality of prediction sub-network parameters to generate the network output; 
 the values of the brain emulation sub-network parameters are derived from a synaptic resolution image of at least a portion of a brain of a biological organism and are not adjusted during training of the reservoir computing neural network; 
 the values of the prediction sub-network parameters are adjusted during training of the reservoir computing neural network.