Recurrent neural network and training process for same

Abstract

In a computer system that includes a trained recurrent neural network (RNN), a computer-based method includes: producing a copy of the trained RNN; producing a version of the RNN prior to any training; trying to solve a control task for the RNN with the copy of the trained RNN and with the untrained version of the RNN; and in response to the copy of the trained RNN or the untrained version of the RNN solving the task sufficiently well: retraining the trained RNN with one or more traces (sequences of inputs and outputs) from the solution; and retraining the trained RNN based on one or more traces associated with other prior control task solutions, as well as retraining the RNN based on previously observed traces to predict environmental inputs and other data (which maybe consequences of executed control actions).

What is claimed is: 
     
         1 . In a computer system that comprises a trained recurrent neural network (RNN), a computer-based method comprising:
 producing a copy of the trained RNN;   producing a version of the RNN prior to any training;   trying to solve a control task for the RNN with the copy of the trained RNN and with the untrained version of the RNN; and   in response to the copy of the trained RNN or the untrained version of the RNN solving the task sufficiently well:
 retraining the trained RNN with one or more traces (sequences of inputs and outputs) from the solution; and 
 retraining the trained RNN based on one or more traces associated with other prior task solutions; and 
 retraining the trained RNN based on previously observed traces to predict environmental inputs (including rewards) and other data (which maybe consequences of executed control actions). 
   
     
     
         2 . The computer-based method of  claim 1 , further comprising:
 designating a finite amount of time for trying to solve the task with the copy of the trained RNN and with the untrained version of the RNN; and   in response to the designated amount of time expiring, adding the task into an unsolved task set stored in a computer-based memory.   
     
     
         3 . The computer-based method of  claim 2 , wherein the copy of the trained RNN and the untrained version of the RNN try to solve the task in a parallel or interleaving manner within the designated amount of time. 
     
     
         4 . The computer-based method of  claim 1 , wherein trying to solve the task comprises:
 applying trial-based black box optimization to weights in the copy of the trained RNN and the untrained version of the RNN.   
     
     
         5 . The computer-based method of  claim 1 , further comprising:
 receiving the task to be solved at the computer system from a human user interacting with the computer system or with an agent of the computer system.   
     
     
         6 . The computer-based method of  claim 1 , further comprising:
 determining whether the task has been solved sufficiently well, by:   recognizing that the copy of the trained RNN or the untrained version of the RNN has solved the task at least once, if trials for the task are repeatable exactly, or   recognizing that the copy of the trained RNN or the untrained version of the RNN has solved the task some predetermined number or percentage of times, more than once, if trials for the task are not necessarily repeatable exactly.   
     
     
         7 . The computer-based method of  claim 1 , further comprising:
 if the task is solved and trials for the task are repeatable exactly, designating only a final trace of the solution as being relevant for retraining the trained RNN on the new task; or   if the task is solved and trials for the task are not necessarily repeatable exactly, designating more than one trace of the solution as being relevant for retraining the trained RNN on the new task.   
     
     
         8 . The computer-based method of  claim 1 , further comprising:
 utilizing the trace or traces marked as relevant for retraining the trained RNN on the new task.   
     
     
         9 . The computer-based method of  claim 1 , wherein retraining the trained RNN based on one or more traces associated with other prior task solutions comprises:
 retraining the trained RNN to reproduce input history-dependent outputs in all traces of all previously learned relevant behaviors that are still deemed useful; as well as   retraining the RNN based on previously observed traces to predict environmental inputs and other data (which maybe consequences of executed control actions).   
     
     
         10 . The computer-based method of  claim 9 , further comprising:
 applying criteria for assessing continued usefulness based on information stored in computer-based memory.   
     
     
         11 . The computer-based method of  claim 9 , wherein retraining the trained RNN utilizes gradient-based learning to reproduce the input history-dependent outputs in all traces of all previously learned relevant behaviors that are still deemed useful. 
     
     
         12 . The computer-based method of  claim 1 , further comprising:
 utilizing all traces, including those from failed trials, to retrain the trained RNN to make better predictions.   
     
     
         13 . The computer-based method of  claim 1 , further comprising performing one or more predictions and one or more controls with the trained RNN. 
     
     
         14 . The computer-based method of  claim 1 , wherein the trained RNN comprises:
 input units to receive input data about the real world outside of the trained from one or more electronic data sources;   model units to predict or model one or more aspects of the real world outside of the trained RNN based on the input data the trained RNN receives; and   controller units to interact with and/or control or influence one or more computer-based components in the trained RNN's external environment.   
     
     
         15 . The computer-based method of  claim 1 , further comprising:
 determining whether the system has spent a predetermined amount of time (c) trying to solve each task in an unsolved task set stored in computer-based memory; and   if so, trying to solve each task in the unsolved task set again, spending more time than (c) on each try.