Condensed memory networks

Abstract

Techniques are described herein for training and applying memory neural networks, such as “condensed” memory neural networks (“C-MemNN”) and/or “average” memory neural networks (“A-MemNN”). In various embodiments, the memory neural networks may be iteratively trained using training data in the form of free form clinical notes and clinical reference documents. In various embodiments, during each iteration of the training, a so-called “condensed” memory state may be generated and used as part of the next iteration. Once trained, a free form clinical note associated with a patient may be applied as input across the memory neural network to predict one or more diagnoses or outcomes of the patient.

What is claimed is: 
     
         1 . A method for training a memory neural network implemented by one or more processors, comprising:
 applying, as training data, a plurality of free form clinical notes and a plurality of clinical reference documents as input across the memory neural network, wherein the applying includes multiple iterations for each instance of the training data, wherein each of the plurality of free form clinical notes includes one or more clinical observations about a patient in textual form, and wherein each clinical reference document describes one or more diagnoses and one or more associated expected clinical observations;   for each iteration of the applying:   generating a memory state associated with the memory neural network, and   reducing a dimensionality of the memory state to generate a condensed memory state;   and   modifying, based on a last memory state and a last condensed memory state generated during a last iteration of the applying, one or more weights associated with the memory neural network;   wherein after the applying and modifying, application of a subsequent free form clinical note associated with a subsequent patient across the memory neural network generates output indicative of one or more predicted diagnoses associated with the patient.   
     
     
         2 . The method of  claim 1 , wherein each of the plurality of free form clinical notes of the training data is labeled with one or more diagnoses. 
     
     
         3 . The method of  claim 1 , wherein reducing the dimensionality includes concatenating data generated from a previous memory state associated with the memory neural network with a current condensed memory state. 
     
     
         4 . The method of  claim 1 , wherein reducing the dimensionality condenses the memory state to half of its original dimension. 
     
     
         5 . The method of  claim 1 , wherein reducing the dimensionality includes computing a weighted average of all previous memory states. 
     
     
         6 . The method of  claim 1 , wherein generating the memory state associated with the memory neural network comprises applying a previous memory state as input across a multi-layer feed-forward neural network with a sigmoid output layer to generate, as output, a weighted sum of memory slots. 
     
     
         7 . At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:
 applying a given free form clinical note associated with a given patient as input across a trained memory neural network to generate output, wherein the memory neural network is trained using the following operations:
 applying, as training data, a plurality of free form clinical notes and a plurality of clinical reference documents as input across the memory neural network, wherein the applying includes, for each instance of the training data, multiple iterations, wherein each of the plurality of free form clinical notes includes one or more clinical observations about a patient in textual form, and wherein each clinical reference document describes one or more diagnoses and one or more associated expected clinical observations; 
 for each iteration of the applying:
 generating a memory state associated with the memory neural network, and 
 reducing a dimensionality of the memory state to generate a condensed memory state; 
 
 modifying, based on a last memory state and a last condensed memory state generated during a last iteration of the applying, one or more weights associated with the memory neural network; 
   predicting, based on the output, one or more diagnoses associated with the patient; and   providing output at one or more output devices, wherein the output is indicative of one or more of the predicted diagnoses.   
     
     
         8 . The non-transitory computer-readable medium of  claim 7 , wherein each of the plurality of free form clinical notes of the training data is labeled with one or more diagnoses. 
     
     
         9 . The non-transitory computer-readable medium of  claim 7 , wherein reducing the dimensionality includes concatenating data generated from a previous memory state associated with the memory neural network with a current condensed memory state. 
     
     
         10 . The non-transitory computer-readable medium of  claim 9 , wherein the data generated from the previous memory state includes a sum of a previous memory state and an output memory representation. 
     
     
         11 . The non-transitory computer-readable medium of  claim 7 , wherein reducing the dimensionality includes computing a weighted average of all previous memory states. 
     
     
         12 . The non-transitory computer-readable medium of  claim 7 , wherein generating the memory state associated with the memory neural network comprises applying a previous memory state as input across a multi-layer feed-forward neural network with a sigmoid output layer to generate, as output, a weighted sum of memory slots. 
     
     
         13 . The non-transitory computer-readable medium of  claim 7 , wherein providing output comprises presenting one predicted diagnosis more conspicuously than others based on a probability associated with the one predicted diagnosis. 
     
     
         14 . A system comprising one or more processors and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:
 applying, as training data, a plurality of free form clinical notes and a plurality of clinical reference documents as input across a memory neural network, wherein the applying includes multiple iterations for each instance of the training data, wherein each of the plurality of free form clinical notes includes one or more clinical observations about a patient in textual form, and wherein each clinical reference document describes one or more diagnoses and one or more associated expected clinical observations;   for each iteration of the applying:   generating a memory state associated with the memory neural network, and   reducing a dimensionality of the memory state to generate a condensed memory state;   and   modifying, based on a last memory state and a last condensed memory state generated during a last iteration of the applying, one or more weights associated with the memory neural network;   wherein after the applying and modifying, application of a subsequent free form clinical note associated with a subsequent patient across the memory neural network generates output indicative of one or more predicted diagnoses associated with the patient.   
     
     
         15 . The system of  claim 14 , wherein each of the plurality of free form clinical notes of the training data is labeled with one or more diagnoses. 
     
     
         16 . The system of  claim 14 , wherein reducing the dimensionality includes concatenating data generated from a previous memory state associated with the memory neural network with a current condensed memory state. 
     
     
         17 . The system of  claim 16 , wherein the data generated from the current memory state includes a sum of a previous memory state and an output memory representation. 
     
     
         18 . The system of  claim 14 , wherein reducing the dimensionality includes computing a weighted average of all previous memory states. 
     
     
         19 . The system of  claim 14 , wherein generating the memory state associated with the memory neural network comprises applying a previous memory state as input across another neural network to generate, as output, a weighted sum of memory slots. 
     
     
         20 . The system of  claim 19 , wherein the another neural network comprises a feed-forward neural network.