Unsupervised neural network classification with back propagation

Abstract

An unsupervised back propagation method for training neural networks. For a set of inputs, target outputs are assigned l's and O's randomly or arbitrarily for a small number of outputs. The learning process is initiated and the convergence of outputs towards targets is monitored. At intervals, the learning is paused, and the values for those targets for the outputs which are converging at a less than average rate, are changed (e.g., 0→1, or 1→0), and the learning is then resumed with the new targets. The process is continuously iterated and the outputs converge on a stable classification, thereby providing unsupervised back propagation. In a further embodiment, samples classified with the trained network may serve as the training sets for additional subdivisions to grow additional layers of a hierarchical classification tree which converges to indivisible branch tips. After training is completed, such a tree may be used to classify new unlabelled samples with high efficiency. In yet another embodiment, the unsupervised back propagation method of the present invention may be adapted to classify fuzzy sets.

I claim: 
     
       1. A method for training a neural network model running on a computer system to generate classifications from a training sample set, said neural network model representing a neural network having an input layer an output layer and at least one hidden layer between the input layer and the output layer wherein each layer includes at least one node and wherein each connection between two nodes of successive layers is characterized by an internal weight, said method comprising the steps of: (a) arbitrarily assigning a numeric label to each training sample signal of a training sample signal set as a target vector signal for an output vector signal for classification, said numeric label selected from a set comprising a plurality of values;   (b) executing a back propagation-like training run by cycling through said training sample set, presenting each said training sample signal to an input of said model, the input representing said input layer, thereby generating said output vector signal for each said training sample signal, and modifying internal weight signals representing said internal weights based on the differences between said output vector signals and their corresponding target vector signals;   (c) estimating the convergence of components of said output vector signal towards respective components of said target vector signal;   (d) permuting said components of each said target vector signal according to said convergence, thereby updating said target vector signals;   (e) repeating steps (b), (c), and (d) until a predetermined condition is satisfied, thereby generating a final set of said internal weight signals to be used with said neural network model or said neural network for future classifications.   
     
     
       2. The method according to claim 1, wherein said permuting step comprises permuting said component for a given said target vector signal only where the rate of convergence of said output vector signal components towards said given target vector signal components is less than the average distance between corresponding target vector signal and output vector signal components. 
     
     
       3. The method according to claim 1, wherein said predetermined condition is satisfied when, for each sample, the difference between said target vector signals and said output vector signals are within a predetermined acceptable maximum error. 
     
     
       4. The method according to claim 1, wherein said assigning step comprises assigning binary numeric values to each said component of said target vector signals. 
     
     
       5. The method according to claim 1, further comprising the steps of: classifying additional samples with said neural network using said internal weight signals, thereby augmenting the number of classified samples, each classified sample an element of a sub-class;   classifying the members of each said sub-class according to steps (a) through (e), thereby generating further sub-classes and an additional independent set of said internal weights;   repeating said steps of classifying additional samples and classifying the members of each said sub-class, thereby generating a classification tree, until a predetermined condition for terminating classification is satisfied, said predetermined condition for terminating classification a separate condition from said predetermined condition; and   saving all said independent sets of internal weights for future classification with the hierarchical neural network classification tree.   
     
     
       6. The method according to claim 5, wherein said predetermined condition is satisfied when said step of classifying each said sub-class does not substantially yield further sub-classes. 
     
     
       7. The method according to claim 5, wherein the number of output nodes in said neural network is up to nine. 
     
     
       8. The method according to claim 5, further comprising, after classifying each said sub-class, the steps of generating a histogram of the output vectors, and dividing said sub-classes into further sub-classes by selecting additional thresholds according to said histogram. 
     
     
       9. The method according to claim 8, wherein the number of output nodes in said neural network is one, and one to nine of said thresholds are used for dividing said sub-classes into two to ten further sub-classes. 
     
     
       10. The method according to claim 1, wherein said computer system is a digital computer system. 
     
     
       11. The method according to claim 10, wherein said digital computer system is a serial computer. 
     
     
       12. The method according to claim 10, wherein said digital computer system is a parallel computer. 
     
     
       13. The method according to claim 1, wherein said computer system includes an analog computer. 
     
     
       14. The method according to claim 2 wherein said method is implemented as a neural network shell running on the computer system, enabling a program to run said neural network model. 
     
     
       15. The method according to claim 1 wherein said method is implemented as a neural network shell application program running on the computer system, enabling the application program to run said neural network model. 
     
     
       16. A method for training a neural network model running on a computer system having a processor and associated memory, said neural network model representing a neural network having an input layer an output layer and at least one hidden layer between the input layer and the output layer wherein each layer includes at least one node and wherein each connection between two nodes of successive layers is characterized by an internal weight, said method comprising the steps of: (a) arbitrarily assigning a numeric label to each training sample signal of a training sample signal set as a target vector signal for an output vector signal for classification, said numeric label selected from a set comprising a plurality of values, the target vector signals stored in the memory;   (b) executing a back propagation-like training run according to said processor cycling through said training sample set, presenting each said training sample signal to an input representing said input layer, thereby generating said output vector signal for each said training sample signal, and modifying internal weight signals stored in memory based on the differences between said output vector signals and their corresponding target vector signals, the stored internal weight signals representing the internal weights;   (c) estimating the convergence of components of said output vector signal towards respective components of said target vector signal;   (d) permuting said components of each said target vector signal according to said convergence, thereby updating said target vector signals;   (e) repeating steps (b), (c), and (d) until a predetermined condition is satisfied, thereby generating a final set of said internal weight signals for future classifications using said neural network model or the neural network represented by said neural network model.   
     
     
       17. A method for training a neural network according to a neural network model running on a computer system, said neural network model thereby becoming the neural network having an input layer an output layer and at least one hidden layer between the input layer and the output layer wherein each layer includes at least one node and wherein each connection between two nodes of successive layers is characterized by an internal weight signal, said method comprising the steps of: (a) arbitrarily assigning a numeric label to each training sample signal of a training sample signal set as a target vector signal for an output vector signal for classification, said numeric label selected from a set comprising a plurality of values;   (b) executing a back propagation-like training run by cycling through said training sample set, presenting each said training sample signal to said input layer, thereby generating said output vector signal for each said training sample signal, and modifying the internal weight signals of said neural network based on the differences between said output vector signals and their corresponding target vector signals;   (c) estimating the convergence of components of said output vector signal towards respective components of said target vector signal;   (d) permuting said components of each said target vector signal according to said convergence, thereby updating said target vector signals;   (e) repeating steps (a), (c), and (d) until a predetermined condition is satisfied, thereby generating a final set of said internal weight signals to be applied to the neural network model or to the neural network for future classifications.   
     
     
       18. A method for training a neural network model running on a computer system to generate classifications from a training sample set, said neural network model representing a neural network having an input layer, an output layer and at least one hidden layer between the input layer and the output layer wherein each layer includes at least one node and wherein each connection between two nodes of successive layers is characterized by an internal weight, said method comprising the steps of: (a) arbitrarily assigning a numeric label to each training sample signal of a training sample signal set as a target vector signal for an output vector signal for classification, said numeric label selected from a set comprising a plurality of values;   (b) initializing values of the internal weight signals of said neural network;   (c) executing a back propagation-like training run by cycling through said training sample set, presenting each said training sample signal to an input of said neural network model, the input representing said input layer, thereby generating said output vector signal for each training sample signal and modifying the internal weight signals representing said internal weights based on the differences between said output vector signals and their corresponding target vector signals;   (d) estimating the convergence of the components of said output vector signal towards the respective components of said target vector signal;   (e) permuting said components of each said target vector signal according to said convergence, thereby updating said target vector signals;   (f) repeating steps (c), (d), and (e) until a first predetermined condition is satisfied, upon which condition said internal weight signals are reinitialized;   (g) repeating steps (c), (d), (e), and (f) until a second predetermined condition is satisfied, thereby generating a final set of said internal weight signals to be applied to be used with said neural network model or said neural network for future classifications.   
     
     
       19. The method according to claim 18, wherein said first predetermined condition is satisfied when the difference between said target vector signals and said output vector signals is within a predetermined acceptable maximum error for a predetermined fraction of said samples. 
     
     
       20. The method according to claim 18, wherein said first predetermined condition includes any one of a plurality of conditions which may be satisfied. 
     
     
       21. The method according to claim 20, wherein a predetermined one of said plurality of conditions is tested for satisfaction at any given occurrence of step (f), and after any one of said plurality of conditions is satisfied. 
     
     
       22. An apparatus for training a neural network or corresponding neural network model characterized by an input layer an output layer and at least one hidden layer between the input layer and the output layer wherein each layer includes at least one node and wherein each connection between two nodes of successive layers is characterized by an internal weight, said apparatus comprising: a first memory for storing internal weight signals representing said internal weights;   a second memory for storing target vector signals obtained from arbitrarily assigning a numeric label to each training sample signal of a training sample signal set as a target value for an output vector signal, said numeric label selected from a set comprising a plurality of values; and   a processor coupled to said first memory and said second memory, and adapted to: execute a back propagation-like training run by cycling through said training sample set, presenting each said training sample signal to said input layer, thereby generating said output vector signal for each said training sample signal, and modifying the stored internal weight signals based on the differences between said output vector signals and their corresponding target vector signals;   estimate the convergence of components of said output vector signal towards respective components of said target vector signal;   permute said components of each said target vector signal according to said convergence, thereby updating said target vector signals;   repeat steps (c), (d), and (e) until a predetermined condition is satisfied, thereby generating a final set of said internal weight signals for application to the neural network or the neural network model for future classifications.     
     
     
       23. The apparatus according to claim 22, wherein said processor is under program control.