Method and apparatus for determining the accuracy limit of a learning machine for predicting path performance degradation in a communications network

Abstract

A method and apparatus for determining the accuracy limit of a learning machine for predicting path performance degradation imposed by the quality of the path performance data is disclosed. A plurality of learning machines of increasing capacity are trained using training data and tested using test data, and the training error rates and test error rates are calculated. The asymptotic error rates of the learning machines are calculated and compared. When the change in asymptotic error rate falls below a certain rate, the asymptotic error rate estimates the accuracy limit for a learning machine for predicting path performance degradation. The accuracy limit is derived from insufficiencies in the path performance data and is applicable to any learning machine trained on and applied to the path performance data, regardless of the complexity of the learning machine or the size of the training data set.

We claim: 
     
       1. A method for determining an accuracy limit of a learning machine for predicting path performance degradation in a telecommunications network, said learning machine analyzing a performance data set for a path having an intrinsic noise level, said path performance data set comprising a training data set and a test data set, the method comprising the steps of: a) training a learning machine having a capacity to produce a classification of said path performance data set using said training data set and calculating a training error, said classification indicating whether said path will likely exhibit a performance degradation in the future;   b) testing said learning machine using said test data set and calculating a test error;   c) calculating an asymptotic error rate based on said test error and said training error;   d) determining whether a stop condition has been satisfied; and   e) if the stop condition is not satisfied, repeating steps a) through d) until said stop condition is satisfied, wherein each repetition of said steps is performed in conjunction with a learning machine having a capacity greater than that of the previous repetition of said steps.   
     
     
       2. The method of claim 1 wherein step c) further comprises calculating the mean of said test error and said training error. 
     
     
       3. The method of claim 1 wherein step d) further comprises comparing the relative difference of the calculated asymptotic error rates to a predetermined stop value. 
     
     
       4. The method of claim 3 wherein the comparing step further comprises comparing the relative difference of the calculated asymptotic error rates to a predetermined stop value which is in the range of 5 to 10 percent. 
     
     
       5. The method of claim 1 wherein said training data set has a size sufficient to have a training error that is greater than zero. 
     
     
       6. A method for determining an accuracy limit of a learning machine for predicting performance degradation of a path in a telecommunications network, said learning machine analyzing a path performance data set comprising training data and test data, the method comprising the steps of: a) training a learning machine having a capacity to produce a classification of the path performance data in said data set using said training data and calculating a training error, said classification indicating whether said path will likely exhibit a performance degradation in the future;   b) testing said learning machine using said test data and calculating a test error;   c) calculating an asymptotic error rate based on said training error and said test error;   d) determining whether a stop condition is satisfied;   e) if said stop condition is not satisfied then increasing the capacity of said learning machine and repeating steps a) through d) until said stop condition is satisfied; and   f) if said stop condition is satisfied then outputting said asymptotic error rate as the limit on learning machine accuracy imposed by said given path performance data set.   
     
     
       7. The method of claim 6 wherein said learning machine is a neural network having at least one hidden unit defining the capacity of said learning machine and wherein said step of increasing the capacity of said learning machine further comprises increasing the number of hidden units of said neural network. 
     
     
       8. The method of claim 6 wherein said learning machine is a learning vector quantization machine having at least one codebook vector defining the capacity of said learning machine and wherein the step of increasing the capacity of said learning machine further comprises increasing the number of codebook vectors of said learning vector quantization machine. 
     
     
       9. The method of claim 6 wherein the step of increasing the capacity of said learning machine further comprises increasing said capacity by a factor of 2. 
     
     
       10. The method of claim 6 wherein step d) further comprises comparing the relative difference of the calculated asymptotic error rates to a predetermined value. 
     
     
       11. The method of claim 10 wherein the comparing step further comprises comparing the relative difference of the calculated asymptotic error rates to a predetermined stop value which is in the range of 5 to 10 percent. 
     
     
       12. A system for determining an accuracy limit of a learning machine for predicting performance degradation of a path in a telecommunications network said accuracy limit being imposed by data integrity, said system comprising: a path performance data set having training data and test data;   a plurality of learning machines, each having a unique capacity, each of said learning machines comprising: a) means responsive to said data set for training said learning machine to produce a classification of said path performance data using said training data and for calculating a training error, said classification indicating whether said path will likely exhibit a performance degradation in the future;   b) means responsive to said data set for testing the classification ability of said learning machine using said test data and for calculating a test error;   c) means for calculating an asymptotic error rate based on said training error and said test error; and   d) means for comparing said asymptotic error rates to determine whether a stop condition has been reached and for outputting the asymptotic error rate as an estimated limit of the accuracy of said learning machine for predicting path performance degradation imposed by the data integrity of said path performance data when said stop condition is reached.     
     
     
       13. The system of claim 12 wherein said means for calculating the asymptotic error rate of said training error and said test error further comprises means for calculating the mean of said training error and said test error. 
     
     
       14. The system of claim 12 wherein said means for comparing said asymptotic error rates to determine whether a stop condition has been reached further comprises means for comparing the relative difference of said asymptotic error rates to a stop value. 
     
     
       15. The system of claim 14 wherein said stop value is in the range of 5 to 10 percent. 
     
     
       16. The system of claim 12 wherein said training data is of sufficient size such that said calculated training error is greater than zero. 
     
     
       17. An apparatus for determining an accuracy limit of a learning machine for predicting performance degradation of a path of a telecommunications network imposed by a data set comprising: a storage unit storing a path performance data set having training data and test data;   a memory unit containing computer program code;   a processor for executing said computer program code to implement separately a plurality of virtual learning machines, each implemented virtual learning machine having a different capacity;   means for training each of said implemented virtual learning machines to produce a classification of said path performance data set using said training data and for calculating a training error, said classification indicating whether said path will likely exhibit a performance degradation in the future;   means for testing the classification ability of each of said implemented virtual learning machines using said test data and for calculating a test error;   means for calculating the asymptotic error rate of said training error and said test error for each of said implemented virtual learning machines; and   means for comparing said asymptotic error rates to determine the accuracy limit of said learning machine for predicting path performance degradation imposed by said path performance data set.   
     
     
       18. The apparatus of claim 17 wherein said comparing means further comprises: means for comparing to a stop value the relative difference of said asymptotic error rates of a first implemented virtual learning machine and a second implemented virtual learning machine, the second implemented virtual learning machine having a capacity greater than the first implemented virtual learning machine; and   means for outputting the asymptotic error rate as the limit on learning machine accuracy imposed by said data set when said relative difference is equal to or less than said stop value.   
     
     
       19. The apparatus of claim 18 wherein said stop value is in the range of 5 to 10 percent. 
     
     
       20. The apparatus of claim 17 wherein said calculating means further comprises means for calculating the mean of said training error and said test error. 
     
     
       21. A system for determining an accuracy limit of a learning machine for predicting performance degradation of a communication path in a telecommunications network, said accuracy limit being imposed by data integrity, said system comprising: a performance monitor for monitoring performance parameters of said communication path and for generating path performance data which characterizes the error distribution of said communication path;   means for separating said generated path performance data into a training data set and a test data set;   a plurality of learning machines, each having a unique capacity, each of said learning machines comprising: a) means responsive to said path performance data set for training said learning machine to produce a classification of said path performance data using said training data and for calculating a training error, said classification indicating whether said path will likely exhibit a performance degradation in the future;   b) means responsive to said data set for testing the classification ability of said learning machine using said test data and for calculating a test error;   c) means for calculating an asymptotic error rate based on said training error and said test error; and     means for comparing said asymptotic error rates to determine whether a stop condition has been reached and for outputting the asymptotic error rate as an estimated limit of the accuracy of said learning machine for predicting path performance degradation imposed by the data integrity of said path performance data when said stop condition is reached.   
     
     
       22. The system of claim 21, wherein said performance monitor generates a bit error rate and a framing error rate for said communication path. 
     
     
       23. The system of claim 22, wherein said bit errors and said framing errors are included in said path performance data set only when said bit errors and said framing errors exceed one or more predefined thresholds. 
     
     
       24. The system of claim 21, wherein said training data set and said test data set evaluated by said learning machine are comprised of historical path performance data for a predefined historical period and a label indicating whether said communication path actually exhibited performance degradation in a predefined future period following said historical period.