Methods and apparatus to improve data quality for artificial intelligence

Abstract

Methods, apparatus, systems, and articles of manufacture to improve data quality for artificial intelligence are disclosed. An example apparatus includes an interface; instructions; and processor circuitry to execute the instruction to: determine an indirect quality of a repository that include datapoints of a dataset; determine a direct quality of the repository that include the datapoints of the dataset; determine a dataset quality based on the indirect quality of the repository and the direct quality of the repository; and when the quality does not satisfy a threshold, filter out a subset of the datapoints to prepare the dataset to support the training of the neural network.

What is claimed is: 
     
       1. An apparatus to support training a neural network, the apparatus comprising:
 an interface; 
 computer-readable instructions; and 
 programmable circuitry to utilize the computer-readable instructions to:
 determine an indirect quality of a repository based on at least one datapoint of a dataset; 
 determine a direct quality of the repository based on the at least one datapoint of the dataset; 
 determine a dataset quality of the dataset based on the indirect quality of the repository and the direct quality of the repository; 
 determine qualities of multiple datapoints from the dataset based on an average repository quality of repositories that include the datapoints, the average repository quality based on indirect qualities and direct qualities of the repositories that include the datapoints; 
 update the dataset by filtering out at least one data point in the dataset based on the qualities of the datapoints; and 
 train a neural network with the updated dataset. 
 
 
     
     
       2. The apparatus of  claim 1 , wherein the programmable circuitry is to:
 determine the indirect quality based on at least one of metadata corresponding to the repository; and 
 determine the direct quality based on a proportion of (a) a number of repositories that include a datapoint of the dataset to (b) a total number of the repositories. 
 
     
     
       3. The apparatus of  claim 2 , wherein the metadata includes at least one of ages of the repository, a total number of contributors to the repository, a total number of commits, a number of lines of code in the repository, a total number of open issues, a total number of closed issues, presence of a unit test in the code, presence of documentation in the code, a number of full continuous integration and delivery (CI/CD) runs style checks, or a number of CI/CD run unit test. 
     
     
       4. The apparatus of  claim 1 , wherein the programmable circuitry is to determine the dataset quality based on an average datapoint quality of the datapoints included in the dataset. 
     
     
       5. The apparatus of  claim 1 , wherein the at least one datapoint negatively affects the dataset quality. 
     
     
       6. The apparatus of  claim 1 , wherein the at least one datapoint positively affects the dataset quality. 
     
     
       7. The apparatus of  claim 1 , wherein the dataset quality is a first dataset quality, the programmable circuitry is to:
 determine a second dataset quality of the updated dataset after the at least one data point has been filtered out; and 
 filter out a second datapoint of the datapoints from the updated dataset to adjust the second dataset quality based on the second dataset quality not satisfying a threshold. 
 
     
     
       8. A non-transitory computer readable medium comprising instructions to cause one or more processors to at least:
 determine an indirect quality of a repository that includes at least one datapoint of a dataset; 
 determine a direct quality of the repository that includes the at least one datapoint of the dataset; 
 determine a dataset quality of the dataset based on the indirect quality of the repository and the direct quality of the repository; 
 determine qualities of multiple datapoints from the dataset based on an average repository quality of repositories that include the datapoints, the average repository quality based on indirect qualities and direct qualities of the repositories that include the datapoints; 
 update the dataset by removing at least one data point in the dataset based on the qualities of the datapoints; and 
 train an artificial intelligence-based model with the updated dataset. 
 
     
     
       9. The non-transitory computer readable medium of  claim 8 , wherein at least one of the one or more processors is to:
 determine the indirect quality based on at least one of metadata corresponding to the repository; and 
 determine the direct quality based on a proportion of (a) a number of repositories that include a datapoint of the dataset to (b) a total number of the repositories. 
 
     
     
       10. The non-transitory computer readable medium of  claim 9 , wherein the metadata includes at least one of ages of the repository, a total number of contributors to the repository, a total number of commits, a number of lines of code in the repository, a total number of open issues, a total number of closed issues, presence of a unit test in the code, presence of documentation in the code, a number of full continuous integration and delivery (CI/CD) runs style checks, or a number of CI/CD run unit test. 
     
     
       11. The non-transitory computer readable medium of  claim 8 , wherein at least one of the one or more processors is to determine the dataset quality based on an average datapoint quality of the datapoints included in the dataset. 
     
     
       12. The non-transitory computer readable medium of  claim 8 , wherein the at least one datapoint negatively affects the dataset quality. 
     
     
       13. The non-transitory computer readable medium of  claim 8 , wherein the at least one datapoint positively affects the dataset quality. 
     
     
       14. The non-transitory computer readable medium of  claim 8 , wherein the dataset quality is a first dataset quality, at least one of the one or more processors is to:
 determine a second dataset quality of the updated dataset after the at least one data point has been removed; and 
 in response to the second dataset quality not satisfying a threshold, remove a second datapoint to adjust the second dataset quality. 
 
     
     
       15. An apparatus to improve a dataset to train a neural network, the apparatus comprising:
 interface circuitry; and 
 programmable circuitry including one or more of:
 at least one of a central processing unit, a graphic processing unit or a digital signal processor, the at least one of the central processing unit, the graphic processing unit or the digital signal processor having control circuitry, one or more registers, and arithmetic and logic circuitry to perform one or more first operations corresponding to instructions in the apparatus, and; 
 a Field Programmable Gate Array (FPGA), the FPGA including logic gate circuitry, a plurality of configurable interconnections, and storage circuitry, the logic gate circuitry and interconnections to perform one or more second operations; or 
 Application Specific Integrate Circuitry (ASIC) including logic gate circuitry to perform one or more third operations; 
 the programmable circuitry to perform at least one of the first operations, the second operations or the third operations to instantiate:
 data quality determination circuitry to:
 determine an indirect quality of a repository including at least one datapoint of a dataset; 
 determine a direct quality of the repository; 
 determine a dataset quality of the dataset based on the indirect quality of the repository and the direct quality of the repository; and 
 determine qualities of multiple datapoints from the dataset based on an average repository quality of repositories that include the datapoints, the average repository quality based on indirect qualities and direct qualities of the repositories that include the datapoints; and 
 
 a filter to update the dataset by filtering out at least one data point in the dataset based on the qualities of the datapoints; and 
 a neural network to train using the updated dataset. 
 
 
 
     
     
       16. The apparatus of  claim 15 , wherein the data quality determination circuitry is to:
 determine the indirect quality based on at least one of metadata corresponding to the repository; and 
 determine the direct quality based on a proportion of (a) a number of repositories that include a datapoint of the dataset to (b) a total number of the repositories. 
 
     
     
       17. The apparatus of  claim 16 , wherein the metadata includes at least one of ages of the repository, a total number of contributors to the repository, a total number of commits, a number of lines of code in the repository, a total number of open issues, a total number of closed issues, presence of a unit test in the code, presence of documentation in the code, a number of full continuous integration and delivery (CI/CD) runs style checks, or a number of CI/CD run unit test. 
     
     
       18. The apparatus of  claim 15 , wherein the data quality determination circuitry is to determine the dataset quality of the dataset based on an average datapoint quality of the datapoints included in the dataset. 
     
     
       19. The apparatus of  claim 15 , wherein the dataset quality is a first dataset quality, wherein:
 the data quality determination circuitry is to determine a second dataset quality of the dataset after the at least one data point has been filtered out; and 
 the filter is to, in response to the second dataset quality not satisfying a threshold, filter out a second datapoint to adjust the second dataset quality. 
 
     
     
       20. The apparatus of  claim 15 , wherein the at least one datapoint negatively affects the dataset quality. 
     
     
       21. The apparatus of  claim 15 , wherein the at least one datapoint positively affects the dataset quality. 
     
     
       22. The apparatus of  claim 15 , wherein the at least one datapoint negatively affects the dataset quality. 
     
     
       23. The apparatus of  claim 15 , wherein the at least one datapoint positively affects the dataset quality.