Analysis operations on data in a native tabular data structure using a proxy data table

Abstract

In one example, a system can receive, from application code including an analysis operation performed on a set of data, an indication to access the set of data included in a tabular data structure using an application programming interface (API), in which the tabular data structure is associated with a memory allocation and a type. The system can determine that the type of the tabular data structure is the native type, the native type characterizing data structures that are accessed using a first programming language and a second programming language. The system can identify a proxy data table that shares the memory allocation, the proxy data table accessed using the API based on the second programming language. The system can issue one or more read commands to the proxy data table to cause the set of data to be read from the tabular data structure.

The invention claimed is: 
     
       1. A system comprising:
 one or more processors; and 
 one or more memories including program code that is executable by the one or more processors to:
 receive, from application code including an analysis operation performed on a set of data, a first indication to access the set of data included in a tabular data structure using a first application programming interface (API) based on a first programming language and a second programming language that is different from the first programming language, wherein the tabular data structure is associated with a first memory allocation and has a type; 
 in response to receiving the first indication:
 determine that the type of the tabular data structure is a native type, wherein the type is selected from among at least two types including the native type and a non-native type, wherein the native type characterizes data structures that are accessed using the first programming language and the second programming language and the non-native type characterizes data structures that are accessed using a data access layer and the second programming language; 
 identify a first proxy data table usable as a proxy for the tabular data structure, wherein the tabular data structure and the first proxy data table share the first memory allocation, wherein the first proxy data table is accessed using the first API based on the second programming language; and 
 issue one or more read commands, using the first API, to the first proxy data table to cause the set of data to be read from the tabular data structure; 
 
 generate, by the application code, output data; 
 receive, from the application code, a second indication to write the output data, the output data being a result of the analysis operation; 
 in response to receiving the second indication:
 generate an output data structure; and 
 issue one or more write commands to cause the output data to be written to the output data structure. 
 
 
 
     
     
       2. The system of  claim 1 , wherein:
 the output data structure is the native type; 
 the output data structure is accessed using the first API; and 
 issuing the one or more write commands to cause the output data to be written to the output data structure comprises:
 identifying a second proxy data table usable as a proxy for the output data structure, wherein the output data structure and the second proxy data table share a second memory allocation, wherein the second proxy data table is accessed using the first API; and 
 issuing the one or more write commands, using the first API, to the second proxy data table to cause the output data to be written to the output data structure. 
 
 
     
     
       3. The system of  claim 1 , wherein:
 the output data structure is the non-native type; 
 the output data structure is accessed using a second API based on the second programming language; and 
 issuing the one or more write commands to cause the output data to be written to the output data structure comprises issuing the one or more write commands to cause the data access layer to write the output data to the output data structure. 
 
     
     
       4. The system of  claim 1 , wherein the first programming language is Python and the second programming language is C. 
     
     
       5. The system of  claim 1 , wherein the tabular data structure is a Python data frame. 
     
     
       6. The system of  claim 1 , wherein the tabular data structure comprises one or more columns and the first memory allocation is a column buffer. 
     
     
       7. The system of  claim 6 , wherein:
 the tabular data structure is accessed using a column-based native API, wherein the column-based native API is based on the first programming language; and 
 issuing the one or more read commands to the first proxy data table to read the set of data from the tabular data structure comprises, for each column in the one or more columns of the tabular data structure, accessing the column buffer to read the set of data. 
 
     
     
       8. The system of  claim 7 , wherein identifying the first proxy data table comprises:
 generating the first proxy data table, comprising:
 for each column of the tabular data structure, determining column information; 
 generating a tabular data structure schema based on the column information; 
 identifying a name for the first proxy data table; and 
 generating the first proxy data table using the name and the tabular data structure schema. 
 
 
     
     
       9. The system of  claim 1 , further including additional program code that is executable by the one or more processors to:
 determine information about the tabular data structure, comprising:
 accessing the set of data using a data access management object; 
 generating the tabular data structure using the data access management object, wherein the type of the tabular data structure is the non-native type, wherein the non-native type characterizes data structures that are accessed using the second programming language; and 
 determining the information about the tabular data structure. 
 
 
     
     
       10. The system of  claim 1 , further including additional program code that is executable by the one or more processors to:
 determine information about the tabular data structure, comprising:
 accessing the set of data using a tabular data structure library; 
 generating the tabular data structure using a data access management object, wherein the type of the tabular data structure is the native type; and 
 determining the information about the tabular data structure. 
 
 
     
     
       11. A method comprising:
 receiving, from application code including an analysis operation performed on a set of data, a first indication to access the set of data included in a tabular data structure using a first application programming interface (API) based on a first programming language and a second programming language that is different from the first programming language, wherein the tabular data structure is associated with a first memory allocation and has a type; 
 in response to receiving the first indication:
 determining that the type of the tabular data structure is a native type, wherein the type is selected from among at least two types including the native type and a non-native type, wherein the native type characterizes data structures that are accessed using the first programming language and the second programming language and the non-native type characterizes data structures that are accessed using a data access layer and the second programming language; 
 identifying a first proxy data table usable as a proxy for the tabular data structure, wherein the tabular data structure and the first proxy data table share the first memory allocation, wherein the first proxy data table is accessed using the first API based on the second programming language; and 
 issuing one or more read commands, using the first API, to the first proxy data table to cause the set of data to be read from the tabular data structure; 
 
 generating, by the application code, output data; 
 receiving, from the application code, a second indication to write the output data, the output data being a result of the analysis operation; 
 in response to receiving the second indication:
 generating an output data structure; and 
 issuing one or more write commands to cause the output data to be written to the output data structure. 
 
 
     
     
       12. The method of  claim 11 , wherein:
 the output data structure is the native type; 
 the output data structure is accessed using the first API; and 
 issuing the one or more write commands to cause the output data to be written to the output data structure comprises:
 identifying a second proxy data table usable as a proxy for the output data structure, wherein the output data structure and the second proxy data table share a second memory allocation, wherein the second proxy data table is accessed using the first API; and 
 issuing the one or more write commands, using the first API, to the second proxy data table to cause the output data to be written to the output data structure. 
 
 
     
     
       13. The method of  claim 11 , wherein:
 the output data structure is the non-native type; 
 the output data structure is accessed using a second API based on the second programming language; and 
 issuing the one or more write commands to cause the output data to be written to the output data structure comprises issuing the one or more write commands to cause the data access layer to write the output data to the output data structure. 
 
     
     
       14. The method of  claim 11 , wherein the first programming language is Python and the second programming language is C. 
     
     
       15. The method of  claim 11 , wherein the tabular data structure is a Python data frame. 
     
     
       16. The method of  claim 11 , wherein the tabular data structure comprises one or more columns and the first memory allocation is a column buffer. 
     
     
       17. The method of  claim 16 , wherein:
 the tabular data structure is accessed using a column-based native API, wherein the column-based native API is based on the first programming language; and 
 issuing the one or more read commands to the first proxy data table to read the set of data from the tabular data structure comprises, for each column in the one or more columns of the tabular data structure, accessing the column buffer to read the set of data. 
 
     
     
       18. The method of  claim 17 , wherein identifying the first proxy data table comprises:
 generating the first proxy data table, comprising:
 for each column of the tabular data structure, determining column information; 
 generating a tabular data structure schema based on the column information; 
 identifying a name for the first proxy data table; and 
 generating the first proxy data table using the name and the tabular data structure schema. 
 
 
     
     
       19. The method of  claim 11 , further comprising:
 determining information about the tabular data structure, comprising:
 accessing the set of data using a data access management object; 
 generating the tabular data structure using the data access management object, wherein the type of the tabular data structure is the non-native type, wherein the non-native type characterizes data structures that are accessed using the second programming language; and 
 determining the information about the tabular data structure. 
 
 
     
     
       20. The method of  claim 11 , further comprising:
 determining information about the tabular data structure, comprising:
 accessing the set of data using a tabular data structure library; 
 generating the tabular data structure using a data access management object, wherein the type of the tabular data structure is the native type; and 
 determining the information about the tabular data structure. 
 
 
     
     
       21. A non-transitory computer-readable medium comprising program code that is executable by one or more processors for causing the one or more processors to:
 receive, from application code including an analysis operation performed on a set of data, a first indication to access the set of data included in a tabular data structure using a first application programming interface (API) based on a first programming language and a second programming language that is different from the first programming language, wherein the tabular data structure is associated with a first memory allocation and has a type; 
 in response to receiving the first indication:
 determine that the type of the tabular data structure is a native type, wherein the type is selected from among at least two types including the native type and a non-native type, wherein the native type characterizes data structures that are accessed using the first programming language and the second programming language and the non-native type characterizes data structures that are accessed using a data access layer and the second programming language; 
 identify a first proxy data table usable as a proxy for the tabular data structure, wherein the tabular data structure and the first proxy data table share the first memory allocation, wherein the first proxy data table is accessed using the first API based on the second programming language; and 
 issue one or more read commands, using the first API, to the first proxy data table to cause the set of data to be read from the tabular data structure; 
 
 generate, by the application code, output data; 
 receive, from the application code, a second indication to write the output data, the output data being a result of the analysis operation; 
 in response to receiving the second indication:
 generate an output data structure; and 
 issue one or more write commands to cause the output data to be written to the output data structure. 
 
 
     
     
       22. The non-transitory computer-readable medium of  claim 21 , wherein:
 the output data structure is the native type; 
 the output data structure is accessed using the first API; and 
 issuing the one or more write commands to cause the output data to be written to the output data structure comprises:
 identifying a second proxy data table usable as a proxy for the output data structure, wherein the output data structure and the second proxy data table share a second memory allocation, wherein the second proxy data table is accessed using the first API; and 
 issuing the one or more write commands, using the first API, to the second proxy data table to cause the output data to be written to the output data structure. 
 
 
     
     
       23. The non-transitory computer-readable medium of  claim 21 , wherein:
 the output data structure is the non-native type; 
 the output data structure is accessed using a second API based on the second programming language; and 
 issuing the one or more write commands to cause the output data to be written to the output data structure comprises issuing the one or more write commands to cause the data access layer to write the output data to the output data structure. 
 
     
     
       24. The non-transitory computer-readable medium of  claim 21 , wherein the first programming language is Python and the second programming language is C. 
     
     
       25. The non-transitory computer-readable medium of  claim 24 , wherein the tabular data structure is a Python data frame. 
     
     
       26. The non-transitory computer-readable medium of  claim 21 , wherein the tabular data structure comprises one or more columns and the first memory allocation is a column buffer. 
     
     
       27. The non-transitory computer-readable medium of  claim 26 , wherein:
 the tabular data structure is accessed using a column-based native API, wherein the column-based native API is based on the first programming language; and 
 issuing the one or more read commands to the first proxy data table to read the set of data from the tabular data structure comprises, for each column in the one or more columns of the tabular data structure, accessing the column buffer to read the set of data. 
 
     
     
       28. The non-transitory computer-readable medium of  claim 26 , wherein identifying the first proxy data table comprises:
 generating the first proxy data table, comprising:
 for each column of the tabular data structure, determining column information; 
 generating a tabular data structure schema based on the column information; 
 identifying a name for the first proxy data table; and 
 generating the first proxy data table using the name and the tabular data structure schema. 
 
 
     
     
       29. The non-transitory computer-readable medium of  claim 21 , further including additional program code that is executable by one or more processors for causing the one or more processors to:
 determine information about the tabular data structure, comprising:
 accessing the set of data using a data access management object; 
 generating the tabular data structure using the data access management object, wherein the type of the tabular data structure is the non-native type, wherein the non-native type characterizes data structures that are accessed using the second programming language; and 
 determining the information about the tabular data structure. 
 
 
     
     
       30. The non-transitory computer-readable medium of  claim 21 , further including additional program code that is executable by one or more processors for causing the one or more processors to:
 determine information about the tabular data structure, comprising:
 accessing the set of data using a tabular data structure library; 
 generating the tabular data structure using a data access management object, wherein the type of the tabular data structure is the native type; and 
 determining the information about the tabular data structure.