User defined functions for database query languages based on call-back functions

Abstract

A parallel database system processes database queries invoking a user defined function (UDF). The database query is specified in a database query language, for example, the structured query language (SQL) and the UDF is specified in a programming language different from the database query language, for example, C/C++. The UDF may include multiple phases. The parallel database system generates optimized code for the UDF. The parallel database system includes a leader node and a plurality of compute nodes. The database query processes one or more database tables such that each compute node processes a slice of each database table. The leader node sets up a global context structure and provides it to each compute node. The global context may be shared across multiple executions of the query. The UDF may be used for efficient execution of extraction, transformation, and loading of data from external systems.

What is claimed is: 
     
         1 . A non-transitory computer readable storage medium storing instructions comprising:
 a user defined function (UDF) specified in a programming language, wherein the UDF comprises instructions for one or more call-back functions, the one or more call-back functions comprising:
 a describe call-back function configured to specify a schema describing data processed by the UDF; 
   instructions for receiving a database query specified in a database query language, the database query specifying an invocation of the UDF;   instructions for executing the describe call-back function to determine the schema representing data processed during the invocation of the UDF by the database query;   instructions for generating code for processing during the invocation of the UDF by the database query, the generated code comprising representations of data objects based on the schema obtained by executing the describe call-back function;   instructions for compiling the generated code to generate executable code for processing the database query; and   instructions for generating results of the database query by executing the compiled executable code for each of a plurality of rows processed by the database query.   
     
     
         2 . The non-transitory computer readable storage medium of  claim 1 , wherein the schema is an output schema, wherein the describe call-back function comprises:
 instructions for receiving an input schema as input to the UDF and generating the output schema based on the input schema, wherein the input schema describes a structure of data input to the UDF during the invocation of the UDF and the output schema describes a structure of data output by the invocation of the UDF by the database query.   
     
     
         3 . The non-transitory computer readable storage medium of  claim 2 , wherein the input schema identifies a first set of columns representing values input to the UDF and the output schema identifies a second set of columns representing values output by the UDF. 
     
     
         4 . The non-transitory computer readable storage medium of  claim 1 , wherein the invocation of the UDF is a first invocation of the UDF, the database query is a first database query, and the schema generated is a first schema, wherein the describe call-back function comprises instructions for generating a second schema representing data processed during a second invocation of the UDF by a second database query. 
     
     
         5 . The non-transitory computer readable storage medium of  claim 1 , wherein the UDF further comprises a process function specifying a computation to be performed on rows of input table processed by the database query, the stored instructions further comprising:
 instructions for executing the process function for each of the plurality of rows processed by the database query.   
     
     
         6 . The non-transitory computer readable storage medium of  claim 1 , wherein the UDF further comprises a finalize function comprising instructions for indicating that there are no more input rows for processing, the stored instructions further comprising:
 instructions for emitting any remaining output rows responsive to an execution of the finalize function indicating that there are no more input rows for processing.   
     
     
         7 . The non-transitory computer readable storage medium of  claim 1 , wherein the database query processes a database table distributed across a plurality of processors, the stored instructions further comprising:
 instructions for sending the generated code to each of the plurality of processors for processing the portion of the database table stored on the processor.   
     
     
         8 . The non-transitory computer readable storage medium of  claim 1 , wherein the UDF is a multi-phase UDF comprising a plurality of phases, wherein the describe call-back function specifies an input schema and an output schema for each phase of the multi-phase UDF. 
     
     
         9 . The non-transitory computer readable storage medium of  claim 1 , wherein the UDF is a multi-phase UDF comprising a plurality of phases comprising a first phase and a second phase, wherein the describe function specifies a first partitioning strategy for partitioning data during the first phase and a second partitioning strategy for partitioning data during a second phase. 
     
     
         10 . A method for executing database queries, the method comprising:
 storing a user defined function (UDF) specified in a programming language, wherein the UDF comprises instructions for one or more call-back functions, the one or more call-back functions comprising:
 a describe call-back function configured to specify a schema describing data processed by the UDF; 
   receiving a database query specified in a database query language, the database query specifying an invocation of the UDF;   executing the describe call-back function to determine the schema representing data processed during the invocation of the UDF by the database query;   generating code for processing during the invocation of the UDF by the database query, the generated code comprising representations of data objects based on the schema obtained by executing the describe call-back function;   compiling the generated code to generate executable code for processing the database query; and   generating results of the database query by executing the compiled executable code for each of a plurality of rows processed by the database query.   
     
     
         11 . The method of  claim 10 , wherein the schema is an output schema, wherein the describe call-back function is configured to receive an input schema as input to the UDF and generate the output schema based on the input schema, wherein the input schema describes a structure of data input to the UDF during the invocation of the UDF and the output schema describes a structure of data output by the invocation of the UDF by the database query. 
     
     
         12 . The method of  claim 11 , wherein the input schema identifies a first set of columns representing values input to the UDF and the output schema identifies a second set of columns representing values output by the UDF. 
     
     
         13 . The method of  claim 10 , wherein the invocation of the UDF is a first invocation of the UDF, the database query is a first database query, and the schema generated is a first schema, wherein the describe call-back function is configured to generate a second schema representing data processed during a second invocation of the UDF by a second database query. 
     
     
         14 . The method of  claim 10 , wherein the UDF further comprises a process function specifying a computation to be performed on rows of input table processed by the database query, the method further comprising:
 executing the process function for each of the plurality of rows processed by the database query.   
     
     
         15 . The method of  claim 10 , wherein the UDF further comprises a finalize function comprising instructions for indicating that there are no more input rows for processing, the method further comprising:
 emitting any remaining output rows responsive to an execution of the finalize function indicating that there are no more input rows for processing.   
     
     
         16 . The method of  claim 10 , wherein the database query processes a database table distributed across a plurality of processors, the method further comprising:
 sending the generated code to each of the plurality of processors for processing the portion of the database table stored on the processor.   
     
     
         17 . The method of  claim 10 , wherein the UDF is a multi-phase UDF comprising a plurality of phases, wherein the describe call-back function specifies an input schema and an output schema for each phase of the multi-phase UDF. 
     
     
         18 . The method of  claim 10 , wherein the UDF is a multi-phase UDF comprising a plurality of phases comprising a first phase and a second phase, wherein the describe function specifies a first partitioning strategy for partitioning data during the first phase and a second partitioning strategy for partitioning data during a second phase. 
     
     
         19 . A computer-system comprising:
 a computer processor; and   a non-transitory computer readable storage medium storing instructions comprising:
 a user defined function (UDF) specified in a programming language, wherein the UDF comprises instructions for one or more call-back functions,
 the one or more call-back functions comprising: 
 a describe call-back function configured to specify a schema describing data processed by the UDF; 
 
 instructions for receiving a database query specified in a database query language, the database query specifying an invocation of the UDF; 
 instructions for executing the describe call-back function to determine the schema representing data processed during the invocation of the UDF by the database query; 
 instructions for generating code for processing during the invocation of the UDF by the database query, the generated code comprising representations of data objects based on the schema obtained by executing the describe call-back function; 
 instructions for compiling the generated code to generate executable code for processing the database query; and 
 instructions for generating results of the database query by executing the compiled executable code for each of a plurality of rows processed by the database query. 
   
     
     
         20 . The computer system of  claim 19 , wherein the schema is an output schema, wherein the describe call-back function comprises instructions for receiving an input schema as input to the UDF and generating the output schema based on the input schema, wherein the input schema describes a structure of data input to the UDF during the invocation of the UDF and the output schema describes a structure of data output by the invocation of the UDF by the database query.