As SQLFlow is supporting more and more machine learning toolkits, the corresponding code generation logics are better being organized as separate packages. An intermediate representation(IR) of the SQL jobs becomes necessary to connect these separate packages with the core
sql package should include the following functionalities:
- The entry point of running extended SQL statements.
- The parsing of extended SQL statements.
- The verification of extended SQL statements, including verifying the syntax, the existence of the selected fields.
- The feature derivation, including name, type, shape, and preprocessing method of the select fields.
- The training data and validation data split.
With these functionalities, the
sql package çan translate user typed extended SQL statements to an IR as an exposed Go struct. The codegen package takes the IR and returns a generated Python program for the
sql package to execute.
We propose the following code structures.
sql/ ... codegen/ feature_column.go intermediate_representation.go tensorflow/ ... xgboost/ ...
The IR and feature column definition will resides in
sql/codegen. Each code generator package forms a subdirectory in