Link

Intermediate Representation

Overview

As SQLFlow is supporting more and more machine learning toolkits, the corresponding code generation logics are better being organized as separate packages. An intermediate representation(IR) of the SQL jobs becomes necessary to connect these separate packages with the core sql package.

The core sql package should include the following functionalities:

  1. The entry point of running extended SQL statements.
  2. The parsing of extended SQL statements.
  3. The verification of extended SQL statements, including verifying the syntax, the existence of the selected fields.
  4. The feature derivation, including name, type, shape, and preprocessing method of the select fields.
  5. The training data and validation data split.

With these functionalities, the sql package ├žan translate user typed extended SQL statements to an IR as an exposed Go struct. The codegen package takes the IR and returns a generated Python program for the sql package to execute.

Code Structure

We propose the following code structures.

sql/
  ...
  codegen/
    feature_column.go
    intermediate_representation.go
    tensorflow/
      ...
    xgboost/
      ...

The IR and feature column definition will resides in sql/codegen. Each code generator package forms a subdirectory in sql/codegen like sql/codegen/tensorflow/.

Intermediate Representation

Please refer to sql/codegen/intermediate_representation.go and sql/codegen/feature_column.go for implementation details.