Introduction

https://e.gitee.com/gudusoft/docs/1006527/file/2767137?sub_id=5738180

Why we need SQLFlow-Ingester?

SQLFlow supports two kinds of file as the input:

• SQL file, including comments. DDL file for an example, sql such as create table can be used as metadata.

• Json files which contain the database metadata

All other kinds of the input will be no longer supported and users shall convert the inputs to the above two formats. As a result, an out-of-box tool is needed to complete the conversion.

SQLFlow-Ingester Basic

SQLFlow-Ingester is a tool that helps you to extract metadata from various databases and create SQLFlow job based on your database. The extracted metadata file can also be used by dlineage tool to generate data lineage.

SQLFlow-Ingester download address: https://github.com/sqlparser/sqlflow_public/releases

SQLFlow-Ingester has three different parts:

• sqlflow-exporter: getting metadata from database

• sqlflow-extractor: processing raw data files such as log files, various script files (from which SQL statements and metadata to be processed are extracted), CSV files containing SQL statements, etc.

• sqlflow-submitter: submitting sql and metadata to the sqlflow server, creating jobs, generating data lineage, and having the results in the UI.

Read more details for the above Ingester components:

SQLFlow-ExporterSQLFlow-ExtractorSQLFlow-Submitter