Introduction

https://e.gitee.com/gudusoft/docs/1006527/file/2767137?sub_id=5738180

Why we need SQLFlow-Ingester?

SQLFlow actually supports two kinds of file as the input:
  • SQL file, including comments. DDL file for an example, sql such as create table can be used as metadata.
  • Json files which contain the database metadata
All other kinds of the input will be no longer supported and users shall convert the inputs to the above two formats. As a result, an out-of-box tool is needed to complete the conversion.

SQLFlow-Ingester Basic

SQLFlow-Ingester is a tool helps you to extract metadata from various database. The metadata file can be later used by dlineage tool to generate data lineage.
SQLFlow-Ingester download address: https://github.com/sqlparser/sqlflow_public/releases
SQLFlow-Ingester has three different parts:
  • sqlflow-exporter: getting metadata from database
  • sqlflow-extractor: processing raw data files such as log files, various script files (from which SQL statements and metadata to be processed are extracted), CSV files containing SQL statements, etc.
  • sqlflow-submitter: submitting sql and metadata to the sqlflow server, creating jobs, generating data lineage, and creating the results in the UI.
Read more details for the above Ingester components:
Last modified 1mo ago