Advanced Usage
This page describes how to discover the data lineage using Python and the SQLFlow REST API.
By using the SQLFlow REST API, you can code in python to discover the data lineage in SQL scripts and get the result in an actionable diagram, json, csv or graphml format.
You can integerate the python code provided here into your own project and add the powerful data lineage analsysis capability instantly.
1. interactive data lineage visualizations
3. Data lineage in CSV, graphml format
Prerequisites
Python 2.7 or higher version must be installed and configured correctly.
Installing Dependency Libraries:
Usage
Example:
Connect to the SQLFlow Cloud Server
Connect to the SQLFlow on-premise This will discover data lineage by analyzing the
python-data-lineage-sqlserver.sql
file. You may also specify a zip file which includes lots of SQL files.
This will discover data lineage by analyzing all SQL files under sqlfiles
directory.
Parameters
path_to_config_file
This can be a single SQL file, a zip file including multiple SQL files, or a directory including lots of SQL files.
server
Usually, it is the IP address of the SQLFlow on-premise version installed on your owner servers such as 127.0.0.1
or http://127.0.0.1
You may set the value to https://api.gudusoft.com
if you like to send your SQL script to the SQLFlow Cloud Server to get the data lineage result.
port
The default value is 8081
if you connect to your SQLFlow on-premise server.
However, if you setup the nginx reverse proxy in the nginx configuration file like this:
Then, keep the value of serverPort
empty and set server
to the value like this: http://127.0.0.1/api
.
Please keep this value empty if you connect to the SQLFlow Cloud Server by specifying the
https://api.gudusoft.com
in theserver
>
userId, userSecret
This is the user id that is used to connect to the SQLFlow server. Always set this value to gudu|0123456789
and keep userSecret
empty if you use the SQLFlow on-premise version.
If you want to connect to the SQLFlow Cloud Server, you may request a 30 days premium account to get the necessary userId and secret code.
databaseType
This parameter specifies the database dialect of the SQL scripts that the SQLFlow has analyzed.
resultType
When you submit SQL script to the SQLFlow server, A job is created on the SQLFlow server and you can always see the graphic data lineage result via the browser,
Even better, This demo will fetch the data lineage back to the directory where the demo is running. Those data lineage results are stored in the data/result/
directory.
This parameter specifies which kind of format is used to save the data lineage result.
Available values for this parameter:
1: JSON, data lineage result in JSON.
2: CSV, data lineage result in CSV format.
3: diagram, in graphml format that can be viewed by yEd.
SQLFlow REST API
Please check here for the detailed information about the SQLFlow REST API
Last updated