Gudu SQLFlow Product Docs
  • 1. Introduction
    • What is Gudu SQLFlow?
      • What SQLFlow can do
      • Architecture Overview
    • Getting Started
      • Sign up a new account
        • Team Management
        • Delete My Account
        • Activate by entering a coupon
      • How to use SQLFlow
      • Different modes in Gudu SQLFlow
        • Query mode
        • Job mode
      • Basic Usage
      • Convert SQL to E-R Diagram
      • Colors in SQLFlow Diagram
      • Show call relationship
    • Installation
      • Version and Users
        • Cloud and On-Premise version
        • SQLFlow before Version 6
          • For older version SQLFlow under Linux
          • For older version SQLFlow under MacOS
          • For older version SQLFlow under Windows
      • Linux
      • MacOS
      • Windows
      • Docker
      • Clickhouse Installation
        • Clickhouse For CentOs
        • Clickhouse For Ubuntu/Debian/RHEL
      • Troubleshooting
      • Upgrade
      • Third Party Components
      • Renew License File
    • UI
      • SQLText Editor
      • Schema Explorer
      • Diagram Panel
      • Settings
      • Job Management
        • Job Sources
    • Dlineage Tool
      • Overview
      • Usage
        • Analyze data linege from SQL files
        • Analyze data linege from a database
        • Resolve the ambiguous columns in SQL query
        • Map the DataFlowAnalyzer and the settings on SQLFlow UI
        • Settings
      • Dataflow.xml structure
      • FAQ
  • 2. CONCEPTS
    • Data Lineage Basics
      • Dataflow
        • Relations generated by SQLFlow
      • Direct Dataflow
      • Indirect Dataflow
      • Aggregate function and Dataflow
      • Dataflow chain
    • Data Lineage Format Reference
  • 3. API Docs
    • Prerequisites
    • Using the Rest API
    • SQLFlow Rest API reference
      • User Interface
      • Generation Interface
        • /sqlflow
        • /sqlflow/selectedgraph/table_level_lineage
        • /sqlflow/selectedgraph/image
        • /sqlflow/graph
        • /sqlflow/graph/table_level_lineage
        • /sqlflow/graph/image
        • /sqlflow/downstreamGraph
        • /sqlflow/upstreamGraph
        • /sqlflow/erdiagramSelectGraph
        • /sqlflow/leftMostSourceTableGraph
      • Job Interface
        • /submitUserJob
        • /displayUserJobSummary
        • /displayUserJobsSummary
        • /exportLineageAsJson
        • /exportFullLineageAsJson
        • /exportLineageAsGraphml
        • /submitPersistJob
        • /displayUserLatestJobTableLevelLineage
      • Export Image
      • Export CSV
        • /sqlflow/exportFullLineageAsCsv
        • /job/exportFullLineageAsCsv
    • Swagger UI
    • Export the data lineage result
    • Python
      • Basic Usage
      • Advanced Usage
    • SQL Parser API
      • checkSyntax
  • 4. SQLFlow Widget
    • Widget Get started
    • Usages
    • Widget API Reference
  • 5. Databases
    • Database Objects
      • Azure
      • DB2
  • 6. SQLFlow-ingester
    • Introduction
      • SQLFlow-Exporter
      • SQLFlow-Extractor
      • SQLFlow-Submitter
    • Get Started
      • SQL Server
    • SQLFlow-Ingester Java API Usage
    • Understand the format of exported data
      • Oracle
      • Microsoft SQL Server
      • MySQL
      • PostgreSQL
    • List of Supported dbVendors
    • Git Repo
    • Third Party Components
  • 7. Reference
    • Lineage Model
      • Json Format Lineage Model
      • XML Format Lineage Model
      • Data Lineage Elements
    • Database Model
  • 8. other
    • FAQ
      • Handling Internal Database
      • Delete Your Account
      • Table Form Data Without Intermediates
      • Not all schema exported from Oracle
      • Lineage Customization
    • Roadmap
    • SQL Samples
      • Exchange table partition
      • Generate relationship for renamed table
      • Snowflake table function lineage detection
    • Change Logs
    • SQLFlow with Oracle XML functions
    • Major Organizations Utilizing SQLFlow
Powered by GitBook
On this page
  • Database who has the database layer and the schema layer (SQL Server for example)
  • Database who has the database layer only (MYSQL for example)
  • Database who has the schema layer only (Currently no supported database belongs to this type)
  • Database list
  • Other clarifications on the json fields:
  1. 6. SQLFlow-ingester

Understand the format of exported data

https://e.gitee.com/gudusoft/projects/151613/docs/617965/file/1485427?sub_id=6032663

This page gives an example and some clarifications on the format of the SQLFlow-Ingester exported data.

We may have some similar fields in the exported json data due to the fact that different databases can have various structure concepts. These similar fields won't all appear in a real case.

Following json is an exhaustive example which lists all possible fields:

{
  "createTime":"", //export time
  "createdBy":"sqlflow-exporter",//name of the export tool
  "physicalInstance":"",//server address
  "servers":[
      {
          "name":"",//server name
          "dbVendor":"",//database type,possible values are: dbvathena,dbvazuresql,dbvbigquery,dbvcouchbase,dbvdb2,dbvgreenplum,dbvhana,dbvhive,dbvimpala,dbvinformix,dbvmdx,dbvmysqldbvnetezza,dbvopenedge,dbvoracle,dbvpostgresql,dbvpresto,dbvredshift,dbvsnowflake,dbvsparksql,dbvmssql,dbvsybase,dbvteradata,dbvvertica
          "supportsCatalogs":false,//whether there's a database layer
          "supportsSchemas":true,//whether there's a schema layer 
          "databases":[
              {
                  "name":"",//database name
                  "schemas":[
                      {
                          "name":"",//schema name
                          "synonyms":[
                              {
                                  "name":"",
                                  "sourceName":"",
                                  "sourceSchema":"",
                                  "sourceDbLinkName":""
                              }
                          ],
                          "sequences":[
                              {
                                  "name":"",
                                  "incrementBy":""
                              }
                          ],
                          "tables":[
                              {
                                  "name":"",//table name 
                                  "columns":[
                                      {
                                          "dataType":"",//column data type
                                          "name":"",//column name
                                          "comment":""//column comment
                                      }
                                  ]
                              }
                          ],
                          "views":[ 
                            {
                                  "name":"",//view name
                                  "columns":[
                                      {
                                          "dataType":"",
                                          "name":"",
                                          "comment":""
                                      }
                                  ]
                              }
                          ],
                          "others":[//others, including resultset, variable, path etc 
                              {
                                  "name":"",
                                  "columns":[
                                      {
                                          "dataType":"",
                                          "name":"",
                                          "comment":""
                                      }
                                  ]
                              }
                          ],
                          "packages":[
                              {
                                  "name":"",//package name 
                                  "procedures":[//prcedures in the package
                                  ],
                                  "functions":[//functions in the package
                                  ],
                                  "triggers":[//triggers in the package
                                  ]
                              }
                          ],
                          "procedures":[
                              {
                                  "name":"",
                                  "type":"",
                                  "arguments":[
                                      {
                                          "name":"",
                                          "dataType":"",
                                          "inout":""
                                      }
                                  ]
                              }
                          ],
                          "functions":[
                              {
                                  "name":"",
                                  "type":"",
                                  "arguments":[
                                      {
                                          "name":"",
                                          "dataType":"",
                                          "inout":""
                                      }
                                  ]
                              }
                          ],
                          "triggers":[
                              {
                                  "name":"",
                                  "type":"",
                                  "arguments":[
                                      {
                                          "name":"",
                                          "dataType":"",
                                          "inout":""
                                      }
                                  ]
                              }
                          ]
                      }
                  ],
                  "synonyms":[//synonym list when there's no schema object, same structure as the above servers.databases.schemas.synonyms

                  ],
                  "sequences":[//sequence list when there's no schema object, same structure as the above servers.databases.schemas.sequences 

                  ],
                  "tables":[//table list when there's no schema object, same structure as the above servers.databases.schemas.tables 


                  ],
                  "views":[//view list when there's no schema object, same structure as the above servers.databases.schemas.views 

                  ],
                  "others":[//others when there's no schema object, including resultset, variable, path etc. same structure as the above servers.databases.schemas.others 

                  ],
                  "packages":[//package list when there's no schema object, same structure as the above servers.databases.schemas.packages 

                  ],
                  "procedures":[//procedure list when there's no schema object, same structure as the above servers.databases.schemas.procedures 

                  ],
                  "functions":[//function list when there's no schema object, same structure as the above servers.databases.schemas.functions 

                  ],
                  "triggers":[//trigger list when there's no schema object, same structure as the above servers.databases.schemas.triggers 

                  ]
              }
          ],
          "schemas":[//schema list when there's no database object, same structure as the above servers.databases.schemas 

          ],
          "dbLinks":[
              {
                  "owner":"",
                  "name":"",
                  "userName":"",
                  "host":""
              }
          ],
          "queries":[//DDL scripts in database
              {
                  "database":"",
                  "schema":"",
                  "name":"",
                  "type":"",
                  "sourceCode":"",
                  "groupName":""
              }
          ]
      }
  ],
  "errorMessages":[//errors during the exporting 
      {
          "errorMessage":"",
          "errorType":"",
          "file":""
      }
  ]
}

You may have found that some of the fields appear multiple times in the above json. The reason is that there are three types of database from the database/schema structure layer:

Database who has the database layer and the schema layer (SQL Server for example)

The database has the database layer and the schema layer. supportsCatalogs and supportsSchemas will be both true. No extra db unit field outside the database list because all units belong to the specific schema and database.

{
    "createTime":"", //export time
    "createdBy":"sqlflow-exporter",//name of the export tool
    "physicalInstance":"",//server address
    "servers":[
        {
            "name":"",//server name
            "dbVendor":"",//database type,possible values are: dbvathena,dbvazuresql,dbvbigquery,dbvcouchbase,dbvdb2,dbvgreenplum,dbvhana,dbvhive,dbvimpala,dbvinformix,dbvmdx,dbvmysqldbvnetezza,dbvopenedge,dbvoracle,dbvpostgresql,dbvpresto,dbvredshift,dbvsnowflake,dbvsparksql,dbvmssql,dbvsybase,dbvteradata,dbvvertica
            "supportsCatalogs":true,// has database layer
            "supportsSchemas":true,// has schema layer
            "databases":[ // No extra db unit field outside the database list because all units belong to the specific schema and database.
                {
                    "name":"",//database name
                    "schemas":[
                        {
                            "name":"",//schema name
                            "synonyms":[
                                {
                                    "name":"",
                                    "sourceName":"",
                                    "sourceSchema":"",
                                    "sourceDbLinkName":""
                                }
                            ],
                            "sequences":[
                                {
                                    "name":"",
                                    "incrementBy":""
                                }
                            ],
                            "tables":[
                                {
                                    "name":"",//table name 
                                    "columns":[
                                        {
                                            "dataType":"",//column data type
                                            "name":"",//column name
                                            "comment":""//column comment
                                        }
                                    ]
                                }
                            ],
                            "views":[ 
                              {
                                    "name":"",//view name
                                    "columns":[
                                        {
                                            "dataType":"",
                                            "name":"",
                                            "comment":""
                                        }
                                    ]
                                }
                            ],
                            "others":[//others, including resultset, variable, path etc 
                                {
                                    "name":"",
                                    "columns":[
                                        {
                                            "dataType":"",
                                            "name":"",
                                            "comment":""
                                        }
                                    ]
                                }
                            ],
                            "packages":[
                                {
                                    "name":"",//package name 
                                    "procedures":[//prcedures in the package
                                    ],
                                    "functions":[//functions in the package
                                    ],
                                    "triggers":[//triggers in the package
                                    ]
                                }
                            ],
                            "procedures":[
                                {
                                    "name":"",
                                    "type":"",
                                    "arguments":[
                                        {
                                            "name":"",
                                            "dataType":"",
                                            "inout":""
                                        }
                                    ]
                                }
                            ],
                            "functions":[
                                {
                                    "name":"",
                                    "type":"",
                                    "arguments":[
                                        {
                                            "name":"",
                                            "dataType":"",
                                            "inout":""
                                        }
                                    ]
                                }
                            ],
                            "triggers":[
                                {
                                    "name":"",
                                    "type":"",
                                    "arguments":[
                                        {
                                            "name":"",
                                            "dataType":"",
                                            "inout":""
                                        }
                                    ]
                                }
                            ]
                        }
                    ],
            "dbLinks":[
                {
                    "owner":"",
                    "name":"",
                    "userName":"",
                    "host":""
                }
            ],
            "queries":[//DDL scripts in database
                {
                    "database":"",
                    "schema":"",
                    "name":"",
                    "type":"",
                    "sourceCode":"",
                    "groupName":""
                }
            ]
        }
    ],
    "errorMessages":[//errors during the exporting 
        {
            "errorMessage":"",
            "errorType":"",
            "file":""
        }
    ]
  }

Database who has the database layer only (MYSQL for example)

The database has the database layer but doesn't have the schema layer(or no actual schema layer). supportsCatalogs will be true and supportsSchemas will be false. No schemas list under the databases list but extra db unit lists will be present.

{
    "createTime":"", //export time
    "createdBy":"sqlflow-exporter",//name of the export tool
    "physicalInstance":"",//server address
    "servers":[
        {
            "name":"",//server name
            "dbVendor":"",//database type,possible values are: dbvathena,dbvazuresql,dbvbigquery,dbvcouchbase,dbvdb2,dbvgreenplum,dbvhana,dbvhive,dbvimpala,dbvinformix,dbvmdx,dbvmysqldbvnetezza,dbvopenedge,dbvoracle,dbvpostgresql,dbvpresto,dbvredshift,dbvsnowflake,dbvsparksql,dbvmssql,dbvsybase,dbvteradata,dbvvertica
            "supportsCatalogs":true,//there's a database layer
            "supportsSchemas":false,//there's no schema layer
            "databases":[
                {
                    "name":"",//database name
    
                    "synonyms":[//synonym list when there's no schema object, same structure as the above servers.databases.schemas.synonyms
  
                    ],
                    "sequences":[//sequence list when there's no schema object, same structure as the above servers.databases.schemas.sequences 
  
                    ],
                    "tables":[//table list when there's no schema object, same structure as the above servers.databases.schemas.tables 
  
  
                    ],
                    "views":[//view list when there's no schema object, same structure as the above servers.databases.schemas.views 
  
                    ],
                    "others":[//others when there's no schema object, including resultset, variable, path etc. same structure as the above servers.databases.schemas.others 
  
                    ],
                    "packages":[//package list when there's no schema object, same structure as the above servers.databases.schemas.packages 
  
                    ],
                    "procedures":[//procedure list when there's no schema object, same structure as the above servers.databases.schemas.procedures 
  
                    ],
                    "functions":[//function list when there's no schema object, same structure as the above servers.databases.schemas.functions 
  
                    ],
                    "triggers":[//trigger list when there's no schema object, same structure as the above servers.databases.schemas.triggers 
  
                    ]
                }
            ],
            "dbLinks":[
                {
                    "owner":"",
                    "name":"",
                    "userName":"",
                    "host":""
                }
            ],
            "queries":[//DDL scripts in database
                {
                    "database":"",
                    "schema":"",
                    "name":"",
                    "type":"",
                    "sourceCode":"",
                    "groupName":""
                }
            ]
        }
    ],
    "errorMessages":[//errors during the exporting 
        {
            "errorMessage":"",
            "errorType":"",
            "file":""
        }
    ]
  }

Database who has the schema layer only (Currently no supported database belongs to this type)

The database doesn't have the database layer(or no actual database layer) but has the schema layer only. supportsCatalogs will be false and supportsSchemas will be true. No databases list under the servers list and no extra db unit list. An extra schemas list will be present directly under the servers list.

{
    "createTime":"", //export time
    "createdBy":"sqlflow-exporter",//name of the export tool
    "physicalInstance":"",//server address
    "servers":[
        {
            "name":"",//server name
            "dbVendor":"",//database type,possible values are: dbvathena,dbvazuresql,dbvbigquery,dbvcouchbase,dbvdb2,dbvgreenplum,dbvhana,dbvhive,dbvimpala,dbvinformix,dbvmdx,dbvmysqldbvnetezza,dbvopenedge,dbvoracle,dbvpostgresql,dbvpresto,dbvredshift,dbvsnowflake,dbvsparksql,dbvmssql,dbvsybase,dbvteradata,dbvvertica
            "supportsCatalogs":false,//there's no database layer
            "supportsSchemas":true,//there's schema layer
            "schemas":[//schema list when there's no database object, same structure as the above servers.databases.schemas 
                {
                    "name":"",//schema name
                    "synonyms":[
                        {
                            "name":"",
                            "sourceName":"",
                            "sourceSchema":"",
                            "sourceDbLinkName":""
                        }
                    ],
                    "sequences":[
                        {
                            "name":"",
                            "incrementBy":""
                        }
                    ],
                    "tables":[
                        {
                            "name":"",//table name 
                            "columns":[
                                {
                                    "dataType":"",//column data type
                                    "name":"",//column name
                                    "comment":""//column comment
                                }
                            ]
                        }
                    ],
                    "views":[ 
                      {
                            "name":"",//view name
                            "columns":[
                                {
                                    "dataType":"",
                                    "name":"",
                                    "comment":""
                                }
                            ]
                        }
                    ],
                    "others":[//others, including resultset, variable, path etc 
                        {
                            "name":"",
                            "columns":[
                                {
                                    "dataType":"",
                                    "name":"",
                                    "comment":""
                                }
                            ]
                        }
                    ],
                    "packages":[
                        {
                            "name":"",//package name 
                            "procedures":[//prcedures in the package
                            ],
                            "functions":[//functions in the package
                            ],
                            "triggers":[//triggers in the package
                            ]
                        }
                    ],
                    "procedures":[
                        {
                            "name":"",
                            "type":"",
                            "arguments":[
                                {
                                    "name":"",
                                    "dataType":"",
                                    "inout":""
                                }
                            ]
                        }
                    ],
                    "functions":[
                        {
                            "name":"",
                            "type":"",
                            "arguments":[
                                {
                                    "name":"",
                                    "dataType":"",
                                    "inout":""
                                }
                            ]
                        }
                    ],
                    "triggers":[
                        {
                            "name":"",
                            "type":"",
                            "arguments":[
                                {
                                    "name":"",
                                    "dataType":"",
                                    "inout":""
                                }
                            ]
                        }
                    ]
                }
            ],
            "dbLinks":[
                {
                    "owner":"",
                    "name":"",
                    "userName":"",
                    "host":""
                }
            ],
            "queries":[//DDL scripts in database
                {
                    "database":"",
                    "schema":"",
                    "name":"",
                    "type":"",
                    "sourceCode":"",
                    "groupName":""
                }
            ]
        }
    ],
    "errorMessages":[//errors during the exporting 
        {
            "errorMessage":"",
            "errorType":"",
            "file":""
        }
    ]
  }

Database list

Other clarifications on the json fields:

  • $.createTime : export time

  • $.createdBy : name of the export tool

  • $.physicalInstance : server address

  • $.servers[0].name : server name

  • $.servers[0].dbVendor : database type,possible values are: dbvathena,dbvazuresql,dbvbigquery,dbvcouchbase,dbvdb2,dbvgreenplum,dbvhana,dbvhive,dbvimpala,dbvinformix,dbvmdx,dbvmysqldbvnetezza,dbvopenedge,dbvoracle,dbvpostgresql,dbvpresto,dbvredshift,dbvsnowflake,dbvsparksql,dbvmssql,dbvsybase,dbvteradata,dbvvertica

  • $.servers[0].supportsCatalogs : whether there's a database layer

  • $.servers[0].supportsSchemas : whether there's a schema layer

  • $.servers[0].databases[0].name : database name

  • $.servers[0].databases[0].schemas[0].others : //others, including resultset, variable, path etc

  • $.servers[0].databases[0].synonyms : synonym list when there's no schema object, same structure as the above servers.databases.schemas.synonyms

  • $.servers[0].databases[0].sequences : sequence list when there's no schema object, same structure as the above servers.databases.schemas.sequences

  • $.servers[0].databases[0].tables : table list when there's no schema object, same structure as the above servers.databases.schemas.tables

  • $.servers[0].databases[0].views : view list when there's no schema object, same structure as the above servers.databases.schemas.views

  • $.servers[0].databases[0].others : others when there's no schema object, including resultset, variable, path etc. same structure as the above servers.databases.schemas.others

  • $.servers[0].databases[0].packages : package list when there's no schema object, same structure as the above servers.databases.schemas.packages

  • $.servers[0].databases[0].procedures : procedure list when there's no schema object, same structure as the above servers.databases.schemas.procedures

  • $.servers[0].databases[0].functions : function list when there's no schema object, same structure as the above servers.databases.schemas.functions

  • $.servers[0].databases[0].triggers : trigger list when there's no schema object, same structure as the above servers.databases.schemas.triggers

  • $.servers[0].schemas : schema list when there's no database object, same structure as the above servers.databases.schemas

  • $.servers[0].queries : DDL scripts in database

  • $.errorMessages : errors during the exporting

PreviousSQLFlow-Ingester Java API UsageNextOracle

Last updated 2 years ago

Check to get a full database list and the type details.

here