Exporter Output Formats¶

The SQLFlow Ingester exporter (also known as Grabit) connects to a live database over JDBC and writes its metadata — catalog structure plus the DDL text of views, procedures, functions, triggers, packages, and foreign keys — to disk for SQLFlow's column-level lineage engine to consume. It reads metadata only; it never reads or moves table row data.

The exporter can write that metadata in two shapes:

	Single JSON	Sharded
On-disk shape	one `metadata.json` file (a `Sqlflow` root object)	a directory: `manifest.json` + `catalog/` + `source/` + `index/`
Selected by	`-format single`	default (`-format sharded`)
`-save` argument	a file path (default `metadata.json`)	an output directory (default `.`)
Where DDL text lives	inline, in `servers[].queries[]`	in `source/.jsonl`, indexed by `index/.idx`
Memory model	whole document held in memory (a ~1 GB single-document ceiling)	streaming; peak memory ≈ one shard
Scale target	small and medium catalogs	large estates (catalog ~0.5–1 GB, DDL tens–hundreds of GB)
Extra capabilities	—	per-shard SHA-256 integrity, per-object content hashing, object-level incremental exports, block compression

Both formats serialize the same underlying model objects (Sqlflow, Server, Database, Schema, Table, Column, Query, Procedure, Package, Synonym, Sequence, DBLink, Error). The per-object field meanings are therefore identical across formats — sharded mode simply splits the same objects into a resident structure (the catalog) and seekable DDL (the source), joined by a stable key.

Single JSON export → — the complete element-by-element structure of metadata.json.
Sharded export → — the directory layout and the purpose of every file.

When to use which¶

Single JSON is the simplest to consume: parse one file, and every object's DDL is already inline in servers[].queries[]. Use it for small to medium catalogs. Because the entire document is built in memory and serialized at once, it is bounded by an approximately 1 GB single-document limit — beyond that, use sharded.
Sharded is the default and the format for large database estates. It streams the export with bounded memory (roughly one shard at a time), keeps the structural catalog small and resident, and makes the potentially multi-gigabyte DDL random-seekable through a byte-offset index. It also unlocks integrity verification, incremental re-exports against a baseline, and optional compression.

Selecting the format and scope¶

Both formats share the same connection and scope options; a few options are sharded-only.

# Sharded (default) — writes a directory
java -jar sqlflow-exporter.jar \
  -dbVendor mssql -host 10.0.0.12 -port 1433 -db SALES \
  -user reader -pwd-env DB_PASSWORD \
  -save /export/sales

# Single JSON — writes one metadata.json
java -jar sqlflow-exporter.jar \
  -dbVendor mssql -host 10.0.0.12 -port 1433 -db SALES \
  -user reader -pwd-env DB_PASSWORD \
  -format single -save /export/sales/metadata.json

Common options:

Option	Purpose
`-format {sharded\\|single}`	Output shape. Default `sharded`.
`-save <path>`	File path (single) or output directory (sharded).
`-dbVendor <name[:version]>`	Source dialect, e.g. `oracle`, `mssql`, `snowflake`.
`-host` / `-port` / `-db` / `-user`	JDBC connection.
`-pwd` / `-pwd-env` / `-pwd-file`	Password inline, from an env var, or from a file. Priority: `-pwd` > `-pwd-env` > `-pwd-file`. Prefer `-pwd-env`/`-pwd-file` so the password never appears on the command line.
`-includeDatabases` / `-excludeDatabases` / `-includeSchemas` / `-excludeSchemas` / `-includeObjects` / `-excludeObjects` / `-objectTypes`	Regex scope filters (whole-name, case-insensitive) that limit exactly what is extracted.
`-extractConstraints`	Extract PK / FK / UNIQUE / INDEX / CHECK constraints.
`-shardCompress {none\\|block}`	Sharded only. `block` gzip-compresses each source record.
`-baseline <dir>`	Sharded only. Produce an incremental export against a previous export directory.

See the two detail pages for the exact fields each format emits.