Sharded Export

Sharded is the exporter's default output format (-format sharded). Instead of one large metadata.json, it writes a directory of files: a manifest, one small structural catalog per shard, the DDL as line-delimited JSON, and a byte-offset index over that DDL. This lets the exporter stream a large database estate with bounded memory, keep the structural catalog resident, and make multi-gigabyte DDL random-seekable. It also enables per-shard integrity checks, per-object content hashing, incremental re-exports, and optional compression.

The sharded format is a stable on-disk contract. Its identifying constants:

• format: sqlflow-sharded
• formatVersion: 2
• hashAlgo: sha256
• canonicalizerId: sqlflow-ingester/todbc-utf8-rev1

Directory layout

<export-dir>/
├── manifest.json                          # read FIRST; written LAST (atomic publish)
├── catalog/<shardId>.catalog.json         # structure only, NO DDL text  (kept resident)
├── source/<shardId>.source.jsonl          # one Query (DDL) object per line (NDJSON)
└── index/<shardId>.source.idx             # byte-offset + content-hash index, 1:1 with source lines

With -shardCompress block, the source files are source/<shardId>.source.jsonl.gz.

Each shard is a set of three files that share one shardId stem. Never reconstruct file paths from an object's name — always use the path values in the manifest.

How sharding is keyed

Sharding is structural, not size-based:

• One shard per database for catalog+schema and database-only vendors.
• One shard per schema for schema-only vendors (Oracle).

The shardId is a stable, filesystem-safe stem: a sanitized form of the object name (non [A-Za-z0-9_.-] characters replaced with _, truncated) plus a short hash suffix derived from the server index and name. Because it is deterministic, the same object produces the same shardId across runs, which is what makes incremental exports possible.

manifest.json

Written last, so a directory that has a valid manifest.json is a complete export. A directory without one is a half-written export and must be rejected. Read and validate the manifest before touching any other file.

Key	Type	Meaning
format	string	sqlflow-sharded.
formatVersion	int	2.
catalogVersion / sourceVersion	int	Independent version numbers for the catalog and source segments (currently 1 / 1).
exporterVersion	string	Tool and version.
createdBy	string	Tool and version.
physicalInstance	string	Source database host.
createTime	string	Human export time, yyyy-MM-dd HH:mm:ss.
createdAt	string	ISO-8601 UTC timestamp.
newline	string	The record separator used in .jsonl and .idx files (\n).
encoding	string	UTF-8.
sourceCompression	string	none or block.
hashAlgo	string	sha256.
canonicalizerId	string	sqlflow-ingester/todbc-utf8-rev1.
incremental	boolean	Present and true only for incremental (-baseline) runs. A consumer that does not understand deltas must refuse an incremental: true export.
baseline	object	{ createdAt, manifestSha256 } of the baseline (incremental only).
sourceServer	object	{ product, version, edition, productLevel } of the source DBMS, when captured.
servers	array	The shard descriptors (see below).
errorMessages	Error[]	Present only on non-fatal errors.
tombstones	array	Objects deleted since the baseline (incremental); otherwise [].
tombstonesComplete	boolean	Present and false when errors prevented full tombstone computation.

servers[] entries

Each server entry describes one source server and lists its shards:

• name, dbVendor, supportsCatalogs, supportsSchemas — as in the single-JSON Server.
• nameLevels — the declarative naming topology (see below).
• dbLinks[] — database links. These live only in the manifest, not in any catalog shard.
• one of databases[] or schemas[] — the array of shard descriptors.
• unscoped[] — optional; queries that mapped to no shard, so that no DDL is silently lost.

Shard descriptor

{
  "shardId": "SALES-a1b2c3d4",
  "name": "SALES",                                                  // real display name
  "catalog": { "path": "catalog/SALES-a1b2c3d4.catalog.json", "bytes": 712,  "sha256": "<hex>" },
  "source":  { "path": "source/SALES-a1b2c3d4.source.jsonl",  "bytes": 360,  "rows": 210, "sha256": "<hex>" },
  "index":   { "path": "index/SALES-a1b2c3d4.source.idx" },
  "counts":  { "tables": 180, "views": 120, "procedures": 210, "columns": 5600 }
}

• catalog / source carry path, bytes, and sha256 for integrity verification; source also carries rows (number of DDL records written — in incremental mode, only new and changed objects).
• counts is a sizing hint (procedures counts procedures + functions + triggers).

nameLevels

nameLevels declares how object names are qualified and, therefore, how the catalog nests:

nameLevels	Name form	Table location in the catalog	Vendors
["catalog","schema"]	db.schema.table	Database.schemas[].tables[]	SQL Server, PostgreSQL, Redshift, Snowflake, Greenplum, Netezza, DB2, Azure SQL
["database"]	db.table	Database.tables[]	MySQL, Teradata, Hive, Impala
["schema"]	schema.table	the shard root is a Schema	Oracle

catalog/<shardId>.catalog.json

The shard's structural catalog — a Database (or, for schema-only vendors, a Schema) object serialized with the DDL text stripped out. The DDL never lived on these structural objects in the first place; it lives in the source shard. The catalog carries the same object lists as the single-JSON Database/Schema (schemas, tables, views, others, procedures, functions, triggers, packages, synonyms, sequences, processes), with the same Table, Column, Procedure, Package, Synonym, and Sequence field sets described in the single JSON reference. Because it has no DDL, the catalog stays small and can be held resident in memory while the source is streamed or seeked.

Owner-key spelling is object-dependent and is part of the contract — preserve it exactly:

Object	Catalog key	Schema key
Table / view / other	databaseName	schemaName
Procedure / function / trigger	databaseName	schemaName
Package	databaseName	schemaName
Synonym	database	schema
Sequence	database	schema

source/<shardId>.source.jsonl

The shard's DDL, as NDJSON — one Query object per line. DDL newlines are JSON-escaped, so every record is exactly one physical line. Each object has the same fields as the single-JSON Query: type, database, schema, name, groupName, sourceCode, and, when a body could not be read, sourceUnavailable / sourceUnavailableReason. An unreadable object still gets a source line (with no sourceCode) and a matching index record, so it remains discoverable.

index/<shardId>.source.idx

An NDJSON index, one record per source line, aligned 1:1 with the source file. It lets a consumer locate and verify any object's DDL without scanning the whole source shard.

Field	Meaning
key	The object's fully-qualified identity, database.schema.name#type (empty namespace segments are skipped; a missing type becomes #unknown). This is the stable join key to the catalog.
type	The query's type.
offset / length	The byte offset and length of the record. In none mode they address the JSON bytes only, excluding the trailing newline. In block mode they address the compressed gzip member. Omitted for unchanged records in an incremental export.
contentHash	Lowercase sha256 hex over the sourceCode string's UTF-8 bytes (not the line bytes). An empty or absent source hashes the empty string. Always present.
status	Incremental only: new, changed, or unchanged.

To fetch one object's DDL on demand: look up its key in the index, seek(offset) in the source shard, read length bytes (gunzip in block mode), parse the JSON, and take sourceCode. Verify it against contentHash if you need integrity or content-addressable deduplication.

Reassembling the export

1. Read and validate manifest.json. Reject the directory if it is missing.
2. For each shard, parse catalog/<shardId>.catalog.json into a resident symbol table (walk all the object lists, descending into schemas[] where present). Read dbLinks[] from the manifest.
3. Load every index/<shardId>.source.idx into a key → { shardId, offset, length, contentHash, status } map.
4. Consume DDL either by streaming each source/<shardId>.source.jsonl end to end, or on demand via seek(offset) / read length. Skip records marked sourceUnavailable: true.

Incremental exports (-baseline)

-baseline <dir> produces a delta against a previous export:

• The source segment is a delta: unchanged objects write no bytes (their index record has a contentHash but no offset/length); new and changed objects write their DDL.
• The catalog segment is always full.
• tombstones[] lists { key, type } for objects present in the baseline but absent this run (deletions). If errors prevented computing them, they are suppressed and tombstonesComplete is false.
• The manifest carries incremental: true. A consumer that does not implement delta application must refuse such an export rather than treat it as a full one.

Block compression (-shardCompress block)

Each source record is written as its own gzip member. Concatenating the members and gunzipping yields exactly the uncompressed .jsonl; the index offset/length address the compressed member boundaries. An empty shard is a single empty gzip member (valid gzip, not a zero-byte file). contentHash is unchanged — it is always computed over the uncompressed sourceCode.

Atomic publish

The exporter writes the whole export to a staging area, then moves files into place with the manifest last: it removes any existing manifest.json, moves catalog/, source/, and index/, and only then moves the new manifest.json in. A consumer therefore never observes a partially written export — the export becomes visible exactly when the manifest appears.

Manifest example

A trimmed SQL Server manifest:

{
  "format": "sqlflow-sharded", "formatVersion": 2, "catalogVersion": 1, "sourceVersion": 1,
  "exporterVersion": "sqlflow-ingester v1.5.1", "createdAt": "2026-06-25T12:00:00Z",
  "newline": "\n", "encoding": "UTF-8", "sourceCompression": "none",
  "hashAlgo": "sha256", "canonicalizerId": "sqlflow-ingester/todbc-utf8-rev1",
  "servers": [{
    "name": "10.0.0.12", "dbVendor": "dbvmssql",
    "supportsCatalogs": true, "supportsSchemas": true,
    "nameLevels": ["catalog", "schema"],
    "databases": [{
      "shardId": "SALES-3f2a9c11", "name": "SALES",
      "catalog": { "path": "catalog/SALES-3f2a9c11.catalog.json", "bytes": 712, "sha256": "…" },
      "source":  { "path": "source/SALES-3f2a9c11.source.jsonl",  "bytes": 360, "rows": 2, "sha256": "…" },
      "index":   { "path": "index/SALES-3f2a9c11.source.idx" },
      "counts":  { "tables": 1, "views": 1, "procedures": 1, "columns": 3 }
    }]
  }],
  "tombstones": []
}

The two matching index lines (offsets exclude the trailing newline):

{"key":"SALES.dbo.V_ACTIVE#view","type":"view","offset":0,"length":168,"contentHash":"<sha256 of the view's sourceCode>"}
{"key":"SALES.dbo.CALC_TOTAL#procedure","type":"procedure","offset":169,"length":188,"contentHash":"<sha256 of the proc's sourceCode>"}