Schema and records
Distinguish connection checks, discovery, hints and actual reads
What endpoint verification does
Verification establishes whether an endpoint selects the intended data and can perform the supported operation. It separates saved configuration, connectivity, discovered schema and actual records.
Plugin used
Checks, discovery and reads come from the endpoint's installed capability. Use its implemented operations; generic connectors do not inherit Iceberg snapshot support. Installation instructions are in plugin setup guides, while each endpoint setup example includes its own installation steps.
What you need
Use a configured endpoint, source access and a small known fixture. Record the expected keys, types and values before testing. To verify ingestion rather than a read, have a project and a bounded pipeline available.
Verify in stages
| Stage | Establishes | Does not establish |
|---|---|---|
| Target validation | Selectors satisfy the installed contract | External target exists |
| Connection check | Supported probe reaches the configured system | Every target is readable |
| Schema discovery | Reported streams and fields | Every row has that shape |
| Bounded read | Actual selected records can be returned | Complete ingestion or recurring freshness |
| Pipeline execution | That run processed data | Every later run will succeed |
Schema hints
schemaHints.columns names expected fields; schemaHints.primaryKey names identity fields. These hints do not create external constraints or enforce uniqueness. Compare them with discovered schema and actual records. Test composite keys, nulls and duplicates before relying on them for upserts.
Record reads
Use a small limit and a known record first. Check scope, sorting, field types and pagination. Only use filters/query operations the connector declares and implements. Historical snapshot/as-of reads are Iceberg-specific; generic connectors do not inherit them.
For assistants, use source validation, checks, schema and records where supported. Discover real endpoint IDs rather than guessing names as IDs.
Schema changes
Compare schema after upstream changes or capability upgrades. Update hints and affected mappings, queries and pipeline steps, then rerun a bounded fixture. A successful connection check does not catch every field rename or type change.