Connectors and fact stores
Implement connection probes, source reads, destinations and ontology materialization.
A connector links Semogram to an external system. A fact-store capability materializes ontology facts and, where supported, reads them back. Choose the interface that matches the operation rather than labelling every integration a source connector.
Define the contracts
Separate install_config from source_stream, source_query and write. The installation holds credentials and connection settings; the endpoint supplies a concrete target. Required fields and defaults must match runtime validation.
For a read-only API connector, define the URL/auth configuration, target selector, record shape, pagination and bounds. Declare incremental behavior only if the implementation persists and resumes the correct checkpoint. For a destination, define supported append/upsert/overwrite behavior and key requirements.
Implement the runtime
The generated package must include actual exported implementations and the corresponding cravenPlugin runtime declaration. Existing connector packages expose check, discover, read and spec according to their supported operations. The read declaration’s capability flags describe probes and bounded query support; they do not supply implementations.
The connector protocol provides configuration/state readers and progress, heartbeat, log, error and checkpoint emitters through @craven/core. Use those public interfaces. Do not invent a different record or telemetry protocol in the package.
Write capabilities implement createWriter and the destination writer lifecycle: init, startFile, commitFile and close. Review file metadata, checksum handling, supported write policy and cleanup. The current governed source-write and action paths require the Iceberg adapter; implementing a generic destination writer does not add those guarantees automatically.
Fact stores
A fact-store target carries ontology_fact_store configuration. A runtime fact materializer implements materialize; check, discover, ensure, read and sparql are optional interfaces and must be declared only when supported.
Use Postgres for relational ontology materialization or Apache Jena for a SPARQL store. A standalone store has its own installation connection; a bundled connector/store can share it.
The generated authoring contract and parser accept specific capability/contract kinds. Do not assume a manually declared read_store package is supported by every authoring validation path merely because official stores exist. Inspect the current validator before introducing a custom store-only package.
Verify
- Validate required and invalid configuration without leaking secrets.
- Check connection failure and timeouts.
- Compare a bounded read with a fixture, including pagination and empty results.
- Verify checkpoint/resume behavior when declared.
- For writes, inspect the receiving state and retry behavior in a dedicated test target.
- For stores, materialize a known fact and read it back with its source evidence.
FAQ
Can I declare exactly-once support?
Only when the implemented storage/receipt protocol provides it and tests demonstrate the promised behavior. An idempotency key or manifest flag alone is not proof of exactly-once external effects.
Does a check need to ingest records?
Keep connection probes bounded and without side effects. Use the actual read or write test to verify data and permissions.
Where do endpoint settings belong?
On the operation’s target contract, not copied into every connection setting. See Capabilities and contracts.