Semogram Docs
Tutorials

Quick: pipeline from Postgres to S3

Configure Postgres and S3 endpoints, review a bounded extract pipeline, save and run it, verify the output object and schedule recurring execution.

Prerequisites: a configured Postgres read capability, an S3 write capability, workspace endpoints for the selected table and object target, and an approved write policy with actor permissions appropriate for the destination.

  1. In Pipeline Studio, describe the selected table and intended S3 output. Specify whether the first read is full or incremental. Incremental requires a usable cursor/checkpoint and connector support.
  2. Review the draft's ingress and egress endpoints and data-pool binding. Check the S3 target and supported file/write behavior. Dated filenames need explicit supported target configuration; request wording alone is not a guarantee.
  3. Validate configuration and save a version. Preflight does not return preview rows or write a temporary file to S3.
  4. Run a bounded first extract. Inspect row counts and the destination step.
  5. Open S3 and verify the actual object, format, count, and one source row.
  6. Configure and save a nightly schedule. Inspect next-run timing and operating status; monitor successful execution and duplicates across reruns.

Egress config fragment:

In Pipeline Studio, ask the assistant:

Configure an egress node that sends the selected input pool to the destination. Data endpoint name: s3.order-extracts; Source: new_orders; Write mode: append. Use the actual accessible resources and required enclosing definition. Show the proposed settings and validate them before saving or executing.

This request describes the example’s intent; review the generated contract and any required permissions. It does not create missing resources or make an unsupported operation available.

This is a serialized configuration/contract fragment, not a complete UI workflow. Use it only in the enclosing operation or advanced editor described on this page.

Serialized example
{
  "dataEndpointName": "s3.order-extracts",
  "source": "new_orders",
  "writeMode": "append"
}

The serialized node kind is egress; the fragment above belongs under config. A graph approval label alone does not implement a human pause. Verify the actual write authorization and use a governed action where request approval is required and supported.

See Workflow nodes and Write policies.