Transform records

This example reads two orders and keeps order_id and total, renaming total to order_total. It uses Postgres read plus the bundled select-project transform. The transform preserves one row per input row and records the declared lineage.

What you need

You need a Semogram account with workspace membership, a project in that workspace and permission to create and run pipelines. Reading or writing also requires access to the selected endpoints and external systems. Creating a pipeline does not grant those permissions.

Use a reachable test PostgreSQL database and its SQL client. You need installation/endpoint management access as well as pipeline read/write/execute access. The connection must be reachable from the execution runtime, not only from your laptop.

Prepare the source

CREATE TABLE public.pipeline_demo_orders (
  order_id integer PRIMARY KEY,
  region text NOT NULL,
  total numeric(12,2) NOT NULL
);
INSERT INTO public.pipeline_demo_orders VALUES
  (1, 'North', 120.00),
  (2, 'South', 75.50);

Open workspace Plugins → Explore, find Postgres (@craven/postgres), inspect its installed contract and install the read capability. Fill these connection fields and run its supported connection check:

Open workspace Data Endpoints → New data endpoint. Use name demo_orders, namespace pipelines, direction Ingress, the installed Postgres read capability and Stream (source_stream) as the source contract. In the target fields set Schema to public and Table to pipeline_demo_orders. Save and inspect its schema/sample. demo_orders is a name you choose in Semogram; public.pipeline_demo_orders is the actual database table. Keep its returned endpoint UUID for programmatic examples.

The Postgres installation guide and endpoint guide provide additional detail; the required setup for this fixture is included here.

Install the transform

In workspace Plugins → Explore, find Craven Transforms (@craven/transforms), inspect the required operation and install/enable it. Its bundle installation configuration is empty; the operation’s business parameters belong on the transform node. Use the capability installation ID, not the bundle installation ID, for capabilityRefId. The Transforms guide provides optional background.

Select Select Project Transform (select-project). Inspect its parameter schema and runtime support before using it.

Build the graph

Create a pipeline with Ingress → Transform. Bind ingress to demo_orders, Full load, parser extraction, output orders. Connect its out port to the transform's in port. Configure:

Create a pipeline reading demo_orders in full mode with parser extraction. Use the installed select-project transform to keep order_id and total, rename total to order_total, and output projected_orders. Show the graph and capability binding before saving; do not write externally.

{
  "capabilityRefId": "<SELECT_PROJECT_CAPABILITY_UUID>",
  "function": "select-project",
  "input": "orders",
  "output": "projected_orders",
  "params": {
    "fields": [
      {
        "source": "order_id",
        "target": "order_id"
      },
      {
        "source": "total",
        "target": "order_total"
      }
    ]
  }
}

Validate, save and run

Validate the draft and resolve errors. Preflight checks configuration only; it does not produce records or run a model. Save version, inspect the active saved graph and Run once. Open the run and compare the named step's output with the expected result below. Keep the run ID and inspect errors/counts, not only the assistant's response. Programmatic launch uses HTTP POST /api/v1/projects/<PROJECT_ID>/pipelines/<PIPELINE_ID>/execute with pipelines:execute and an Idempotency-Key, or MCP pipeline_execute with pipelineId, projectId and idempotencyKey.

If the Studio result only exposes an artifact handle, open the corresponding output/artifact inspection rather than treating that handle as a row preview. To persist rows externally, add an Egress endpoint and test its write semantics; the examples below do not silently create a destination.

Expected result

There should be two rows, no region field in the projected result and no change to the database source. Test a missing selected field before relying on it in real data. A different operation such as filter, aggregate or join has different row-count and lineage behavior; selecting an installed operation is not interchangeable with writing its name in function.