S3 Bucket

What this endpoint does

An S3 endpoint selects an object path in a bucket configured on an installation. A source endpoint reads supported data files; a separate destination endpoint writes output through an installed write capability. The endpoint does not create a bucket.

Like all data endpoints, it belongs to a workspace and can be referenced by projects in that workspace, subject to permissions. Saving it configures access; it does not start ingestion.

Directions and supported operations

Create separate source and destination endpoints, each selecting its corresponding capability. The bucket and credentials belong to the installation; the path belongs to the endpoint. Read access needs listing and object reads; writing needs object uploads and the permissions required by the storage service's multipart-upload flow.

The destination is object append, not row upsert, row delete or table replacement. Atomicity is per object, not the whole run. Policy writes reject existing object keys. A retry that creates a new key can still duplicate logical records. The destination target uses path; do not add a writeMode field that its contract does not declare.

Plugin used

This plugin provides separate read and write capabilities. Select read for Ingress or write for Egress; the example below installs read first and then explains how to install write. Installation steps for this example are included below. The S3 Bucket installation guide provides optional further detail. Connection details and credentials stay on the installation; the endpoint selects a particular target through that installed capability.

What you need

• A Semogram account with workspace access and permission to manage endpoints
• An existing matching installation, or the connection details to create one using the steps below
• The installation selects your bucket and region, and its credentials can list/read the intended objects. Know the object path and file formats, and prepare a small set of known rows

Read example setup

This example assumes the configured bucket already contains supported order-export files under orders/. That path comes from your storage layout. We choose order_exports as the endpoint name and files as its namespace. order_id is an illustrative column in those files, not a field supplied by S3.

Install and configure the plugin

1. Open Plugins in this workspace and select Explore.
2. Find S3 Bucket, inspect its publisher/version and select its read capability.
3. Name the installation and fill its connection settings using your actual external-system details.
4. Save and run the supported connection check. Fix any reported error before creating the endpoint.

Example installation configuration:

Install the plugin described on this page in this workspace. Discover its catalog entry, select the matching capability and propose the installation using the connection settings shown here. Ask me to enter credentials in protected installation fields. Show the selected plugin/version, capability and non-secret settings before saving.

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "plugin_installation_create",
    "arguments": {
      "capabilityClass": "<MATCHING_CAPABILITY_CLASS>",
      "discoveryId": "<DISCOVERY_ID_FROM_CATALOG>",
      "name": "<INSTALLATION_NAME>",
      "config": {
        "bucket": "YOUR_BUCKET",
        "region": "YOUR_REGION",
        "credentials": {
          "accessKeyId": "YOUR_ACCESS_KEY",
          "secretAccessKey": "YOUR_SECRET_KEY"
        }
      },
      "idempotencyKey": "<UNIQUE_KEY_FOR_THIS_INSTALLATION>"
    }
  }
}

Replace every YOUR_… value. Enter credentials only in protected installation configuration. The configuration above is separate from the endpoint target below; selecting a target does not create or authenticate the connection.

Prepare example data

The current reader selects Parquet objects, not CSV or arbitrary bucket files. Create a two-row fixture locally with Python and PyArrow:

python -m pip install pyarrow

import pyarrow as pa
import pyarrow.parquet as pq
pq.write_table(pa.table({
    "order_id": [1, 2],
    "total": [120.00, 75.50]
}), "orders.parquet")

Upload orders.parquet with your storage console to orders/orders.parquet in the configured test bucket. Allow the read installation to list this prefix and fetch this object. Use target path orders without a trailing slash: the reader matches the exact key or keys beginning with orders/. A full S3 URI is not the path selector. The first bounded read should contain the two fixture rows.

For compatible storage, set the installation's HTTPS endpoint and path-style option as required by your provider.

Use the assistant

Open Data Endpoints → New data endpoint in the workspace. Describe your actual target and choose a name for the endpoint. For the example above, you could ask:

Create a source endpoint named order_exports in namespace files.
Use our installed S3 Bucket read capability with contract source_stream
and set these target fields:
Path: orders
Review the proposed connection and target before saving.

Replace example values with your own. Give the assistant the installed connection reference; keep credentials in protected installation settings.

Configure manually

Choose Edit manually, select the matching installed capability and configure:

Create the endpoint described on this page using these settings:
name: <ENDPOINT_NAME_FROM_THIS_EXAMPLE>
namespace: <ENDPOINT_NAMESPACE_FROM_THIS_EXAMPLE>
role: source
contractKind: source_stream
target / path: orders
pluginCapabilityInstallationId: <INSTALLED_CAPABILITY_UUID>
Use the actual installed capability and the endpoint name/namespace selected in this example. Show the proposed direction, connection and target before saving. Keep credentials on the installation.

curl --request POST "https://platform.semogram.com/api/v1/data-endpoints" \
  --header "Authorization: Bearer ${SEMOGRAM_API_KEY}" \
  --header "Idempotency-Key: <UNIQUE_KEY_FOR_THIS_ENDPOINT>" \
  --header "Content-Type: application/json" \
  --data-binary @- <<'JSON'
{
  "name": "<ENDPOINT_NAME_FROM_THIS_EXAMPLE>",
  "namespace": "<ENDPOINT_NAMESPACE_FROM_THIS_EXAMPLE>",
  "role": "source",
  "contractKind": "source_stream",
  "target": {
    "path": "orders"
  },
  "pluginCapabilityInstallationId": "<INSTALLED_CAPABILITY_UUID>"
}
JSON

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "source_create",
    "arguments": {
      "name": "<ENDPOINT_NAME_FROM_THIS_EXAMPLE>",
      "namespace": "<ENDPOINT_NAMESPACE_FROM_THIS_EXAMPLE>",
      "role": "source",
      "contractKind": "source_stream",
      "target": {
        "path": "orders"
      },
      "pluginCapabilityInstallationId": "<INSTALLED_CAPABILITY_UUID>",
      "idempotencyKey": "<UNIQUE_KEY_FOR_THIS_ENDPOINT>"
    }
  }
}

Set the primary-key hint to order_id after checking uniqueness and nulls. Add a useful description, owner and expected freshness. Save in the workspace. Inspect the installed version’s contract before adding optional settings.

Verify the endpoint

1. Open the saved endpoint and run its supported connection check and schema inspection.
2. Read a small sample through the supported preview. If the connector has no preview, select a project, open Pipeline Studio and create a pipeline with a source node referencing this endpoint.
3. Configure a bounded test input using the connector's supported settings, validate the pipeline, save a version and run it.
4. Inspect returned records or the completed run's output. Compare expected keys and values with the example fixture before scheduling anything.

Compare selected object names, supported file formats, rows and keys with storage. Confirm what the path selects; not all files under a prefix necessarily share a schema.

Check connectivity, target validity and actual data separately. Saving does not import records or schedule execution.

Write example: export two orders as Parquet objects

Use the two-row orders/orders.parquet fixture and order_exports source created above. The destination will be named verified_order_files, namespace files, and will write to a separate bucket prefix verified-orders/. It does not modify the input Parquet file.

1. In workspace Plugins → Explore, install S3 Bucket → write using the bucket, region and protected credentials configuration above. For compatible storage, include the required HTTPS endpoint and path-style setting. Give this installation upload permissions on the dedicated output prefix, including the multipart-upload permissions required by your provider.
2. Open Data Endpoints → New data endpoint → Edit manually. Set name verified_order_files, namespace files, Egress (role destination), the installed write capability and contract write.
3. Save this target; the writer combines the prefix with each runtime output file name:

Create the endpoint described on this page using these settings:
name: <ENDPOINT_NAME_FROM_THIS_EXAMPLE>
namespace: <ENDPOINT_NAMESPACE_FROM_THIS_EXAMPLE>
role: destination
contractKind: write
target / path: verified-orders/
pluginCapabilityInstallationId: <INSTALLED_CAPABILITY_UUID>
Use the actual installed capability and the endpoint name/namespace selected in this example. Show the proposed direction, connection and target before saving. Keep credentials on the installation.

curl --request POST "https://platform.semogram.com/api/v1/data-endpoints" \
  --header "Authorization: Bearer ${SEMOGRAM_API_KEY}" \
  --header "Idempotency-Key: <UNIQUE_KEY_FOR_THIS_ENDPOINT>" \
  --header "Content-Type: application/json" \
  --data-binary @- <<'JSON'
{
  "name": "<ENDPOINT_NAME_FROM_THIS_EXAMPLE>",
  "namespace": "<ENDPOINT_NAMESPACE_FROM_THIS_EXAMPLE>",
  "role": "destination",
  "contractKind": "write",
  "target": {
    "path": "verified-orders/"
  },
  "pluginCapabilityInstallationId": "<INSTALLED_CAPABILITY_UUID>"
}
JSON

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "source_create",
    "arguments": {
      "name": "<ENDPOINT_NAME_FROM_THIS_EXAMPLE>",
      "namespace": "<ENDPOINT_NAMESPACE_FROM_THIS_EXAMPLE>",
      "role": "destination",
      "contractKind": "write",
      "target": {
        "path": "verified-orders/"
      },
      "pluginCapabilityInstallationId": "<INSTALLED_CAPABILITY_UUID>",
      "idempotencyKey": "<UNIQUE_KEY_FOR_THIS_ENDPOINT>"
    }
  }
}

4. In a workspace project, open Pipeline Studio. Connect an ingress node referencing order_exports to an egress node referencing verified_order_files. Use only the fixture file. Review the target, supported object-append policy and caller permissions, validate, save a version and run once.
5. Inspect the run and list verified-orders/ using your storage console. Download the newly produced Parquet file(s) and inspect them with a Parquet reader. Across the output files, expect two records with order IDs 1 and 2 and totals matching the fixture. File count can vary with batching; verify record count rather than assuming one file per run.
6. Confirm the original orders/orders.parquet object is unchanged. Record the output keys and run ID so you can distinguish a retry from a new export.

A failed run can leave already committed objects. Policy writes reject an existing key; new keys can still duplicate the same logical rows. Do not blindly rerun append after partial failure. No destination field adds row-level upsert or rollback to this object writer.

Manage the endpoint

For output, create a separate destination endpoint on the installed write capability with write contract and a dedicated target such as {"path":"verified-orders/"}. Test its files before production use.

Keep credentials on the installation. Review consumers before replacing capabilities, changing targets or deleting endpoints.

FAQ

Why does verification fail?

Check bucket/region, path semantics and object permissions. Do not assume a full S3 URI is interchangeable with the published path selector.

Can another project use it?

Yes, within the same workspace and subject to permissions. The endpoint remains workspace-scoped.

What comes after verification?

Use sources in a bounded pipeline, destinations in a supported write flow, and stores in ontology bindings. Inspect real output before scheduling recurring work.