Private API source

What this endpoint does

A custom API endpoint selects records from a private service through a connector your team implements. It can read only the resources and operations that the installed custom capability actually implements.

Directions and supported operations

This example is Ingress only: private service → Semogram, using a custom read capability and source_stream contract. It does not write to the vendor. A package may bundle a separate write capability, but that requires its own implementation, installation selection and destination endpoint; read authentication does not grant vendor write access.

Plugin used

This scenario uses a custom read capability, not an existing named internal plugin. The endpoint stores selectors, not implementation code. For additional package-authoring details, see custom plugins.

What you need

You need a Semogram account with workspace access and permission to manage plugins and endpoints, the private service’s API contract and a test account you can read. The package creation and installation steps for this example are below.

Example setup

For this scenario the connector implements a source_stream contract with required accountId and resource selectors. Its protected installation configuration holds the vendor base URL and token. These are example custom fields, not built-in Semogram selectors; your package must implement and validate them.

In this example vendor.shipment_events is the endpoint name we choose, vendor is its grouping namespace, test_account is an account in the external service, and shipment_events is a resource the custom plugin implements. event_id must be present in the returned records. None of these example names creates an external account or resource.

Define the target contract

{
  "type": "object",
  "required": ["accountId", "resource"],
  "properties": {
    "accountId": { "type": "string" },
    "resource": { "type": "string", "enum": ["shipment_events"] }
  },
  "additionalProperties": false
}

Implement actual bounded reads, pagination, errors and telemetry following the connector runtime guidance. Validate against a fixture before publishing. Declare discovery or incremental support only when implemented and tested.

Prepare and install the custom plugin

1. Open Plugins in the workspace and start custom plugin creation.
2. Describe the service's real authentication, operation, target fields and response/write behavior. Supply service documentation and a test fixture without exposing credentials.
3. Generate the package and run sandbox validation. Review its actual runtime exports, target schema, failure handling and fixture result; a manifest alone cannot implement the operation.
4. Publish the reviewed package to the workspace catalog and install its matching capability.
5. Configure the service URL and auth on the installation and run its supported check. Choose that installed capability when creating the endpoint below.

This example assumes the resulting package implements the exact target fields shown here. It does not supply a ready-made vendor connector. If your package's contract differs, use its real fields instead.

Use the assistant

Open Data Endpoints → New data endpoint and ask for vendor.shipment_events using the installed custom read capability, account test_account and resource shipment_events. Review the target against the published contract.

Configure manually

Choose Edit manually, source role, installed custom read capability and source_stream. Set namespace vendor and the target:

Create the endpoint described on this page using these settings:
name: <ENDPOINT_NAME_FROM_THIS_EXAMPLE>
namespace: <ENDPOINT_NAMESPACE_FROM_THIS_EXAMPLE>
role: source
contractKind: source_stream
target / accountId: test_account
target / resource: shipment_events
pluginCapabilityInstallationId: <INSTALLED_CAPABILITY_UUID>
Use the actual installed capability and the endpoint name/namespace selected in this example. Show the proposed direction, connection and target before saving. Keep credentials on the installation.

curl --request POST "https://platform.semogram.com/api/v1/data-endpoints" \
  --header "Authorization: Bearer ${SEMOGRAM_API_KEY}" \
  --header "Idempotency-Key: <UNIQUE_KEY_FOR_THIS_ENDPOINT>" \
  --header "Content-Type: application/json" \
  --data-binary @- <<'JSON'
{
  "name": "<ENDPOINT_NAME_FROM_THIS_EXAMPLE>",
  "namespace": "<ENDPOINT_NAMESPACE_FROM_THIS_EXAMPLE>",
  "role": "source",
  "contractKind": "source_stream",
  "target": {
    "accountId": "test_account",
    "resource": "shipment_events"
  },
  "pluginCapabilityInstallationId": "<INSTALLED_CAPABILITY_UUID>"
}
JSON

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "source_create",
    "arguments": {
      "name": "<ENDPOINT_NAME_FROM_THIS_EXAMPLE>",
      "namespace": "<ENDPOINT_NAMESPACE_FROM_THIS_EXAMPLE>",
      "role": "source",
      "contractKind": "source_stream",
      "target": {
        "accountId": "test_account",
        "resource": "shipment_events"
      },
      "pluginCapabilityInstallationId": "<INSTALLED_CAPABILITY_UUID>",
      "idempotencyKey": "<UNIQUE_KEY_FOR_THIS_ENDPOINT>"
    }
  }
}

Set event_id as the key hint only if actual records establish its uniqueness. Keep the token on the installation, then save the endpoint in the workspace.

Verify the endpoint

1. Open the saved endpoint and run its supported connection check and schema inspection.
2. Read a small sample through the supported preview. If the connector has no preview, select a project, open Pipeline Studio and create a pipeline with a source node referencing this endpoint.
3. Configure a bounded test input using the connector's supported settings, validate the pipeline, save a version and run it.
4. Inspect returned records or the completed run's output. Compare expected keys and values with the example fixture before scheduling anything.

Check a fixture account with two small pages. Compare event IDs, timestamps and account isolation. Test the final page, an empty response and a rejected account. If discovery is absent, configure selectors from the contract and verify the read directly.

Run a small pipeline and inspect records and telemetry. Test checkpoint/resume before choosing incremental execution or adding a schedule.

Manage the endpoint

Rotate auth on the installation. Change account/resource on the endpoint after reviewing consumers. For package upgrades, compare target contracts and record shape, then rerun the fixture before replacing production bindings.

FAQ

Can I configure these selectors on HTTP Dataset?

No. HTTP Dataset has its own request/response contract. These fields belong to the example custom package.

Does publishing create the endpoint?

No. Publish, install the capability, then create and verify the endpoint.

What if the custom capability is missing?

Confirm publication, installation, workspace and supported capability kind. See troubleshooting.