Operating pipelines
Move from a verified run to scheduled execution, monitoring and reviewed recovery
Operating a pipeline means knowing which saved version ran, what data it read, what it changed and whether its result is still useful. A schedule triggers execution; a run records execution; checkpoints support recovery. None of those alone proves that the source was current or the output was correct.
You need a Semogram account with workspace/project access. Workspace owners and administrators manage schedules and checkpoint recovery. Running and reading through MCP or the public API also requires the corresponding pipeline/run permissions and project grants.
The operating cycle
| Stage | What to do | Evidence to retain |
|---|---|---|
| Prepare | Validate a draft, save a version and test a bounded input | Version, endpoint bindings and expected rows |
| Launch | Run manually or enable a reviewed schedule | Run ID and trigger |
| Inspect | Follow node progress and inspect output | Status, node results, errors and actual destination state |
| Monitor | Compare due occurrences, successful completion and source age | Schedule history, last successful run and business checks |
| Recover | Review unfinished effects before resuming | Checkpoints, provider receipts and recovery history |
| Change | Validate and activate a new version | Before/after results and active version |
Example: weekday order refresh
Assume a saved pipeline reads two test orders from a Postgres source and writes them to a dedicated output table. The plugin installation holds connection settings; read and write endpoints select the actual tables. The pipeline lives in a project; the installations and endpoints belong to its workspace.
First run the pipeline once and compare the two order IDs at the source and destination. Determine whether the destination appends, replaces or upserts before adding a schedule. If it appends, every full-load run can repeat those orders; scheduling does not add deduplication.
Create a weekday 09:00 Europe/Berlin schedule with overlap skip and missed-time run_once. Inspect upcoming times, then compare the next occurrence with its run and destination result. This example needs a working scheduler deployment; a saved schedule in a local development server does not run itself.
Choose a guide
- Runs: launch, reconnect and interpret execution state.
- Schedules: timing, policies, revisions and pausing.
- Monitoring: completion freshness, schedule history and output checks.
- Recovery: retained checkpoints and uncertain external effects.
- Incremental reads: cursor advancement and replay boundaries.
- Troubleshooting: isolate the failing stage.
Prediction nodes may queue separate prediction records. A completed pipeline does not establish that those predictions completed or that their outcomes were resolved. Follow each returned prediction ID in Forecasting and predictions.