This Is How Observability Starts - With Modeling Pipeline Runs

In data platforms, it is easy to confuse business workflow state with job execution state. A company moves through statuses like "analyzing collection," "saving results," or "generating reports." Those labels are useful for the product. They tell users and operators where the account sits in a longer process.But they are a poor substitute for knowing whether a specific pipeline is pending, running, finished, or failed right now.In this post, I will walk through why that gap matters, what we changed in a real project, and how treating pipeline runs as first-class processes improves both observability and control.What we had before. The platform processes data for multiple companies, each progressing through different stages and pipeline actions. Before this change, pipeline execution was mostly inferred from company status transitions.The API would get a trigger, enqueue a message, and a worker would pick up the job. When the worker finished, it patched the company status forward. On failure, it patched an error substatus.That pattern works for a coarse product timeline. It answers questions like: Has this company reached the analysis stage? Should the UI unlock the next step? Did the last transition succeed or fail? It does not answer questions like: Is an extraction job already running for this company? Was this run cancelled and replaced by a newer one? How long did extraction take versus transformation versus loading? Which exact pipeline type failed, and with what message? In short: the system knew where the company was in the workflow. It did not reliably know what the workers were doing.Why that was not enough. Long-running pipelines invite ambiguity. Users retry. Clients double-submit. Operators re-trigger a stuck stage. Messages get delivered more than once. A worker may still be running while the product UI already looks idle, or the UI may look busy while the queue job never started.When job identity lives only in logs and status side effects, debugging becomes archaeology. You reconstruct a run from timestamps, prefixes in object storage, and scattered status history rows. That is slow, and it does not scale as a control mechanism.We needed an explicit process model for the pipelines themselves: create a process when a pipeline is triggered, update it while the worker runs, and finish it with a clear terminal state.What we implemented. The design is intentionally simple. We added one process table typed by pipeline action. Each row belongs to a company and has a status lifecycle similar to other job systems in the platform: pending, running, completed, error, cancelled.When the API triggers one of the tracked pipelines, it creates a process first and passes the process id into the job arguments. The worker then reports: RUNNING when execution starts optional phase timings while the job progresses COMPLETED or ERROR when it finishes Concurrency is explicit. At most one active process (pending or running) is allowed per company and pipeline type. A force flag decides what happens on conflict: force=true: cancel the active process and start a new one force=false: reject the trigger with a conflict response That turns "someone clicked again" from an accidental double run into a deliberate policy.Observability gets a spine. Once pipeline runs are rows in the database, observability stops depending on log spelunking. You can ask the API for the latest process of a given type for a company. You can inspect status, message, and phase timings. You can correlate a failing company status with the exact process that produced it.And if you do need to check the logs, the table also stores a trace_id generated by the API and propagated to the pipeline process.Phase timings are especially useful for operational diagnosis. A job that "takes forever" is not one problem. It may be a slow JDBC extraction, a shuffle-heavy transform, or an expensive load into object storage. It may also be a database connection that was dropped but somehow never raised an error. Persisting those durations next to the process makes the bottleneck visible without opening a Spark UI for every incident.Control becomes intentional. Better visibility is only half the win. The other half is control. Without a process record, preventing concurrent runs is awkward. You might try to infer activity from company substatus, but that mixes product state with execution state. A company can sit in an "analysis" substatus for reasons unrelated to a live worker. Or a worker can still be alive after the UI has already moved on.With typed processes and an active uniqueness rule, the policy is clear: one analysis run at a time per company one persist run at a time one memory-generation run at a time one conflict-resolution run at a time The force parameter makes restart behavior part of the API contract instead of an unspoken convention. Operators and clients can choose between "replace the current run" and "do not start if something is already active."That is a small interface change with a large operational effect.What we deliberately did not change. We did not change the company status machine. Product workflow transitions remain intact. Pipeline processes sit beside them as job-tracking records, not as a competing workflow engine. That separation matters. Company status answers, "Where is this account in the business flow?" Process status answers, "What is the worker doing right now?" Both questions are useful. They should not share one overloaded field.Final Words If your platform launches long-running pipelines from an API, entity status alone is not enough. Status is a product narrative. A process record is an execution contract. By creating a typed process when a pipeline is triggered, constraining concurrency per entity and action, and letting workers report status and phase timings, we made pipeline runs observable and controllable without inventing a heavyweight orchestration layer.The result is simpler operations: fewer mystery double runs, clearer failure messages, and phase-level timing that turns "the job is slow" into something you can actually investigate.