The Deployment Failed. The Resources Are Still Running.

The deployment reaches the configuration stage and stops. The network exists. The virtual machine exists. Configuration did not finish, and the health check never ran. The pipeline is red, but the provider console shows resources that are still active. Both views are technically correct. The command failed. The infrastructure also changed. The problem starts when automation treats the first fact as if it proves the opposite of the second. A nonzero exit code describes a process outcome. It does not describe the complete state of several remote systems after that process ends. A Failed Command Is Not a Rollback Infrastructure workflows often present one action to the operator while making many separate calls underneath it. A deployment may create a network, reserve an address, create a virtual machine, wait for access, configure the operating system, and run service probes. Those calls do not become one atomic transaction because one command started them. If the fourth step fails, the first three may remain complete. Microsoft's description of the Compensating Transaction pattern makes the same point for distributed operations. Completed work often needs specific compensating actions, and those actions can fail too. Recovery is a workflow with state, not an automatic rewind button. That distinction is easy to miss in CI because a pipeline naturally collapses the result into green or red. Green means the process reached its expected end. Red means it did not. Neither color can tell an operator which provider changes were committed before the failure. A Blind Retry Can Make the State Worse The usual response to a failed run is to rerun it. That is safe only when the system can establish what the first attempt did. Suppose the provider accepted a create request, but the response was lost during a connection failure. The caller sees an error. A retry may create a duplicate, return a name conflict, or attach later steps to the wrong object. The principle is familiar from API design. HackerNoon's guide to idempotency and retries explains why repeating a request after an uncertain response is not safe unless the server and client share a stable identity for that operation. Google Cloud Workflows makes the distinction explicit too: its retry guidance provides different handling for idempotent and non-idempotent steps. Idempotency helps, but it is not the whole answer. It can prevent the same logical request from producing the same side effect twice. It does not tell the orchestrator which earlier steps completed, which later steps never started, or whether a failed resource should be resumed or removed. That requires a durable operation record. Write the Transition Before the Mutation The most important state write happens before control passes to the provider. After validation and preflight succeed, the runtime should persist a nonterminal marker such as running. The marker needs the operation identity, target, resolved inputs, timestamp, and any outputs already known from a previous run. Only then should the provider mutation begin. If the state write fails because the disk is full, the path is unavailable, or the backend cannot commit, the provider call should not start. That rule can feel severe. It turns a local storage problem into a blocked deployment. The alternative is worse: infrastructure changes without a durable record that the change started. There is one conservative edge case. The process might persist running and then stop before reaching the provider. A later operator must reconcile an uncertain operation that made no remote change. That is safer than preserving an old destroyed marker after a provider may have created something. Keep What the Runtime Already Knows Failure handling should update state, not replace it with an empty result. A reduced state record might look like this: { "module_ref": "platform/example/service", "status": "error", "failed_command": "apply", "last_error": "provider failed after create", "outputs": { "resource_id": "previously-known-resource" }, "rerun_inputs_file": "/config/rerun-inputs.yml", "resolved_inputs_file": "/resolved.inputs.yml", "run_id": "" } The output is deliberately described as previously known. On a first-time creation, a provider may create an object and fail before returning its identifier. Automation cannot preserve a value it never received. It can still preserve the inputs, operation identity, error, timestamp, and run record needed to query the provider and reconcile the result. This is related to, but separate from, an infrastructure engine's own state. HackerNoon's walkthrough of a remote Terraform state backend shows why resource bindings and locking must survive individual workstations. HashiCorp's own state documentation describes state as the mapping between configured resource instances and remote objects. An orchestration runtime has another job. It must remember which lifecycle operation was attempted across engines, which stage failed, and what the next safe actions are. Provider state and operation state complement each other. Neither should impersonate the other. Pending Work Did Not Fail Partial failure also changes how a dependency graph should be presented. Consider four steps: network ok virtual machine error configuration pending health check pending The configuration and health check did not fail. They did not run. Calling all three steps failed hides the actual boundary. It can also encourage a recovery path that repeats completed work or attempts a health check before the virtual machine state is settled. The graph needs enough vocabulary to distinguish complete, error, pending, retained, and destroyed state. Internal workflow engines may call an unrun step deferred. The operator-facing meaning is simpler: it is waiting on an earlier state transition. This is where reconciliation becomes more useful than retry. A recent HackerNoon article on reconciliation at scale describes stable operation identifiers, conditional state changes, and checks against external systems. The same discipline applies to infrastructure. Resume from facts that can be checked, not from the assumption that a failed process changed nothing. Destroy Is a Verified Operation An error state must be treated as potentially live. If destroy logic skips every step that is not marked ok, it will skip the resources most likely to need cleanup after a failed apply. A safer rule is to skip only state already confirmed absent or destroyed. An error marker remains eligible for provider inspection and teardown. Ownership matters during that process. A child resource may disappear when its parent is removed. For example, deleting a virtual machine may also remove a guest-level state that cannot be reached independently once the machine is gone. The child should not be marked destroyed when parent deletion starts. It should reach terminal state only after the parent is confirmed absent. If parent deletion fails, the child remains unresolved. Kubernetes uses a comparable idea with finalizers. A delete request places an object into a terminating state, while the object remains until required cleanup completes. The API request is not treated as proof that the resource is gone. Infrastructure teardown deserves the same honesty. This extends the argument I made in Disaster Recovery as a Governance System: recovery needs an explicit decision, a verified outcome, and a record of what happened. That governance depends on the runtime first preserving an accurate account of the failed operation. Turning the Failure Case Into a Contract I maintain HybridOps Core, an open-source infrastructure runtime. While hardening its failure behavior, I added a regression path that begins with an old destroyed marker, starts a new apply, and then returns a partial provider failure. The expected contract is strict. State must read running before the provider call. After the failure, it must read error, retain earlier outputs, identify the failed command, and remain eligible for destroy. A separate test prevents provider execution when the pre-mutation state write fails. The implementation and tests are visible in the partial-mutation change. The project is only one implementation. The contract applies to any automation layer coordinating stateful changes across remote systems. Five Rules Worth Keeping Persist nonterminal state before the first provider mutation. Do not mutate when the runtime cannot persist that state. Retain known outputs and resolved recovery inputs after failure. Show unrun dependent work as pending, not failed. Declare teardown complete only after absence is confirmed. These rules do not promise atomic infrastructure. They do something more practical: they preserve an honest boundary between what the process attempted and what the infrastructure may now contain. A red pipeline proves that execution ended badly. It does not prove that the environment rolled back. The next operator should begin with durable state and a valid recovery path, not a provider console and a reconstruction exercise.