AI Is Changing DevOps, but Production Access Is Still the Line

Last year, SaaStr founder Jason Lemkin let Replit's AI agent work on a real database — and it deleted the database during a code freeze, despite direct prohibitions against changing anything. When Lemkin tried to roll back, the AI lied that it was impossible. The rollback did eventually work: the database was sitting in the cloud. The price of this kind of AI “willfulness” in IoT is a device that can go out of service. You won't be able to bring it back online — you'll have to physically reach it and reflash it by hand. And if the error has already spread via OTA across the entire fleet, that's thousands of devices scattered across regions.DevOps engineers in IoT balance the risk of using AI against its benefit. What follows is our experience: where AI helps, which tasks are better kept away from it, and why it won't replace the DevOps engineer in IoT.Three filters before handing a task to AIIn my team, before any task reaches AI, it passes through three filters:Blast radius. What is the scale of the damage if the model makes a mistake? This is the first thing we assess. Generating a draft of an internal alert and changing access rights in production are tasks with completely different levels of risk: in the first case a mistake creates noise; in the second it can block a service or open up excessive access to data. The larger the blast radius, the less autonomy AI gets.Rollback. Can the system be quickly returned to its previous state? In ordinary software you can often roll back a deployment, restore the previous configuration, or verify changes through a dry run. In IoT, once a change reaches connected devices, rollback can be difficult or altogether impossible without physical intervention.Real-time visibility. Will the engineer see immediately that something has gone wrong, and will they have time to react? If the process is covered by observability, automated checks, and instant notifications in an incident channel — Slack or PagerDuty — the risk can be assessed. But if a failure can go unnoticed until it affects users, devices, or client data, AI does not act autonomously.What a DevOps engineer can delegate to AIThe tasks that pass all three filters share a common trait: the model does not change production directly. In my team the DevOps engineer uses AI for the following categories of tasks:Data analysis and observability. IoT infrastructure generates an enormous stream of signals: telemetry from devices, service metrics, logs, pipeline statuses. A human sees the overall picture, but a weak signal in a large array is hard to spot. Here AI is useful as a tool for finding anomalies and patterns. In a large fleet of connected devices it quickly splits the data by region or device type and sees what would look unremarkable in the general flow: a battery drain in one region, a change in behavior after an update, a drop in the volume of data from part of the fleet.Troubleshooting and onboarding. AI can also work as a consultant that knows the project. For a new engineer it provides a first “map of the terrain”: it explains how a part of the infrastructure is arranged, where the services live, what the databases are responsible for — instead of spending a week gathering this from people. In troubleshooting, within minutes it collects the relevant logs, points to recent changes, and suggests a few hypotheses about where to dig — things the engineer would arrive at on their own, but much more slowly.Drafts and templates. AI works well where you need to quickly get a first version: an IaC template, a single step of a CI/CD pipeline, a test for a pipeline, a small utility for maintaining infrastructure. A small utility that takes an engineer 4–5 hours with tests and security checks, AI sketches out in a few minutes. Of course, this isn't code that ships to production without review, but it's a good starting point. AI works especially well with documentation. A DevOps engineer does many small but important things — updates configurations, rotates certificates, migrates services, records incidents. When AI writes these steps up into a proper document, the team doesn't spend hours on formatting and doesn't lose knowledge that would otherwise stay in one person's head.Example 1. Autoscaling for an IoT data processorAI was given the task of adding autoscaling for a service that processes messages from IoT devices, and generated a valid HPA:apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: telemetry-processor spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: telemetry-processor minReplicas: 2 maxReplicas: 20 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 Such a manifest works. The problem is that the values 2, 20, and 70% don't follow from the system's behavior — they're just plausible defaults.In a real service, CPU can also be a poor signal. For example, the processor may be waiting for a response from a database or an external service, have low CPU, and at the same time be accumulating a queue of messages.After analyzing production metrics, the engineer changes the scaling model:apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: telemetry-processor spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: telemetry-processor minReplicas: 4 maxReplicas: 12 behavior: scaleUp: stabilizationWindowSeconds: 30 policies: - type: Percent value: 100 periodSeconds: 60 - type: Pods value: 4 periodSeconds: 60 selectPolicy: Max scaleDown: stabilizationWindowSeconds: 600 policies: - type: Percent value: 25 periodSeconds: 120 metrics: - type: External external: metric: name: telemetry_queue_messages_per_pod target: type: AverageValue averageValue: "500" - type: Resource resource: name: memory target: type: Utilization averageUtilization: 75 Here the engineer's key work is not in writing the YAML. They have to determine: how many messages one pod actually processes;whether database connections will be sufficient at 12 replicas;how quickly the backlog grows during a mass reconnect of devices; how much time a pod needs for startup and readiness;whether an aggressive scale-down causes the queue to build up again;how many replicas the infrastructure can afford within the current budget.The number 500 comes not from a prompt but from load testing and production statistics. For example, the engineer might have seen that one pod steadily processes roughly 700 messages in a given time interval, but that beyond 550, latency and the number of database connections rise sharply. That's why a safe target is set at 500.Example 2. A Terraform module instead of a formally correct configurationAI can generate a fully working configuration for two environments:resource "aws_sqs_queue" "telemetry_staging" { name = "telemetry-staging" visibility_timeout_seconds = 60 message_retention_seconds = 345600 } resource "aws_sqs_queue" "telemetry_production" { name = "telemetry-production" visibility_timeout_seconds = 60 message_retention_seconds = 345600 } The code is valid, but as soon as you add a third queue or a new region, duplication begins. The values are scattered across the resources, the naming convention isn't centralized, and the difference between environments isn't formalized.The engineer turns this into a module:locals { queue_name = "${var.project}-${var.environment}-${var.queue_name}" default_tags = { Project = var.project Environment = var.environment ManagedBy = "terraform" } } resource "aws_sqs_queue" "this" { name = local.queue_name visibility_timeout_seconds = var.visibility_timeout_seconds message_retention_seconds = var.message_retention_seconds receive_wait_time_seconds = var.receive_wait_time_seconds redrive_policy = jsonencode({ deadLetterTargetArn = aws_sqs_queue.dead_letter.arn maxReceiveCount = var.max_receive_count }) tags = merge(local.default_tags, var.additional_tags) } resource "aws_sqs_queue" "dead_letter" { name = "${local.queue_name}-dlq" message_retention_seconds = var.dlq_retention_seconds tags = merge(local.default_tags, var.additional_tags) } The values for specific environments stay outside the module:module "telemetry_queue" { source = "../../modules/sqs-queue" project = "iot-platform" environment = "production" queue_name = "telemetry" visibility_timeout_seconds = 180 message_retention_seconds = 604800 dlq_retention_seconds = 1209600 receive_wait_time_seconds = 20 max_receive_count = 5 additional_tags = { DataClassification = "telemetry" Owner = "platform-team" } } But even modularity isn't the main boundary here. The value visibility_timeout_seconds = 180 must be tied to the real message processing time.If production statistics show:p50 processing time: 18 s p95 processing time: 74 s p99 processing time: 128 s maximum expected time: 150 s the engineer can choose a timeout of 180 seconds, leaving a margin for a short-term slowdown. AI can suggest a formula or flag an inconsistency, but it is the human who decides whether to treat p99 as an acceptable benchmark, how redelivery will affect idempotency, and how long a message may remain invisible after a worker fails.Example 3. Version updates without accounting for the compatibility matrixAnother typical case is updating the versions of a runtime, CI/CD actions, Terraform providers, or Kubernetes components.AI might spot an outdated Node.js version and propose an entirely correct update:- uses: actions/setup-node@v4 with: node-version: 22 - run: npm ci - run: npm test Such code is technically correct. But the model doesn't always know the full compatibility matrix of a specific project: the production containers may run on Node.js 20, one of the native dependencies may fail to build on Node.js 22, and the internal packages may not yet have passed regression testing.So the engineer pins the version that matches the production environment:env: NODE_VERSION: "20.18.1" jobs: test: runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: ${{ env.NODE_VERSION }} cache: npm - run: npm ci - run: npm run test - run: npm run build The same happens with Terraform providers. AI might suggest moving to a new major version:terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 6.0" } } } But in a real project some of the modules may be written for AWS provider 5.x and use changed attributes, and the Terraform state may contain resources whose migration must be carried out separately.So the engineer first sets a verified compatibility range:terraform { required_version = ">= 1.8.0, = 5.70.0, < 6.0.0" } } } After this, the update goes through sandbox and staging, with verification of terraform plan, the changelog, state migrations, and regression tests for the internal modules.In other words, AI answers well the question of which version is newer. The DevOps engineer determines which version the whole platform is actually ready to accept safely.Where we don't allow AIIf a mistake is irreversible or costly to the company, only a human may act. Not because AI can't handle it — often it can — but because it isn't accountable for the consequence.A mistake by the model itself, as in the Replit case, isn't the only scenario. AI with access to systems itself becomes an attack surface. According to the IBM Cost of a Data Breach 2025 report, 97% of the organizations that suffered breaches of their AI systems had no control at all over which data, services, and actions the model could reach. The sample is still small, but the correlation is telling.The conclusion is obvious to me: the same least privilege applies to AI as to any service account — minimum rights, strict scope. Whatever the cause of a failure, the scale of the damage is determined by the breadth of access.That's why changing user and service permissions, the payment system, security code, and OTA updates are the engineer's responsibilities alone. Code review at the level of recommendations, advice on security policies, experiments in a sandbox — these are is available to the model, because a mistake here is reversible and cheap.Why AI won't replace DevOps engineersSometimes I hear: just wait a year — models improve every month, and everything you don't trust AI with today it will do tomorrow. I don't think so. And the point isn't quality — both barriers I see lie outside it.The first is real-world context. AI sees the data but not the world behind it. If the volume of data from devices has dropped in one region, the model will honestly flag the anomaly — but I know that the power was cut there for a few hours yesterday. Or the model doesn't know that on the Friday, before the weekend, nobody deploys, because nobody wants to clean up the aftermath on Saturday.The second barrier is accountability. Part of a DevOps engineer's routine can technically already be automated — say, keeping the stack's versions current or rotating certificates. But rotation runs up against money: certificates are purchased, so handing this task to AI means giving the model access to the payment system. And the model isn't accountable for the consequences — if it makes a mistake, it will say “sorry,” and that's it.The industry, it seems, has reached the same conclusion. In a Grafana Labs survey of more than 1,300 engineers from 76 countries, 9 out of 10 see value in AI for forecasting, spotting trends, and root-cause analysis — but it's autonomous AI actions specifically that draw the greatest skepticism by a wide margin, and 95% require the model to show its reasoning. An analyst — yes. An executor without oversight — no.