Why AI Productivity Is a Faulty Metric

The most dangerous AI productivity metric is the one that is easiest to screenshot.Prompts sent, tokens spent, suggestions accepted, lines of code generated. These numbers look impressive on a dashboard, but they often mask a more pointed question: “Did our software actually get better?” We’re now using AI coding tools at a pace few predicted years ago. I've spent my career on both sides of the equation, first building the systems that measure developer productivity, and now watching how those measurements get used (and misused) once AI enters the picture. Today, leading product at Stack Overflow means sitting at the intersection of both: the knowledge technologists rely on, and the tools now built to access and contribute to it.In fact, our own annual survey found that 84% of respondents use or plan to use AI tools in development work — but 46% don’t trust the accuracy of AI’s output.This is why AI productivity can be a misleading KPI. So often, we turn the complexity of engineering and software development into a flat number that executives can point to. Our rush to adopt AI has outpaced trust in its results, or even in the context and knowledge it is given to deliver those results.Even still, many companies continue to measure success by how often the tools are used, not by whether the software has any meaningful impact or improvement.The dashboard is not the workDashboard metrics are easy to collect and explain when having to justify the use and cost of AI tools to the C-suite. However, this is a rollout question masquerading as a productivity question.A prompt count tells us that developers are turning to AI, but it is not an indicator of whether or not those answers were helpful. Token spend may rise every month, but the software itself may not improve. Accepted suggestions offer one part of the story, but what we need to know is what survived review, testing, and the human scrutiny that comes before code goes into production.Seeing more generated code can look like progress, but the technical debt of review, testing, and bug fixes that follow is often ignored. Faster writing does not equal faster deliveryThe first wave of AI coding productivity focused on speed. AI could "steer" us in the right direction when prompted, making us feel faster. For many, this was determined to be the golden solution to getting more done in the same amount of time. For example, researchers leading a GitHub Copilot study found that developers finished a JavaScript task 55.8% faster with the tool.However, most software engineers know that increasing speed is not a neat exercise, especially when it comes to the quality of work. Software engineering is messy. Old code breaks, tests suddenly fail and no one knows why. Adding speed for the sake of increasing productivity makes it worse.The result of speed for the sake of it throws a burden on reviewers to determine if all this new generated code fits the product, not just whether it runs. This is where the first major productivity metric can be found: Human judgment.Developers can generate new code in seconds and push to review but they still have to ask: Is it correct? Is it maintainable? Is there any risk? The more AI code produced that meets these questions and passes, the more successful the productivity is.The work does not disappear — it movesUsing AI, junior developers produce more code and senior developers now spend their time reviewing. The work simply moves, which can make productivity feel higher than it actually is. METR tracked 16 open-source developers working on 246 real tasks. The developers expected AI to make them 24% faster and by the end, they felt 20% faster, but the results showed they took 19% longer.The lesson is that perception is not proof. A tool can feel fast and still slow the path to production. This is the gap leaders need to measure.The baseline never existedThere's an assumption buried in every "AI made us X% faster" claim: that we knew how fast we were before. We didn't. Most engineering orgs never rigorously measured human-only productivity — not because it wasn't valuable, but because it's genuinely hard to instrument, and there was no urgent business reason to fund it. Velocity was a vibe, not a dataset.So when a company reports a productivity lift from AI, it's often a comparison against a baseline that was never actually measured. A memory of how things felt, not a number that existed. That's a structural problem, not a knock on the intent behind these studies. You can't rigorously measure a delta when one side of the equation was never rigorous to begin with.In this new stage of AI adoption, it is often the first time organizations are measuring developer productivity seriously at all. It didn't just add a new variable. It's forcing the measurement discipline that should have existed for human-only work all along.What engineering leaders should measure insteadThe best AI metrics are software metrics sprinkled with AI context. For example, measuring markers such as if AI-generated code survived review, improved quality, sped up delivery or reduced your organization's workload (and didn't just shift it elsewhere) can provide additional meaningful context. Numbers like these are harder to gather than looking at a token dashboard — but that's the point. They are closer to what actually matters. Good metrics pair AI activity with outcomes such as:Accepted suggestions next to defect ratePull requests next to review cycle timeLines generated code next to the code retained for a set period of time (e.g. 60 days) Perhaps most importantly, developer satisfaction should sit next to the time spent building, testing, and fixing AI-generated code. All of these markers paint a more complete picture that effectively demonstrates how AI may help one area but add drag to another. Every AI initiative will eventually elicit a discussion of ROI, which despite best efforts, remains hard to prove. So, we need to stop asking, "How much AI did we use?" but instead reflect on what changed because AI was used, and then expand from there. Did we reduce our backlog? Do we have less critical bugs? Are engineers spending less time waiting for answers? Did we shrink our onboarding time? These questions will keep us grounded in the reality of what AI tools are providing for teams.The cost nobody puts on a dashboardAI has a startup cost, and it isn't the license fee. It's the maturity of the knowledge you're feeding it. If your organization's knowledge is scattered across a dozen tools, half-documented, stale, or quietly contradicting itself, AI doesn't fix that. A model answers confidently from outdated or conflicting sources with the same tone it uses for accurate ones. There's no intuition behind the answer, no sense of "this doc is old, go ask someone." A senior engineer knows which wiki page to distrust. AI, by default, does not.This dynamic only gets worse at scale, not better. Tribal knowledge used to be self-limiting: you asked the one person who knew, in a conversation where they could add caveats and "well, it depends." That friction was doing real work, filtering who accessed what, and how it got interpreted. AI removes the friction. The same fragile, informal knowledge that used to live safely in one senior engineer's head is now queried by anyone at any time, but stripped of the context that made it safe to rely on in the first place. Easier access to incomplete context isn't a productivity win — it's a bigger blast radius. Which is, not coincidentally, the problem I spend my days on.It's also why a single productivity number is the wrong approach for the question. Software development is a lifecycle, not one task, including ideation, implementation, review, testing, maintenance, and incident response. AI's speedup is real but uneven across it. Bug bashing and boilerplate move faster. Ideation moves faster. But complex system design and anything that depends on organizational knowledge rather than syntax still needs a human who can tell good context from bad. Collapsing all of that into one "X% faster" headline hides exactly the parts of the lifecycle where speed without judgment is most dangerous.Trusted software is the goalWe don't need to slow AI adoption. AI is here to stay and it will only get better over time as it leverages training data developed by actual humans. But we don’t need to give AI credit for productivity gains until the work proves itself. Progress isn't a prompt entered, a token used, or code generated. It is what teams confidently ship that survives review, reaches production and delivers value. Measure those, and you will learn more about your organization's AI productivity than any token dashboard can show you. Otherwise, you're just measuring activity.The 46% of developers who don't trust the output of AI is not a side note. That's the productivity gap, measured in a different unit. If we figure out how to close that gap, the dashboard numbers will finally reflect the headline productivity gains that everyone hopes have been true all along.