Towards Spec-Driven Test Automation: Part 1

IntroductionModern AI coding agents can implement a specification and produce a passing test suite in the same workflow. The problem is that both the code and the tests come from the same reading of the same ambiguous sentences, which means the tests cannot disagree with the implementation. This breaks the core rule that makes Verification & Validation meaningful: the person who builds the system must never be the person who verifies it.I recently worked on General Motors' Super Cruise and Ultra Cruise programs in a Verification & Validation setting, where our entire job was to check whether systems actually did what their specifications said. That environment made the importance of independent verification impossible to ignore, and it’s what led me to build a new open‑source python project that automates that separation.An ExampleLet's first look at an example of the kind of specification line that the rule exists for, taken from a real task specification for a test vehicle's forward radar. It says that the gap to the car ahead "must be a distance the forward radar can measure, up to its rated range of 250m." Up to 250m, starting where? Can the radar measure a gap of 0m? Unfortunately, this type of specification vagueness means that the software developer who is assigned this task will have to make that decision. Who is allowed to verify that decision? V&V methodology ensures that it is verified by an independent testing team, not the software development team.Verification & Validation, in one line eachThe field splits the question "is this system okay?" into two smaller questions:1. Verification asks whether we built the system right: does it meet its own specification, measured against requirements someone wrote down before the code existed.2. Validation asks whether we built the right system: does it actually solve the problem, out in the world, for the person using it.For example, a perception module can verify perfectly against its spec and still fail validation, if the spec described the wrong thing. Most engineering teams know this distinction in the abstract. Fewer people outside the field know the part that makes verification worth anything at all, and that part is not in the two sentences above.Why the coder cannot be the graderOnly somebody who did not write the code is positioned to read the specification in a truly alternative way, looking for the cases a coder in flow might skip. When the same person writes the code and the check, the check inherits every assumption the code was built on.Let's say a specification tells an engineer to "reject malformed input" but never defines malformed. Someone has to decide what that word means. If the person deciding is also the person who will later write the test that confirms the decision was correct, there is no decision being tested at all as the test is biased and essentially just a form of self-confirmation. So no surprise here that the test will pass due to the lack of an independent reviewer.Give the same ambiguous sentence to two different engineers, one to implement it and one to check it, and something different happens: two readings exist, and they might not agree. When they don't, that mismatch is a form of new information. It tells you the sentence was genuinely ambiguous, in a meaningful way, before the system ships rather than after.That is the key idea. For traditional product development, it has been standard practice for decades. I experienced this firsthand during contract work I did at NASA, doing the full hardware and software calibration of head-tracking and eye-tracking equipment in the cockpit. I calibrated the system and a fully independent quality assurance team then verified it. This idea isn’t new. What’s new is that software tools quietly abandoned it just as AI made it cheap to automate.What changes when the coder and the grader collapse into one modelHere is the setup that got me thinking about all of this again. Ask a modern AI coding agent to implement a specification, and it hands you back two things: working code, and a passing test suite. Look closely at how it produced the passing test suite. It read the same specification once and resolved every ambiguity in it exactly one way. Then it wrote the implementation from that reading, and wrote the tests from that same reading. Where the spec said "reject malformed rows" and never said what malformed meant, it picked a definition, built the code around it, and then wrote a test that checks for precisely that definition.Everything passes, the test suite goes green. Of course it does. One model read one sentence one way and then confirmed, in a second pass, that it had read the sentence the way it had just read it the first time. That is not verification. The obvious objection is to turn the temperature up, so the two passes do not have to land in the same place. That buys inconsistency rather than independence, and the difference matters. When two samples of the same reader disagree, nothing tells you which reading was the right one, and the disagreement says nothing about the specification that produced it. When two different readers disagree, that tells you the sentence could be read two ways, which is a fact about the sentence and is exactly what you wanted to find out. Sampling noise cannot do that job. So my agents both run at temperature zero, and the thing I vary is what each of them is allowed to see.I'm not talking about some hypothetical failure mode that is specific to one tool or one vendor. It is closer to the default architecture of an AI coding assistant today: one context window, one model, asked to produce an implementation and, in the same breath or the next prompt, asked to produce the tests that will judge it. Safety-critical industries like automotive and aerospace stopped allowing an engineer to test their own work a long time ago. But in many other industries this mode is now the default way a large share of AI-assisted software gets written.Automated Spec Driven Test DevelopmentCurrently, most tools write tests for code that already exists, which means the code implementation shapes the tests. Instead, in order to achieve truly independent tests, we can generate our own independent code implementations derived only from the requirements (the spec), and that way the key separation of duties is enforced in code.What the new AI workflow looks like for V&VIndependence has always been the expensive part of V&V. It means a second team, a second reading of every requirement, and a sign-off process that keeps the two apart, which is probably why it stayed inside safety-critical engineering and never spread to ordinary software. AI agents now change that cost equation. Two agents can play the two roles, and software, not an org chart, can decide who is allowed to see what. That is the part I think is new: not the V&V rule itself, but running it automatically, on every task, with the separation of the acceptance criteria enforced in code and paid for in model calls rather than headcount.I just built an open-source Python tool, qikly, to enable exactly that. The source is on GitHub at: https://github.com/gal-a/qiklyThe workflow is easier to see than to describe. One specification is split between two teams, and the diagram below shows both who holds each half and when each half comes into existence. The dev team holds the requirements and builds from them, and never sees the acceptance criteria. The QA team holds the criteria and writes the suite from them, before any code exists. At check time the suite runs, and two things come back: failures to the coding agent as error text and nothing else, and whatever the run revealed to the criteria and the tasks. Dev vs QA Timeline (Source: Claude Opus 5)It starts with a task which is a single YAML specification divided into 3 parts:The requirements say what the code must do, in the words a person would use, which often includes unintentional ambiguity.The interface names the module and the function signatures, as a description rather than code. The acceptance criteria are the specific, checkable statements of what must be true if the requirements were implemented correctly. Here is a trimmed piece of a real one:This is the radar line from the start of this piece. The requirement gives the upper limit and leaves the bottom of the range open; the criterion closes it, a gap of exactly 0 is rejected, and only the test-writing agent ever reads that sentence.The test-writing agent works from all of it. It reads the requirements, the interface and the acceptance criteria, and writes the integration and system tests before any implementation exists. It never reads the implementation for those stages, because there is none yet.The coding agent works only from part of it. It receives the requirements and the interface. The acceptance criteria are stripped out of the task in code before its prompt is assembled which means no representation of them exists in its context to be recalled, prompted around, or accidentally referenced. A test in the project's own suite fails the build if any code path ever lets a criterion through, including one a future contributor adds without having read this far.You do not have to take that on trust. qikly --explain ADAS_HEADWAY --html prints the same task file as each agent receives it, built by the functions a real run uses, so the criteria are visibly present on one side and visibly absent from the other.Coding Agent with Acceptance Criteria Removed (Source: Claude Opus 5)Then the suite runs, one stage at a time. Integration tests first, then system tests. When a test fails, the coding agent sees what pytest prints: the test name, the failing line, and the assertion error. That is what any developer sees when CI turns red. What it never sees is the acceptance criteria. It has the requirements, like any developer, and has to work out from the failure which rule it broke. It writes a FIX (its reasoning about what the failure means) then a PATCH (a diff to its own code). Clearing a stage re-runs the earlier ones, so a later repair cannot quietly break something that already passed (this is regression testing). The loop is where almost all of a run happens.One thing worth being straight about. Pytest also prints the failing test's own source, and a generated test's docstring usually restates the rule it was written from, so a failing test tends to give away its own case. That does not undo the split. The test suite was written first, from criteria the coder never read, and nothing the agent learns afterwards can change a test that is already on disk. The latest release adds an option to narrow the feedback anyway: the agent can start with a one-line error and see more only once a patch stops making progress. It is off by default, because I have not yet measured what starting narrow does to the convergence rate.Unit tests come last, and they are the one exception. Unit tests have to name real functions, so they are written from the code that just cleared the earlier stages. By this stage the behavior has already been checked against a standard the coding agent never saw.A run ends one of two ways. It converges, and you keep qikly's code implementation, a pytest suite and the full record of every FIX and PATCH. Or it spends its retry budget and stops: it exits non-zero, names the tests that blocked it, and nothing gets shipped. Importantly, there is no path by which the tool reports success on code its own tests reject.The model improvement paradoxOne obvious objection is that the tests still run, and they still pass, so why is that a problem? The answer is that a test's entire value is that it might fail. Otherwise what is the point of running it?A good test represents a question genuinely independent of the answer being checked. When the code implementation and the test come from the same resolution of the same ambiguity, the test cannot disagree with the code, so a green result carries no information about correctness. It only confirms that the model was consistent with itself, which was never in doubt.Paradoxically, what makes this worse over time, rather than better, is model improvement itself. A more capable, more deterministic model resolves an ambiguous sentence the same way more reliably, not less. Every gain in consistency tightens the agreement between the code an agent writes and the test it writes to check that code, which looks, from the outside, exactly like getting better at testing. But the opposite is true. It is getting better at producing a test suite that was never capable of disagreeing with the code in the first place.The usual first fix people reach for is to use a second model for the tests. That helps a little but it does not resolve the actual problem, because both models are still reading the same specification and the same acceptance criteria, and both are still free to resolve any ambiguity the same way a careful reader would.Model diversity changes who is looking. It does not change what they were shown. The newer version of the same fix, an independent reviewer agent that checks a first agent's work, runs into the identical wall: a reviewer handed the same specification has read the same criteria and will resolve the same ambiguous line the same way. It will catch an obvious bug or a plainly skipped requirement. It will not catch the case that matters most, an ambiguous line that could be read two ways. Within the setup nobody is wrong, which is exactly why nothing looks wrong.Avoiding common automation problems like endless loopsAutomation only helps if each specification line ends up on the correct side, and one question decides which side that is:Given only the requirements, could two competent developers legitimately disagree about this line in the spec?If yes, it is a decision, and it belongs in the requirements, where the coder can see it. If no, it is a consequence, and it belongs in the acceptance criteria. Mix ups. How to deal with these common mistakes:A decision in the acceptance criteria. The coding agent is forced to guess a choice it was never told, and usually the loop makes that visible. The signature is repetition: either the same test failing while the FIX and PATCH come back near identical each time, because nothing the agent can see would lead it anywhere else, or two tests that disagree, where each patch makes one pass and the other fail. Sometimes, though, the agent simply guesses right and the run goes green, and that is the worse outcome, because nothing then tells you a decision was in the wrong half. Only a human can fix it, by moving the decision into the requirements.A consequence in the requirements. Both agents read the same boundary value, so the test that checks it passes on the first attempt and proves nothing. The rest of the suite is unaffected and still bites, which is what makes this one easy to miss: the run looks entirely normal. Only a human can fix it, by moving the consequence into the acceptance criteria.That question is worth asking of any spec, whether or not an agent ever runs on it. We will get back to this question soon.That's basically the design. Whether it survives contact with a real specification is a different question, and the next post in this series answers it: one run followed to the end, the mistake I made writing the spec the first time, what happens when the code already exists, and the numbers I have, and the ones I do not.Gal Arav is the author of Applied Statistics for Data Science (qikly.com) and maintains his new open‑source project for spec‑driven test automation at:https://github.com/gal-a/qiklyReaders are encouraged to give it a spin, and any (human) feedback is much appreciated as the codebase continues to evolve.