The Slop Should Not Be Tolerated

TL;DR: Coding agents decide they're done by asking a model whether they're done. That's not a quality check, that's a vibe. Here's what happens when you make the exit condition a command and why the agent has to be locked out of changing it. A harness is easy; the rest is not No, a markdown won't fix it Merriam-Webster crowned "slop" as its ‘2025 Word of the Year’, which captures the collective exhaustion with low-quality content mass-produced by AI. Both big AI labs shipped a `loop` this year: Codex has /goal , and Claude Code has /goal and /loop. If your harness is "run the agent again," you're playing table stakes because the loop was never the hard part - the challenge is at the end when something has to tell an agent 'stop' or 'continue,' and then a human has to stamp it. Short version: you hand an agent a prompt, it does work, it commits. Repeat. Sometimes you use a database as a memory, sometimes the code is the memory, sometimes you have a web of agents working at the same time. It's so simple and it just functions. The catch is the exit condition, where something or someone has to decide when an iteration is finished and that the code is 'good enough'. And that's why harness engineering is becoming its own field and AI slop has Xfka Twitter all abuzz. Do YOU look at the code? Model-graded “done” is not done Here's how a loop knows you're finished: an agent model looks at the work and grades it. The thing that wrote the code decides the code is good. You know how this goes:"✅ All tests passing! Feature complete! Test coverage is 100%”... But take a peek. Every test-covered line runs under a test that asserts... nothing. There's also a # type: ignore where the types were being too fussy for the agent to deal with, and now a # noqa where the linter complained. Anthropic has names for this in their own failure list: premature victory declaration, fake-done features. Run the same agent review three times, get three answers. Anthropic describes familiar failures in its engineering write-up on long-running agents. An agent sees existing progress and concludes that the project was complete. Then other sessions attempted too much and left unfinished work behind. So Anthropic gave the agent a feature checklist and a progress log so each new session could see what still needed work. But a checklist is useful only if you can test the items on it; otherwise, it's a wishlist. “Supports CSV” is not a spec; it's a wish. Can the code handle a comma inside a quoted field? Does a bad row produce an error without saving half the file? Those are things you can run and check before marking the feature done, and the agent saying “CSV support implemented” is the agent grading its own homework. That's not a stopgap. That's a vibe Calling a feature done too early matters even more when the next iteration builds on it. That's how unfinished work becomes a codebase full of AI slop, low-quality generated code dressed up as finished work, runs, passes tests, and is still full of duplicated logic, hidden failures, abstractions, and spaghetti that make a simple change viscerally painful to look at. You ask for another field on a form and discover an added factory, two adapters, three fixtures, a helper, and a meeting with whoever named AbstractFieldOrchestrator. Nobody signed off on that name, and it just showed up one night and got tenure. And that code lingers on as memory where each agent iteration builds on previous decisions and yesterday's shortcut becomes today's architecture. One copied block becomes five, an exception gets swallowed because it's the happy path (yay!), and by the time you review the whole thing, the diff is large enough that “well, the tests pass” starts sounding like a good basis for a career decision. Something smells. Researchers are starting to measure AI slop. The May 2026 revision of SlopCodeBench evaluated 15 coding agents on 36 problems and 196 checkpoints. Agents had to keep extending their own code as requirements changed and across those runs about 3 out of 4 packed more complexity into already complicated functions, accumulated redundant code, and overall just got messier. Telling the agents to prioritize quality improved their starting code but it didn't stop deterioration.SlopCodeBench charts showing code becoming more complicated and redundant as agents extend it. The benchmark doesn't prove every loop makes code worse or that some harness fixes it. It shows why 'it works' is only part of the story because the code has to survive the next feature request. I'd rather discover the mess while the diff still fits in my head. In “An Endless Stream of AI Slop,” researchers analyzed 1,154 posts across 15 Reddit and Hacker News threads about AI slop and found recurring complaints that time saved generating code becomes extra work for reviewers and maintainers, which is a lovely productivity gain if you don’t count your coworkers or your future self. And humans aren't especially reliable judges of progress either. InMETR's early-2025 randomized study 16 experienced open-source developers tackled 246 tasks in repositories they knew well. With AI tools allowed, they took 19% longer on average. Afterward, they still estimated that AI had made them about 20% faster. Apparently you can lose time and enjoy the experience. Finally! A productivity tool with the same business model as scrolling your phone. Screenshot comparing forecasts and developer estimates with measured results In its February 2026 follow-up METR said newer tools likely provided more speedup, although selection effects and measurement problems made the new estimate unreliable. Is that the writing of better code or faster slop? That study measured time, not code quality, so we can't quite answer that question. A harness needs to check what got built as well as how long it took. But you say watching a thousand lines of code arrive is exciting! Code goes brrr... Discovering you only needed forty is less exciting, especially after you've read all thousand. So what to do?Gate it Before the next iteration builds on a change, something needs to check whether it actually meets the requirements. You want a loop to come back with evidence. Run a formatter, type checker, regression tests, and whatever security checks make sense for the project... all of it. I don't care how confident a commit message sounds, keep failures visible and feed them into the next attempt. Let's use coverage on a CSV parser as an example since a line running under a test doesn't mean the test really checks the result. Call the CSV importer and you've exercised some code with 100% coverage. But check the saved rows and the error message and you've tested behavior. An application's parse_csv that blindly splits on every comma would fail this test and one that returns anythingwon't just get a participation trophy. For a parser that returns a list of dictionaries, an explicit test could look like: test_comma_inside_company_name(): rows = parse_csv('company,amount\n"ACME, Inc.",12\n') assert rows == [{"company": "ACME, Inc.", "amount": "12"}] Then, add mutation testing to test the tests. This goes further by making small changes to the source code and seeing whether the tests even notice or just wave things through like a bored bouncer. **Have you ever had an agent mock basically everything in a test, essentially creating a mirror app? \ And don't even get me started on frontend. How do we deal with a lack of determinism? Deterministic unit tests won't catch visual degradation and an agent can pass every test and ship a blank page, a layout that falls apart on mobile, or a form you can't use with a keyboard. Playwright, a11y, and Lighthouse might catch broken DOMs and layout shifts but none of them can explain why every button looks like it's trying to sell me crypto. Someone still has to open the thing and look at it and agents are NOT at the level to build and test good frontend. For visual regression gates: Does the page render and load its production assets? Do the controls respond, including when used with a keyboard? Does the layout fit mobile screens and meet accessibility standards? .... (As an aside, how will we even begin to test for style? Or taste?) Once checks decide if an agent in a loop can continue, the checks themselves need protection too! An agent told to make tests pass can delete the assertion that's failing. Green! Same bug survives. I don't want the agent fixing the implementation to also decide which requirements count, so required validation belongs somewhere an agent can't quietly bypass, for example, in CI. Tests can change if and when requirements change and “this assertion was making my job difficult” is not a requirements change, it’s a confession. You don't have to be a nanny …and you don’t have to be afraid to look at the code In a harness, the control flow can stay simple with some `trusted_checks` that live outside an agent's editable workspace: for each attempt in 5: agent.work(task) result = trusted_checks.run() if result.passed: agent.commit() else: raise Run the required checks after each attempt: If they fail, give the errors back to the agent to fix. The agent saying “done” should NOT count as passing. Make checks mandatory: Run them automatically at set points instead of adding “Please run the tests” in a markdown Keep changes reviewable: Start with one bounded task and enforce diffs small enough to review (research hasn't agreed on one number but hovers around the number ~200 lines of code as "ok" before human eyes glaze over) Check for slop, not just bugs: Run duplication and complexity checks - code can pass every behavior test and still be miserable to maintain Check before merging: Have fast checks that can run while an agent works and have deeper validations that allow a merge Stop the reruns: Set max attempts or time limits to bound runtime and cost instead of paying for another hour of the same wrong attempts politely rephrased Keep the standard independent: In an existing repo start with its tests and conventions and add checks for problems you'd actually encounter (especially end-to-end behavior) Don't trust, do verify: Keep tests and implementation separate and written or reviewed by a different pass than the one that wrote the code Lock the checks out of the workspace: Trusted_checks should live somewhere the agent can't `git` or `mv` its way around, e.g. CI config it has no write access to, otherwise "fix the bug" quietly becomes "fix the test that caught the bug" Fail loudly: Get a plain summary of what's broken (especially if a human has to pick up the thread) instead of re-reading five commits of "attempt 4, trying again" Automate test permutations: Add property tests that vary test inputs in countless ways so you don't have to write huge test files Watch the tests, not just the code: Add and run mutation testing to catch tests that always pass - the software equivalent of a smoke detector with no battery Loop it: If anything changes after checks pass you have to run them again - yesterday’s green check doesn't apply anymore regardless of what code was touched That's how I'd start to judge a harness. The number of iterations it can survive overnight is just a billing question and if 90% of written lines are garbage (though garbage that does run) you might have a problem. A loop can give you more attempts. That's table stakes. Making loop attempts useful requires a definition of done that survives human contact. Disclosure: I wrote LoopGate my own harness that runs agents in a loop with configurable quality checks and also attempted a frontend-focused harness which still gives me a migraine