The smolagents bug that made my agent retry the same valid code three times

Third entry in the DEV x Sentry Bug Smash. Entry 1 was a crash with a confusing message. Entry 2 was a freeze the timeout could not catch. This one is quieter and sneakier: valid Python that the sandbox rejects with an error pointing at the wrong thing entirely. When the open issues run out, fuzz By entry 3 every obvious open smolagents bug was already claimed or had a competing PR. So instead of reading the issue tracker I pointed a small fuzzer at the piece of smolagents that runs the most untrusted code: LocalPythonExecutor, the sandbox that executes model-generated Python. The method is boring and effective: feed it ordinary, valid Python one snippet at a time, and flag anything that raises InterpreterError. Valid Python that the sandbox refuses to run is, by definition, a bug, because the model writes valid Python and expects it to work. That surfaced four unreported bugs in one afternoon. This post is about the one I shipped: dict unpacking. The bug config = {**{"temperature": 0.7, "max_tokens": 512}, "top_p": 0.9} Enter fullscreen mode Exit fullscreen mode Merging dicts with ** is one of the most common things an LLM writes. Under smolagents it fails with: InterpreterError: NoneType is not supported. Enter fullscreen mode Exit fullscreen mode There is no None anywhere in that line. The message sends you looking for a null value that does not exist. Why it happens In Python's AST, a dict literal keeps its keys and values in two parallel lists. For a normal entry the key is an AST node. For a **mapping spread entry, the key is literally None, a signal that says "this is a spread, not a key/value pair." smolagents evaluated every key by walking expression.keys and calling evaluate_ast(key, ...) on each one. When the key is None, that call falls through every isinstance branch to the catch-all raise InterpreterError(f"{type} is not supported"). So the spread marker got evaluated as if it were an expression, and the model got blamed for a None it never wrote. Why silence is the expensive part Here is the part the Sentry view made obvious. The error is handled: the agent catches it and feeds it back to the model as "here is what went wrong, try again." But the message names NoneType, and the model's code has no None, so the model cannot act on it. It retries the exact same valid syntax. And again. Every step burns a real LLM call and a slot in the step budget until the run gives up. One bug, one misleading message, three identical failures: Three events on a single issue is not noise. It is the agent stuck in a loop, and without Sentry counting the events you would never see the loop, only a run that quietly underperformed. Sentry's Seer read the same event and reached the exact root cause: smolagents' LocalPythonExecutor doesn't handle dict unpacking (**) syntax: None keys in ast.Dict cause an unsupported type error. [...] evaluate_ast(None, ...) matches no isinstance branch and falls to the else clause. The interpreter raises InterpreterError: NoneType is not supported, the agent retries with identical code, burning steps in a loop. The fix Evaluate the dict pairwise instead of evaluating keys blindly. A None key means "merge this mapping": result = {} for key_node, value_node in zip(expression.keys, expression.values): if key_node is None: value = evaluate_ast(value_node, *common_params) if not hasattr(value, "keys"): raise InterpreterError(f"'{type(value).__name__}' object is not a mapping") result.update(value) else: key = evaluate_ast(key_node, *common_params) result[key] = evaluate_ast(value_node, *common_params) return result Enter fullscreen mode Exit fullscreen mode This matches CPython exactly: spreads merge in order, later keys win, and unpacking a non-mapping raises 'list' object is not a mapping. A reviewer caught my fix being too strict I first gated the spread on isinstance(value, Mapping). Minutes after the PR opened, OpenAI's Codex reviewer flagged it (P2): CPython does not require the Mapping ABC, it only requires an object with a keys() method. Since the sandbox lets users define their own classes, a duck-typed mapping with keys() and __getitem__() would have been wrongly rejected. I switched the check to hasattr(value, "keys") and added a test for exactly that case. AI wrote the code, AI reviewed the code, I kept score. After On the patched build the same line just runs: app: step 1 ok, config = {'temperature': 0.7, 'max_tokens': 512, 'top_p': 0.9} Enter fullscreen mode Exit fullscreen mode One step, no loop, no phantom None. Numbers 4 unreported bugs found by fuzzing valid Python through the sandbox; this is the first fix Misleading NoneType error reproduced on current main and 1.26.0 3 wasted agent steps per occurrence, visible only because Sentry counts events 9 new tests: spreads, double spreads, override order both ways, a duck-typed mapping class, empty spread, non-mapping rejection 406 passing, ruff clean Links Issue: https://github.com/huggingface/smolagents/issues/2552 PR: https://github.com/huggingface/smolagents/pull/2553 The pattern across all three entries: the worst agent bugs do not throw a red stack trace at you. They hand the model a plausible-but-wrong message and let it fail politely, on repeat. Count your events.