Pokémon Red's most brutal obstacle isn't a gym leader — it's a doorway that broke my AI

Published Jul 31, 2026, 5:00 PM EDT Maker, meme-r, and unabashed geek, Joe has been writing about technology since starting his career in 2018 at KnowTechie. He's covered everything from Apple to apps and crowdfunding and loves getting to the bottom of complicated topics. In that time, he's also written for SlashGear and numerous corporate clients before finding his home at XDA in the spring of 2023. He was the kid who took apart every toy to see how it worked, even if it didn't exactly go back together afterward. That's given him a solid background for explaining how complex systems work together, and he promises he's gotten better at the putting things back together stage since then. I’ve been spending a lot of time using local LLMs lately, and most of the time the process is boredom-inducing. Load the model, run a CLI command, nod at the tokens-per-second figure as if I understand what’s going on. Those numbers only tell part of the story, as knowing how fast something is doesn’t tell you how long it can stay on task. So when I saw NousResearch had followed up Hermes Agent with a harness that plays Pokémon through a headless Game Boy emulator, I knew what I was doing with my weekend. Seeing if AI could play Pokémon on the AMD Ryzen AI Halo. Local LLM vs Cloud LLM, first to the first gym battle. NousResearch built a Game Boy emulator with a REST API It’s a really neat idea Pokemon-agent essentially runs an emulator, translates it into computer language for your LLM to read, and then shows what’s going on in a web-based dashboard. And I mean everything: you see the screen, badges, the current objectives, and the thinking-plus-action turns taken by your agent. To start with, I wired it into the Lemonade server with Qwen 3.6 loaded. I’ve always wanted to see what a local LLM can handle outside of chat responses and coding, and this seemed the perfect time. I used Qwen3.6-27B quantized to about 17GB, because it was the only model I had downloaded with vision capabilities. NousResearch Pokemon Agent Pokémon Agent plays the classic games autonomously so you don't have to (you should still play them, though). My local LLM spent three hours trying to walk out of a bedroom I don’t blame it, there was a Nintendo plugged in The first few seconds of the local LLM playing Pokémon brought the first hallucination. The title screen appeared, and it decided it was already in Pallet Town and started walking toward Professor Oak’s lab. Nothing happened, of course, but then it noticed “the walk_up actions did not move me,” reasoned the next turn, and decided that the game hadn’t started and pressed A to try and get past the title screen. That’s a closed-loop recovery from a mini PC sitting on my desk, and is more than I expected to see. Then it walked to the SNES in the middle of the bedroom and got stuck looking at it for 20 minutes. Teenagers, amirite? Final tally of 164 thinking turns, 371 button presses, and not even the starter Pokémon to show for it. The cloud model got a Squirtle in 112 turns and still couldn’t leave a building It also cheated, and I don’t entirely blame it I didn’t exactly expect the local LLM to do much, but I had higher hopes for a frontier cloud model. I pointed the same agent at GPT-5.6 Sol via the Codex CLI and loaded the game again. GPT noticed it was on the starting screen immediately and pressed A to skip ahead. It only got better from there, picking up Squirtle as its starter on turn 112, beating the rival on turn 234 with 7 HP left, and reaching Viridian City on turn 672. And then disaster struck, in the shape of a door. It couldn’t get out of the Pokémon Center and stayed there for the rest of the night. Not because of an enticing game console, but because the collision map indicated that the door tiles were impassable. It did a three-step circle for the rest of the night. Final tally: 2,072 actions, one Pokémon, no badges. When it first started moving through Pallet Town, it announced it was heading for the northern edge of town to trigger Oak’s starter sequence. That’s not information that the game gives you, and human players go to Professor Oak’s lab first, see he’s not there, and then try to leave town when he’s nowhere to be found. The LLM didn’t read the screen or go into the lab to check. It knows that entering the grass at the top of the town triggers the Oak cutscene because it’s read walkthroughs online. It wasn’t only using the emulator data to play; it was recalling training data. For what it’s worth, the local model also did this, but it misremembered and thought it had to go to Viridian City to meet Professor Oak to get the starter Pokémon. Both models failed but in different ways Neither failure mode was really the model’s problem Neither model ended up being able to play the game as well as me, and I’m a beginner to the series. The local LLM lost the plot and invented a room when it was on the start screen. The cloud model held a plan in memory and then walked into a wall. I had to intervene with both, and that’s where they became similar. The points I had to nudge the model into a new area were the same. The opening naming sequence, how to get out of Red’s bedroom, how to walk into the grass to trigger Oak’s cutscene, through a front door, and through the Oak cutscene. The common thread is that all of these are scripted events or transitions. They’re not navigation problems, and the models were perfectly happy walking around the environment. The other things they share also caused issues. The state reader reports that you are in Pallet Town before the game boots, because the map ID for uninitialized memory just happens to use the same code for the starting area. And the overlay grid is centered on the player, so the agent treats the constant as a location and never realizes when it has moved. One failure turned out to be a bug in the harness Nothing will work when the map is lying to the model The models were consistent on one thing. Every time they needed to go through a door to the next area, they couldn’t. First it was the stairs out of the bedroom, then the door of the house, and so on. I took the controls, moved the player character one square to exit, and let the model take over again. The door was working; the collision map supplied to the agent was not. I even thought it was my problem, because I’d added a few prompt rules to avoid this kind of halted progress. It wasn’t; even the cloud model with no rules had the same issue. The funny thing is that the local LLM on its first run saw the stairs with its vision model and used them to get out of the bedroom anyway. It ignored the collision map because the multi-modal model trusted what it could reason, rather than the information it was given. The models are ready but the scaffolding isn’t I know where local LLMs have shortcomings, and I fully expected this to show that they’re still behind the frontier models. And I got that, but I never expected the models to fail at the same parts of their task. Neither could remember the room they were in 90 seconds earlier; both had a map that lied about where the doors were; and both failed to sit through the cutscenes. All of those problems come from the agent harness, not the model, and maybe one day Nous will get the thing to beat doors as easily as it can beat battles.