Can an LLM Forget the Right Things?

Key takeaways : point a normal chat-oriented LLM runtime at a live robot camera and three things break at once: (i) VRAM overflows from a vision stream that never ends, (ii) control-loop deadlines get missed silently, and (iii) a 60Hz camera easily outpaces a much slower reasoning step. The fix: an admission controller that estimates a reasoning chunk’s cost with an online exponential moving average and refuses to even start it rather than gamble on the 33ms deadline; a KV cache that evicts the most redundant frame, by cosine similarity, instead of automatically selecting the oldest one; a lock-free double buffer so perception never blocks on a stale backlog. The engine room: the entire transformer for Qwen2.5-Coder-1.5B-Instruct: RMSNorm, RoPE, grouped-query attention, SwiGLU — is hand-written in CUDA, no cuBLAS, no libtorch; validated against a real HuggingFace forward pass at ≥0.999 cosine similarity. The honest part: this post is an architecture, not a benchmark. Built on a single cloud NVIDIA Hopper GPU (sm_90), the 8GB VRAM ceiling is a modeled budget in the code, not a number measured on a Jetson or any other physical edge board. TL;DR: Point a normal LLM server at a live robot camera and it will happily flood the VRAM, blow through the control-loop deadlines, and choke on a camera that’s faster than its own brain. vla-edge-backend is a hand-written CUDA runtime that refuses reasoning it can’t finish in 33ms, evicts KV cache by ‘meaning’ instead of just age, and never lets perception block on the reasoning. It’s an architecture, not a benchmark; built on a cloud Hopper GPU, with an 8GB ceiling that’s a design constraint, not a field measurement. Architecture mental model: 60Hz camera → lock-free double buffer → admission controller (admit / refuse) → [vision encoder → KV manager (semantic eviction) → hand-written CUDA transformer] → action or fallback. Everything below is commentary on one piece of that line. Perception, admission control, and reasoning are three separate jobs here — and the diagram is the only place all three meet. The Github repo: https://github.com/AnubhabBanerjee/vla-edge-backend 1. The clock that never stops Somewhere inside this project, a camera produces a fresh frame every 16.7 milliseconds. It doesn’t care whether anything downstream is ready. It doesn’t pause, it doesn’t retry, and it definitely doesn’t file a support ticket. It just keeps going, 60 times a second, forever, because that’s what a camera bolted to a robot does. Meanwhile, a small language model wired to a vision encoder has a much harder job: grab the freshest frame it can get, decide what to do about it, and finish that decision inside a 33-millisecond window. Not “usually”, not “on a good day”. Thirty-three milliseconds, for reference, is faster than just the closing half of a single human blink, eyelids take about 50 to 100 milliseconds to snap shut. The robot doesn’t get to blink and think about it. That number isn’t a vibe. It’s a compile-time constant sitting at the top of the codebase, in include/physical_ai/common.hpp: constexpr double DEADLINE_MS = 33.0; constexpr double SAFETY_MARGIN_MS = 2.0; Every other piece of this runtime exists to either hit that number or fail loudly and on purpose instead of quietly missing it — which, if you’ve ever worked with a deadline-blind system, is already a personality upgrade. An upfront confession before we go further, because the whole point of starting a new series is refusing to launder one kind of truth as another: this runtime was built and tested on a single NVIDIA Hopper GPU (sm_90) in a cloud dev box. There is no Jetson in this story. The “8GB VRAM ceiling” you’re about to read a lot about is a design constraint baked into the code, a budget the KV manager is built to respect, not a number measured on a robot’s onboard compute. If you came for a benchmark table off a robot arm in a lab, it isn’t here; that’s future work, and I’d rather say so now than bury it at the bottom. What follows is the architecture of a system that treats a hard deadline and a fixed memory budget as top-priority constraints, instead of doing what most LLM stacks do: pretend neither exists until something falls over, usually expensively. 2. Why your favorite LLM runtime breaks the moment you bolt it to a robot Take any excellent serving stack the LLM world already has, and point one at a live camera feed instead of a chat window. Three things will go wrong almost at the same time. VRAM explodes, because chat-oriented runtimes assume a conversation eventually ends. A robot’s camera never sends a “goodbye” token. It produces new visual tokens every frame, forever, and a context window that keeps growing to hold all of them eventually runs out of VRAM on an edge GPU, uncomfortably soon, since edge GPUs don’t have a data center’s spare closet. Deadlines get missed silently, because a chat completion that takes four seconds instead of two is mildly annoying, while a robot control step that takes 45ms instead of 33ms is a physical action that happened after the moment it was supposed to. Standard inference runtimes have no notion of a deadline at all. They compute as long as computing takes, then hand back an answer whenever it feels like it, like a contractor who “will be there sometime Tuesday.” The frequencies don’t match, because a camera can push 60Hz and a vision-language-action model doing real reasoning is not going to keep up with that pace. Something has to sit between the two, or perception either blocks the camera (now it’s lying about when things happened) or piles up a backlog of stale frames while reasoning falls further behind reality. Put it in three lines and it reads like a status report nobody wants to send: VRAM: filling up like a group chat nobody has the heart to leave. The deadline: missed, quietly, with no one in the runtime even filing the incident. The camera and the model: running two different races, at two different speeds, and only one of them has noticed. None of this is a knock on vLLM, TensorRT-LLM, or llama.cpp’s own server, they’re excellent at what they were built for: serving as many chat-shaped requests as a data center can throughput. A robot’s camera is not a chat-shaped request; it’s a continuous, unstoppable stream with a physical clock duct-taped to it, and that shape of workload asks a different question, not “how many tokens per second,” but “which 33ms window am I inside right now, and can I finish before it closes.” vla-edge-backend treats those three failure modes as design constraints from line one, instead of retrofitting them onto a chat-shaped runtime later. 3. A camera that never asks permission The perception side of this runtime has exactly one job: keep publishing the freshest frame it has, at 60Hz, and never, under any circumstances, block waiting for the reasoning side to catch up. Reasoning is allowed to be a little slow. The camera is not allowed to care. The mechanism is a lock-free single-producer/single-consumer double buffer: two frame slots and one atomic version counter that says which one is currently valid. Think of it as a reverse “musical chairs” game with exactly two chairs and exactly one player at a time: there is always a free chair, so nobody ever has to stand around waiting for the music to stop. struct PerceptionPipeline { PerceptionFrame frame_buffers[2]; std::atomic published_version; std::thread producer_thread; std::atomic producer_running; std::vector clip_frame_storage; std::uint32_t clip_num_frames; std::uint32_t clip_read_cursor; double producer_start_time_sec; PerceptionFrame last_consumed_frame; bool has_consumed_frame; }; A producer thread writes into whichever buffer isn’t currently being read, then flips the counter: const int write_buffer_index = static_cast(local_version & 1U); PerceptionFrame* write_frame = &pipeline->frame_buffers[write_buffer_index]; std::memcpy(write_frame->pixels, &pipeline->clip_frame_storage[frame_offset], frame_byte_count); write_frame->frame_index = frame_index; const double elapsed_sec = physical_ai_steady_time_sec() - pipeline->producer_start_time_sec; write_frame->capture_timestamp_sec = elapsed_sec; ++local_version; pipeline->published_version.store(local_version, std::memory_order_release); The consumer never blocks. It checks whether the counter moved since it last looked, and helps itself to whatever is freshest: bool perception_pipeline_consume_latest(PerceptionPipeline* pipeline, PerceptionFrame* out_frame, std::uint64_t* last_seen_version) { const std::uint64_t current_version = pipeline->published_version.load(std::memory_order_acquire); if (current_version == 0U) { return false; } if (current_version != *last_seen_version) { const int read_buffer_index = static_cast((current_version - 1U) & 1U); pipeline->last_consumed_frame = pipeline->frame_buffers[read_buffer_index]; pipeline->has_consumed_frame = true; *last_seen_version = current_version; } if (!pipeline->has_consumed_frame) { return false; } *out_frame = pipeline->last_consumed_frame; return true; } No version change, no new copy, just the reader just reuses what it already has instead of waiting around like it’s got nothing better to do. The acquire/release pair on the atomic is the entire safety argument: no mutex, no blocking, no “please hold” music. Getting an actual 60Hz out of this — instead of “60Hz, roughly, on a good day” — needs a hybrid sleep/spin loop, because sleep_for on most of the operating systems is honest to within about a millisecond, which is a fine margin of error when you have tens of milliseconds to spare and an actively embarrassing one when your entire cycle budget is tens of milliseconds: void hybrid_sleep_until(const std::chrono::steady_clock::time_point& target_time) { while (true) { const auto now_time = std::chrono::steady_clock::now(); if (now_time >= target_time) { return; } const auto remaining = target_time - now_time; const auto remaining_ms = std::chrono::duration_cast(remaining).count(); if (remaining_ms > 1) { std::this_thread::sleep_for(std::chrono::milliseconds(remaining_ms - 1)); } } } Sleep through everything except the last millisecond, then busy-spin the rest of the way. Trading a sliver of CPU for actually landing on time instead of landing “on time, ish.” The exact same trick shows up again, almost word for word, inside the admission controller’s cycle loop. Same problem, same fix, no shame in reusing a good idea twice. 4. Teaching the brain to say “no” This is the part of the runtime that commits to a genuinely unusual decision: when the runtime isn’t confident it can finish in time, it does not just give its best shot and hope for the best. It refuses to start. No optimism. No “we’ll make it up on decode.” Just a flat no, delivered before any GPU cycle is wasted finding out the hard way. It is, as far as I can tell, the one part of this codebase with cleaner boundaries than most humans manage on a Monday morning. Here’s the actual decision, short enough to read in one breath: bool admission_controller_should_admit(const AdmissionEmaState* ema_state, double elapsed_since_cycle_start_ms) { if (!ema_state->has_bootstrap_sample) { return true; } const double remaining_budget_ms = DEADLINE_MS - elapsed_since_cycle_start_ms - SAFETY_MARGIN_MS; const double estimated_chunk_ms = static_cast(ema_state->ema_prefill_per_token_ms) * static_cast(VISION_TOKENS_PER_FRAME) + static_cast(ema_state->ema_decode_per_token_ms) * static_cast(ACTION_DECODE_TOKENS); return estimated_chunk_ms ema_prefill_per_token_ms = (1.0f - ADMISSION_EMA_ALPHA) * ema_state->ema_prefill_per_token_ms + ADMISSION_EMA_ALPHA * measured_prefill_per_token_ms; ema_state->ema_decode_per_token_ms = (1.0f - ADMISSION_EMA_ALPHA) * ema_state->ema_decode_per_token_ms + ADMISSION_EMA_ALPHA * measured_decode_per_token_ms; ADMISSION_EMA_ALPHA is 0.2f, meaning new measurements get 20% of the vote, history keeps the other 80%. Prefill and decode get separate EMAs on purpose, timed as two distinct boundaries rather than one number split in half. Prefill is a 32-token parallel pass; decode is eight one-token-at-a-time passes, each with its own transformer forward and its own argmax. Different shapes, different costs, no pretending otherwise. The very first cycle always gets admitted, because there’s no history yet to distrust and every decision after that is grounded in something the runtime measured about its own hardware, not a number copied off a spec sheet. A small aside for anyone who’s spent time around telecom networks: “admission control” isn’t a term this project invented. Cellular and ATM networks have been deciding whether to accept a new call without wrecking everyone else’s quality of service for decades. A robot’s 33ms cycle asking “can I admit this chunk without breaking my one promise” is the exact same decision, just made 30-ish times a second instead of once per phone call. 5. The memory problem: forgetting on purpose Every vision frame processed costs 32 slots in a KV cache with room for exactly N_MAX_TOKENS = 4096 slots total, shared across the system prompt, all retained frames, and a few transient action tokens during decode. Do the division and the ceiling shows up fast: worst case, this cache holds floor(4096 / 32) = 128 frames, and at roughly one admitted frame per 33ms cycle, it fills up in a matter of seconds. After that, every new frame means an old one has to go and the interesting decision is which one. FIFO, the obvious baseline, always evicts the oldest frame, no questions asked. It’s the nihilist of eviction policies: everything is temporary, being young is the only crime. // FIFO eviction baseline — evicts the single oldest retained frame-block on each overflow step. bool kv_manager_fifo_evict_if_needed(KvManager* manager, int slots_needed, double eviction_timestamp_sec) { // Repeat oldest-frame eviction until enough free physical slots exist for the incoming frame. while (manager->num_free_slots num_retained_frames num_retained_frames - 1; ++pair_index) { const float* embedding_a = &manager->pooled_embeddings[pair_index * HIDDEN_SIZE]; const float* embedding_b = &manager->pooled_embeddings[(pair_index + 1) * HIDDEN_SIZE]; const float similarity = kv_manager_cosine_similarity_host(embedding_a, embedding_b); if (similarity > best_similarity) { best_similarity = similarity; best_pair_index = pair_index; } } // ... evict the older frame of best_pair_index, shift retained frames down, rebuild indices ... Think of it as evicting whichever frame has a near-identical roommate, rather than whichever frame simply moved in first. If two consecutive frames look almost identical — the arm hasn’t moved, the scene hasn’t changed — you lose almost nothing dropping one. If a frame captures something genuinely different, it survives, because there’s no similarly-boring neighbor to pair it with and blame. That’s “saliency retention” designed straight into the eviction rule instead of bolted on as an afterthought. Cosine similarity itself is nothing exotic — a 1536-dimensional dot product and two norms, computed on the host: float kv_manager_cosine_similarity_host(const float* embedding_a, const float* embedding_b) { float dot_product = 0.0f; float norm_a = 0.0f; float norm_b = 0.0f; for (int hidden_index = 0; hidden_index (frame_rgb[pixel_offset + channel_index]) / 127.5f) - 1.0f; flattened_patch[flat_index++] = normalized_pixel; } } } // Linear projection: patch_embedding[hidden] = W[hidden,768] @ flat_patch + bias. float* output_patch_ptr = patch_embeddings + patch_index * hidden_size; for (int hidden_index = 0; hidden_index (kv_head_index) * N_MAX_TOKENS + physical_slot) * head_dim; float dot_product = 0.0f; for (int dim_index = 0; dim_index (head_dim)); my_logits_scratch[slot_list_index] = scaled_logit; thread_max_logit = fmaxf(thread_max_logit, scaled_logit); } Two shared-memory tree reductions later (max, then softmax sum), the weighted value-sum hits a genuinely fun problem: every thread owns a disjoint set of KV slots, but all of them write into the same head-dimension output vector, 128 coworkers all reaching for the same shared whiteboard at once. The fix is atomicAdd straight into shared memory: fine on any GPU since Kepler, and much simpler than hand-rolling a second reduction just to avoid a little contention: for (int dim_index = 0; dim_index 5.0e-2f) { std::fprintf(stderr, "hidden max abs diff %.6f > 5e-2 threshold\n", max_hidden_abs_diff); return false; } Cosine similarity of at least 0.999 against the HuggingFace output, max per-dimension difference under 0.05. Not a benchmark but a much more basic claim: this hand-written math is actually computing the thing it says it’s computing, not just producing numbers that look plausible and run fast, though I built the scaffolding for a hard 33ms deadline, and honestly, the transformer itself is currently ~100x too slow to live inside it (~ 3.1s). Everything else in this post depends on that gate passing first. 7. The honest list of what this is not (yet) If you’ve read this far expecting a table of latency numbers: there isn’t one, on purpose. Consider this the part of the post where the runtime sits down, makes eye contact, and lists its own limitations before you can find them yourself. No measured performance numbers in this post. No deadline-miss rate, no p50/p95/p99 latency, no fallback rate, no peak-VRAM measurement — even though the runtime is fully instrumented to compute every one of them, and tools/run_benchmarks.py is wired to run all three eviction policies back to back. This post is the architecture; the measurements are separate, undated work. Built and tested on a Hopper GPU (sm_90), not an edge board. CMakeLists.txt hardcodes CMAKE_CUDA_ARCHITECTURES 90. No Jetson, no DRIVE, no physical robot anywhere in this project’s development loop. The 8GB VRAM ceiling is modeled, not measured. VRAM_CEILING_BYTES is a constant a sampler thread checks every 100ms against a KV cache sized to fit comfortably underneath it is a design target, not a wall this has actually been crashed into. The vision encoder is untrained. Same PRNG-seeded weights every run, real shapes and memory footprint, zero labeled examples ever seen. Any claim about what it perceives would be fiction; this post only claims things about the systems engineering, never about perception quality. Sliding window and FIFO are the same policy here, mathematically, as section 5 already confessed still worth repeating because a less honest project would quietly bury it. The correctness gate covers the transformer, not the whole pipeline. It says nothing about the admission controller’s cost estimates or the semantic policy’s behavior on real footage. That’s what a future benchmarking pass is for. None of this changes the architectural claim that a hard deadline, a fixed memory ceiling, and a mismatched sensor frequency can all be first-class design constraints instead of afterthoughts. It just means the claim, for now, is about the shape of the system, not a number on a chart. 8. Where this leaves things This is the first post in what I’m calling “Physical AI Systems” — not because I have a five-part roadmap already written (I don’t, and I’m not inventing one just to sound organized), but because the questions this project raised are bigger than one repo. What does it actually take for a model to operate under a clock it cannot negotiate with? How much of “robotics AI” is just systems engineering wearing a friendlier name? If you build inference infrastructure for a living: go ask whatever’s serving your models, honestly, what happens when the input never stops and the output has a deadline. If the answer is “it just keeps computing until it’s done,” you’ve found the same gap this project is trying to close. If you build robots for a living: I’d genuinely like to hear how far 4096 tokens and 8GB are from your real hardware budget, and whether a semantic eviction policy would survive contact with real footage instead of a replayed clip. And if this is your first real look inside an inference runtime: you now know what an admission controller, a KV eviction policy, and a grouped-query attention kernel each do, and why a robot needs all three running at once. Most tutorials skip straight to model.generate(), as if the hard part were saying please. This is what’s underneath that call, for a workload that isn’t allowed to be late. Now go find out what your own inference stack does when nobody’s watching the clock. Disclaimer: The illustrations in this article were generated with the help of AI image tools. They are illustrative, not photographic, and any labels visible inside the images are stylized rather than authoritative — refer to the article body and the code itself for precise function names, constant values, and architecture details. The article text, code references, and constant values are not AI-generated.