From API to GPU, Week 2: What Actually Happens Behind the API

Phase 1 of 8: Comfortable running local models. Week 2 of 32. In Week 1 I went through the machine and proved the GPU works. This week I run an actual language model on it and, for the first time, look at what sits behind the "AI API" I have called for years. The short version: the API is not magic. It is an HTTP layer over a local process that loads a file of numbers into memory and does the matrix math from Week 1. By the end of this post that sentence will be clear, backed by real commands and real output you can reproduce. I am using Ollama, a runtime that makes running a local model about as easy as running a container. Other names you will see in this space are llama.cpp for lightweight local inference, vLLM for high-throughput serving, and NVIDIA TensorRT-LLM for optimized NVIDIA inference. Hugging Face Transformers is also common, but it is a broader Python framework for running and training models, not a ready-made local model service. These tools overlap, but they are not exact replacements. I chose Ollama because it gives me a CLI and local HTTP API with very little setup.1 Everything below runs on the DGX Spark and is reached over SSH as spark, the same setup as Week 1. Concept 1: a model is not a runtime The first idea to keep straight, because it holds for the rest of the series, is the split between the model and the runtime. The model is trained data: weights plus the metadata needed to use them. On its own it does nothing. It is data on disk. The runtime is the program that loads those weights into memory and runs the math to turn your prompt into text. Ollama is the runtime here. If you know Docker, there is a useful analogy, but it is not exact. A model package is like an image made of versioned, content-addressed layers. Ollama is like the engine that pulls those layers and starts the workload. Unlike a container image, the main model layer is trained numeric data, not an app and its operating-system files. Ollama calls each stored package file a blob. Here, a blob is just a file kept under a name derived from its content digest. The inspection below follows the package index to the model blob and checks that it is the expected file. Rather than run a wrapper script, I inspected the model with a few direct commands, one at a time. Each command answers a single question, so you can paste it, read the output, then move to the next. First, list what Ollama has downloaded locally: ssh spark 'ollama list' Enter fullscreen mode Exit fullscreen mode NAME ID SIZE MODIFIED nomic-embed-text:latest 0a109f422b47 274 MB 2 days ago llama3.2:3b a80c4f17acd5 2.0 GB 3 days ago phi4:latest ac896e5b8b34 9.1 GB 7 days ago qwen3.6:35b-a3b-bf16 94061ddd23a7 71 GB 8 days ago Enter fullscreen mode Exit fullscreen mode Next, ask Ollama for the focused facts about Phi-4. The Ollama server listens on localhost:11434 on the Spark, so run this in a shell on the Spark (ssh spark): curl -s http://localhost:11434/api/show -d '{"model": "phi4"}' | jq '{ format: .details.format, architecture: .details.family, parameters: .details.parameter_size, context_length: (.model_info | to_entries | map(select(.key | endswith(".context_length"))) | first.value), embedding_length: (.model_info | to_entries | map(select(.key | endswith(".embedding_length"))) | first.value), quantization: .details.quantization_level, capabilities, runtime_parameters: (.parameters | split("\n") | map(select(length > 0) | gsub(" +"; " "))) }' Enter fullscreen mode Exit fullscreen mode { "format": "gguf", "architecture": "phi3", "parameters": "14.7B", "context_length": 16384, "embedding_length": 5120, "quantization": "Q4_K_M", "capabilities": [ "completion" ], "runtime_parameters": [ "stop \"\"", "stop \"\"", "stop \"\"" ] } Enter fullscreen mode Exit fullscreen mode That call gives readable model facts, including "format": "gguf". For a normal check, this is how I know Ollama identifies the model as GGUF. I do not need to locate the raw file and inspect its bytes just to answer that question. To see the exact files that make up the package, read Ollama's manifest. It is a small JSON file on the Spark, so jq can read it directly. I select just the package version and the file list: jq '{schemaVersion, layers: [.layers[] | {mediaType, digest, size}]}' \ /usr/share/ollama/.ollama/models/manifests/registry.ollama.ai/library/phi4/latest Enter fullscreen mode Exit fullscreen mode { "schemaVersion": 2, "layers": [ { "mediaType": "application/vnd.ollama.image.model", "digest": "sha256:fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20", "size": 9053114464 }, { "mediaType": "application/vnd.ollama.image.template", "digest": "sha256:32695b892af87ef8fca6e13a1a31c67c1441d7398be037e366e2fc763857c06a", "size": 275 }, { "mediaType": "application/vnd.ollama.image.license", "digest": "sha256:fa8235e5b48faca34e3ca98cf4f694ef08bd216d28b58071a1f85b1d50cb814d", "size": 1084 }, { "mediaType": "application/vnd.ollama.image.params", "digest": "sha256:45a1c652dddc9efdcefa977ab81cfbe26b6e52bc8e78f2f4c698538783e0ac80", "size": 82 } ] } Enter fullscreen mode Exit fullscreen mode The registry path ends in library/phi4/latest, so this manifest is the index for the model name phi4 and tag latest. It does not hold the 9.1 GB of weights. It lists the files that make up the Ollama package, much like a lock file maps package names to exact artifacts. Each entry under layers is one file in the package (one blob). It has three useful fields: mediaType says what the file's role is. This package has a model, a prompt template, a license, and default parameters. digest is the file's content identity. The sha256 prefix names the hash algorithm, and the characters after the colon are the hash of the file's bytes. size says exactly how many bytes that file should contain. The model layer's digest is also how Ollama names the file on disk. It stores each blob under a blobs directory using the digest as the filename, with the colon changed to a hyphen, so sha256:fd7b...df20 becomes sha256-fd7b...df20. That file is 9,053,114,464 bytes, the size shown above, and its SHA-256 matches the digest. The linked investigation independently checks the file header with xxd: bytes 47 47 55 46 spell GGUF in ASCII. That deeper check is useful when reusing the raw file in another runtime, but .details.format is the simple Ollama API answer.2 So ollama show and the manifest answer different questions. The metadata call gives readable model facts such as architecture and context length. The manifest tells Ollama which exact files form the runnable package and where to find them by content ID. You might now wonder whether another runtime can use the same GGUF file. That is useful, but it is not part of the main Ollama lesson. I answer it near the end in Can llama.cpp reuse this Ollama model?. I am using phi4, Microsoft's 14.7B-parameter model. At 16 bits per weight, 14.7 billion weights alone would need about 29.4 GB. The local package is only 9.1 GB, which leads to the first surprise. A quick tour of what each line means, since these terms come up constantly: Field Value Plain meaning format gguf model file format reported by Ollama architecture phi3 the neural-network design (phi4 reuses the phi3 family) parameters 14.7B how many trained weights the model has context length 16384 the most tokens (prompt + reply) it can consider at once embedding length 5120 width of each token vector; covered later quantization Q4_K_M the weights are stored at about 4 bits each, not 16 That last row answers the 9.1 GB surprise. This build is quantized: its weights use a mixed low-bit Q4_K_M representation instead of 16-bit values. Four bits per weight would be about 7.35 GB before metadata and quantization overhead, so a 9.1 GB model layer is reasonable. Quantization is a whole phase later in this series (weeks 14 to 16). For now the only thing to take away is that Ollama runs lower-precision weights, and that is why this 14.7B model takes much less space than its 16-bit source model. The three stop entries (, , ) are special tokens: markers the model was trained to use as turn boundaries in a chat. The runtime watches for them to know when the model has finished its reply. They show up again in the next section, so keep them in mind. Concept 2: the API is just HTTP over a local process Ollama exposes a local HTTP API on port 11434,3 the same shape of thing as any cloud AI API. The difference is that the "server" is a process on my own machine. I called the generate endpoint with curl and formatted the reply with jq. Run it directly on the Spark: curl -s http://localhost:11434/api/generate -d '{ "model": "phi4", "prompt": "Explain Kubernetes scheduling in three sentences.", "stream": false }' | jq Enter fullscreen mode Exit fullscreen mode Here is the complete response, nothing removed. It is long, but seeing all of it is the point, because two parts of it explain how the model actually works: { "model": "phi4", "created_at": "2026-07-14T03:11:50.862545331Z", "response": "Kubernetes scheduling is the process of assigning workloads, such as pods, to appropriate nodes within a cluster based on resource availability and constraints. The scheduler evaluates factors like CPU, memory requirements, affinity/anti-affinity rules, taints/tolerations, and node selectors to determine the best fit for each pod. This ensures optimal utilization of resources, workload distribution, and adherence to specified policies, thereby maintaining efficient cluster operation and performance.", "done": true, "done_reason": "stop", "context": [ 100264, 882, 100266, 198, 849, 21435, 67474, 38952, 304, 2380, 23719, 13, 100265, 198, 100264, 78191, 100266, 198, 42, 30927, 38952, 374, 279, 1920, 315, 61853, 990, 33785, 11, 1778, 439, 55687, 11, 311, 8475, 7954, 2949, 264, 10879, 3196, 389, 5211, 18539, 323, 17413, 13, 578, 29909, 67349, 9547, 1093, 14266, 11, 5044, 8670, 11, 51552, 14, 15719, 71260, 13797, 5718, 11, 259, 1673, 82, 5640, 22847, 811, 11, 323, 2494, 57137, 311, 8417, 279, 1888, 5052, 369, 1855, 7661, 13, 1115, 26420, 23669, 50549, 315, 5070, 11, 54696, 8141, 11, 323, 71628, 311, 5300, 10396, 11, 28592, 20958, 11297, 10879, 5784, 323, 5178, 13 ], "total_duration": 3974942694, "load_duration": 100037676, "prompt_eval_count": 18, "prompt_eval_duration": 46132000, "eval_count": 89, "eval_duration": 3817895000 } Enter fullscreen mode Exit fullscreen mode Reading it top to bottom: The response field is the answer, a real reply from a model running on my hardware with no network call leaving the box. The context field is the part worth staring at. You might be wondering what those numbers are. They are tokens. A model does not read text; it reads token IDs, which are integers. Before anything runs, a tokenizer splits the text into tokens and maps each one to an integer. And this array is not just the prompt, it is the whole conversation as tokens: the chat template, my question, and the model's full answer. Decoding the first several so it is not a mystery: 100264 is the special marker (start of a turn), and 882 is the word "user". So the conversation begins "start of turn, user". 849, 21435, 67474, 38952 are "Ex", "plain", " Kubernetes", " scheduling". Notice "Explain" is split into two tokens. Tokens are often sub-word pieces, not whole words. 304, 2380, 23719, 13 are " in", " three", " sentences", ".", finishing my prompt. 100265 is (end of turn), then 100264 78191 100266 is the start of the assistant's turn ("start, assistant, separator"). 42, 30927 onward ("K", "ubernetes", ...) is the model's answer, token by token, which is exactly the text in the response field above. The special IDs 100264, 100265, 100266 are the chat markers (, , ), the turn boundaries from the model's embedded tokenizer and chat template. Two questions probably popped into your head reading that, the same ones that popped into mine: wait, how do I even know 849 is "Ex" or that 100264 is ? And how does Ollama know which tokenizer to use? Both have clear answers, but they would break the flow here, so I answer them at the end. If you want them now, jump to the token questions, answered. This is why token counts matter everywhere. On a cloud API you pay per token. In a model, tokens are what fill the context window, and the whole conversation is carried forward as this growing list of integers. When people say "tokens per second," this array is the unit being counted. Concept 3: inference has two measured phases The bottom of that same output has the timing, and it turns vague words like "latency" into measured numbers. All the durations are in nanoseconds, so here they are converted: Field Raw value Converted value load_duration 100,037,676 ns 0.10 s prompt_eval_count 18 18 input tokens prompt_eval_duration 46,132,000 ns 46 ms eval_count 89 89 output tokens eval_duration 3,817,895,000 ns 3.82 s total_duration 3,974,942,694 ns 3.97 s prompt_eval_duration is the time spent reading the tokenized input. That is the prefill phase. eval_duration is the time spent generating output tokens. That is the decode phase. load_duration is time Ollama spent loading or preparing the model for this request. One headline number can be calculated directly: Tokens per second (generation speed): 89 tokens divided by 3.82 s is about 23.3 tokens/sec. I cannot honestly calculate time to first token (TTFT) from this non-streaming response. Adding load and prefill gives 146 ms of server work before decode, but that is not the same as observing when the first token reaches the client. To measure TTFT, the client must request a streamed response and timestamp the first non-empty token. I do that in the benchmark later in this post. The timing split also shows that generating text has two phases that behave differently: Prefill (prompt_eval): the model reads all 18 prompt tokens at once. This is fast (46 ms) because the tokens are processed together, which is the parallel, compute-heavy work the GPU from Week 1 is good at. Decode (eval): the model generates output tokens one at a time, each one depending on the previous. This is the slow, sequential phase, and it is where memory bandwidth matters. Each token passes through the model layers and uses their weights again, while the key-value cache avoids recalculating all prior tokens from scratch. That is the core of Week 2. The API is HTTP over a local process, the "text" is really a stream of integer tokens, and a request is load, then prefill, then decode, each one measurable. Now I can start changing the settings that shape the output. Concept 4: temperature, or how random the model is allowed to be Back in Concept 2 I ran the same prompt twice and got slightly different wording each time. That is not a bug. It is a setting called temperature, and it controls how random the model is allowed to be when it picks each next token. The setting is options.temperature. I sent the same coffee-shop prompt twice at each temperature to watch it change. I changed the value in this direct call and ran it twice: curl -s http://localhost:11434/api/generate -d '{ "model": "phi4", "prompt": "Give me a one-sentence tagline for a coffee shop.", "stream": false, "options": { "temperature": 0 } }' | jq -r .response Enter fullscreen mode Exit fullscreen mode Twice at temperature 0: temperature=0 run 1: "Awaken Your Senses, One Cup at a Time." run 2: "Awaken Your Senses, One Cup at a Time." Enter fullscreen mode Exit fullscreen mode These two runs matched. The same call with "temperature": 1.2 gives variety: temperature=1.2 run 1: "Where every sip is a moment of delight." run 2: "Where Every Cup is a Perfect Brew—Awaken Your Senses." Enter fullscreen mode Exit fullscreen mode Same input, two different answers. So what is temperature actually doing? At each step, the model does not just pick one next token. It produces a probability for every token in its vocabulary, like "there is a 40% chance the next token is Awaken, 8% chance it is Where, and so on". Temperature reshapes that list before one token is picked. Top-p and seed also affect the choice, but each control has a different job: Control Changes Lower/same Higher/different Temperature Probability shape Favors likely tokens Allows more variety Top-p Candidate set Fewer likely tokens More possible tokens Seed Random sequence Same starting sequence Another sequence Top-p is also called nucleus sampling. At 0.9, it keeps the most likely tokens whose probabilities add up to 90%, then samples only from that set. Temperature and top-p can be used together, but while learning I would change one at a time so I know which setting changed the output. You might also see a seed beside temperature. A model runtime uses a pseudorandom number generator when it samples from the possible next tokens. Pseudorandom means the values look random, but they are generated by a formula. The seed is the starting number for that formula, like starting with the same shuffle of a deck. The same seed gives the sampler the same sequence of random choices when the model, prompt, options, runtime, and execution conditions stay the same. I tested that with temperature 1.2, where sampling has room to vary: for RUN in 1 2 3 4; do printf "run=%s seed=42: " "$RUN" curl -s http://localhost:11434/api/generate -d '{ "model": "phi4", "prompt": "Reply with one invented coffee shop name and nothing else.", "stream": false, "options": {"temperature": 1.2, "seed": 42, "num_predict": 16} }' | jq -r .response | tr '\n' ' ' printf "\n" done Enter fullscreen mode Exit fullscreen mode run=1 seed=42: Espresso Enchantments run=2 seed=42: Espresso Enchantments run=3 seed=42: Espresso Enchantments run=4 seed=42: Espresso Enchantments Enter fullscreen mode Exit fullscreen mode These four warm-model runs matched. A fixed seed improves repeatability, but it is not a universal guarantee. A different runtime version, model build, hardware path, parallel execution order, or other nondeterministic GPU behavior can still change the result. A seed controls the sampler's random sequence; it does not freeze the whole software and hardware stack. The practical takeaway is simple. Start near zero when repeatability matters, and raise temperature when you want variety. Add a fixed seed when you want more repeatable comparisons, and record the runtime and model version too. Everything else this week uses the same options object in the API call. Concept 5: chat is a list of role-tagged messages So far I used /api/generate, which takes one plain prompt. Ollama also has chat interfaces. I will use them in increasing depth: The CLI proves I can have a one-shot conversation. The HTTP API exposes the role-tagged message structure. The Python client keeps that structure across multiple turns. First, chat through the CLI I keep using phi4, already pulled in Concept 1, so there is nothing new to download. Ollama's CLI is terminal-aware, so I forced SSH to allocate a terminal with -tt: ssh -tt spark \ 'ollama run --nowordwrap phi4 \ "Reply with exactly these three words: local model ready"' Enter fullscreen mode Exit fullscreen mode Local model ready. Connection to spark closed. Enter fullscreen mode Exit fullscreen mode This proves the CLI can send a prompt and display a reply. It does not show the message roles or how a client maintains a conversation. For that, I need the HTTP API. Next, inspect messages through the API The /api/chat endpoint takes a messages array. Each item has a role that says who wrote it and content that holds the text: Role Who writes it What it does system Application Sets behavior, style, or limits user Person using the app Carries the user's prompt or follow-up assistant Model Stores a previous model reply in the history The system prompt is an instruction the application places before the user conversation. It sets the expected role, style, or limits of the assistant. It is configuration, not a security boundary. A user can still send conflicting instructions, so production systems need checks outside the prompt too. The user prompt is simply the content of a message with role: "user". After the model replies, the client stores that answer as an assistant message. The next request sends the earlier messages plus the new user message. Why resend them? The Ollama API does not remember this client's conversation between independent HTTP requests. I sent two consecutive requests to the same running Ollama process. First, I included the earlier user question and assistant reply: curl -s http://localhost:11434/api/chat -d '{ "model": "phi4", "stream": false, "messages": [ {"role": "system", "content": "Reply with only a name or UNKNOWN."}, {"role": "user", "content": "My cluster is called Atlas."}, {"role": "assistant", "content": "Atlas"}, {"role": "user", "content": "What name did I give it?"} ], "options": {"temperature": 0, "seed": 42} }' | jq '{reply: .message, prompt_tokens: .prompt_eval_count}' Enter fullscreen mode Exit fullscreen mode { "reply": { "role": "assistant", "content": "Atlas" }, "prompt_tokens": 50 } Enter fullscreen mode Exit fullscreen mode Then I sent only the follow-up, with no earlier turns: curl -s http://localhost:11434/api/chat -d '{ "model": "phi4", "stream": false, "messages": [ {"role": "system", "content": "Reply with only a name or UNKNOWN."}, {"role": "user", "content": "What name did I give it?"} ], "options": {"temperature": 0, "seed": 42} }' | jq '{reply: .message, prompt_tokens: .prompt_eval_count}' Enter fullscreen mode Exit fullscreen mode { "reply": { "role": "assistant", "content": "UNKNOWN" }, "prompt_tokens": 31 } Enter fullscreen mode Exit fullscreen mode Request Reply Prompt tokens Earlier turns included Atlas 50 Earlier turns omitted UNKNOWN 31 What changed between the two calls? Only the messages array. In the first request, the JSON body included the earlier line My cluster is called Atlas, the model's earlier answer, and the new follow-up. Ollama could therefore pass all of that text to the model, so the answer was Atlas. In the second request, the conversation history contained only What name did I give it?. There was no mention of Atlas in its messages, so the model answered UNKNOWN. Ollama did not look up the earlier HTTP call and add it automatically. That is what the client owns the conversation state means. The application, such as this Python client, keeps the message list in memory or a database. For each new turn, it appends the latest user message and sends the relevant history again. Ollama processes the messages it receives in that request. Resending history has a cost. The request with earlier turns contained 50 prompt tokens; the request without them contained 31. The 19-token difference includes the added message text, role labels, and chat-template separators. I measured the total difference here; I did not split it token by token. All 19 additional tokens used positions in the context window. Finally, implement the conversation in Python The required Python client now implements the message flow I just tested. It uses /api/chat, keeps prior turns in the messages list, and streams the reply as it arrives. With "stream": true, Ollama sends one JSON object per line; each text fragment is in message.content. The loop prints each fragment and joins them into the completed assistant message. Here is the complete file: #!/usr/bin/env python3 """Small command-line chat client for Ollama's streaming /api/chat endpoint.""" import argparse import json import os import urllib.error import urllib.request def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--model", default=os.getenv("MODEL", "phi4")) parser.add_argument("--url", default="http://localhost:11434") parser.add_argument("--system", default="You are a concise assistant.") parser.add_argument("--temperature", type=float, default=0.0) parser.add_argument("--top-p", type=float, default=0.9) parser.add_argument("--num-ctx", type=int, default=4096) parser.add_argument("--once", help="Send one prompt and exit") return parser.parse_args() def request_chat( url: str, model: str, messages: list[dict[str, str]], options: dict[str, float | int], ) -> str: body = { "model": model, "messages": messages, "stream": True, "options": options, } request = urllib.request.Request( f"{url.rstrip('/')}/api/chat", data=json.dumps(body).encode(), headers={"Content-Type": "application/json"}, ) parts: list[str] = [] try: with urllib.request.urlopen(request) as response: for line in response: event = json.loads(line) text = event.get("message", {}).get("content", "") if text: print(text, end="", flush=True) parts.append(text) except urllib.error.URLError as exc: raise SystemExit(f"Ollama request failed: {exc}") from exc print() return "".join(parts) def main() -> None: args = parse_args() messages: list[dict[str, str]] = [] if args.system: messages.append({"role": "system", "content": args.system}) options = { "temperature": args.temperature, "top_p": args.top_p, "num_ctx": args.num_ctx, } if args.once: messages.append({"role": "user", "content": args.once}) request_chat(args.url, args.model, messages, options) return print(f"Chatting with {args.model}. Type /exit to quit.") while True: try: prompt = input("you> ").strip() except (EOFError, KeyboardInterrupt): print() return if prompt.lower() in {"/exit", "/quit"}: return if not prompt: continue messages.append({"role": "user", "content": prompt}) print("model> ", end="", flush=True) answer = request_chat(args.url, args.model, messages, options) messages.append({"role": "assistant", "content": answer}) if __name__ == "__main__": main() Enter fullscreen mode Exit fullscreen mode I used --once for a repeatable blog example and passed a system prompt. If I leave off --once, the same program opens an interactive loop and keeps each user and assistant turn in the next request. ssh spark '~/venvs/w1/bin/python - \ --model phi4 \ --system "Answer in exactly one short sentence." \ --once "What does the Kubernetes scheduler do?"' \ argparse.Namespace: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--model", default="phi4") parser.add_argument( "--prompt", default="Explain Kubernetes scheduling in three sentences.", ) parser.add_argument("--contexts", default="2048,4096") parser.add_argument("--runs", type=int, default=3) parser.add_argument("--url", default="http://localhost:11434") parser.add_argument("--output", type=Path) return parser.parse_args() def unload(model: str) -> None: subprocess.run( ["ollama", "stop", model], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False, ) time.sleep(1) def run_once( url: str, model: str, prompt: str, num_ctx: int, cold: bool, ) -> dict[str, Any]: if cold: unload(model) body = { "model": model, "prompt": prompt, "stream": True, "options": { "temperature": 0, "seed": 42, "num_ctx": num_ctx, "num_predict": 96, }, } request = urllib.request.Request( f"{url.rstrip('/')}/api/generate", data=json.dumps(body).encode(), headers={"Content-Type": "application/json"}, ) started = time.perf_counter() first_token_at: float | None = None final: dict[str, Any] = {} with urllib.request.urlopen(request) as response: for line in response: event = json.loads(line) if event.get("response") and first_token_at is None: first_token_at = time.perf_counter() if event.get("done"): final = event finished = time.perf_counter() if first_token_at is None or not final: raise RuntimeError("Ollama stream ended without timing data") eval_seconds = final["eval_duration"] / 1e9 return { "context": num_ctx, "cold": cold, "ttft_ms": round((first_token_at - started) * 1000, 1), "client_total_ms": round((finished - started) * 1000, 1), "load_ms": round(final["load_duration"] / 1e6, 1), "prompt_eval_ms": round(final["prompt_eval_duration"] / 1e6, 1), "prompt_tokens": final["prompt_eval_count"], "output_tokens": final["eval_count"], "tokens_per_second": round(final["eval_count"] / eval_seconds, 1), } def main() -> None: args = parse_args() contexts = [int(value) for value in args.contexts.split(",")] rows: list[dict[str, Any]] = [] print("ctx run cold ttft_ms total_ms load_ms out_tok tok/s") for context in contexts: for run_number in range(1, args.runs + 1): row = run_once( args.url, args.model, args.prompt, context, cold=run_number == 1, ) rows.append(row) print( f"{context:>4} {run_number:>3} {str(row['cold']):>5} " f"{row['ttft_ms']:>7.1f} {row['client_total_ms']:>8.1f} " f"{row['load_ms']:>7.1f} {row['output_tokens']:>7} " f"{row['tokens_per_second']:>5.1f}" ) result = { "model": args.model, "prompt": args.prompt, "runs_per_context": args.runs, "measurements": rows, } if args.output: args.output.write_text(json.dumps(result, indent=2) + "\n") if __name__ == "__main__": main() Enter fullscreen mode Exit fullscreen mode I ran three requests per context size. The first is cold because the script stops Phi-4 before it. The next two reuse the loaded model. ssh spark '~/venvs/w1/bin/python - --contexts 2048,4096 --runs 3' \ None: enc = tiktoken.get_encoding("cl100k_base") print("encode:", enc.encode(PROMPT)) print("decode individual IDs:") for t in SAMPLE_IDS: print(f" {t:>6} {enc.decode([t])!r}") if __name__ == "__main__": main() Enter fullscreen mode Exit fullscreen mode Then I ran the file with the pinned tiktoken 0.13.0 package: ssh spark '~/venvs/w1/bin/python -' \ argparse.Namespace: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--model", default=os.getenv("MODEL", "phi4")) parser.add_argument( "--models-dir", type=Path, default=Path("/usr/share/ollama/.ollama/models"), ) return parser.parse_args() def field(reader: GGUFReader, name: str): value = reader.get_field(name) if value is None: raise KeyError(f"Missing GGUF field: {name}") return value.contents() def main() -> None: args = parse_args() name, separator, tag = args.model.partition(":") if not separator: tag = "latest" manifest = ( args.models_dir / "manifests" / "registry.ollama.ai" / "library" / name / tag ) data = json.loads(manifest.read_text()) digest = next( layer["digest"] for layer in data["layers"] if layer["mediaType"].endswith("image.model") ) blob = args.models_dir / "blobs" / digest.replace(":", "-") reader = GGUFReader(blob) tokens = field(reader, "tokenizer.ggml.tokens") template = field(reader, "tokenizer.chat_template") print("tokenizer model:", field(reader, "tokenizer.ggml.model")) print("tokenizer pre:", field(reader, "tokenizer.ggml.pre")) print("token count:", len(tokens)) for token_id in (100257, 100264, 100265, 100266): print(f"token {token_id}: {tokens[token_id]!r}") for marker in ("", "", ""): print(f"template contains {marker}: {marker in template}") if __name__ == "__main__": main() Enter fullscreen mode Exit fullscreen mode I ran it with the pinned gguf 0.19.0 package: ssh spark '~/venvs/w1/bin/python -' \ ' token 100264: '' token 100265: '' token 100266: '' template contains : True template contains : True template contains : True Enter fullscreen mode Exit fullscreen mode Everything the runtime needs is in the file: tokenizer.ggml.model = 'gpt2' is the tokenizer type. It is a byte-pair encoding (BPE), the same family GPT-2 and GPT-4 use. tokenizer.ggml.tokens is the full vocabulary, all 100,352 entries, the list that maps text pieces to IDs. The three IDs are the exact special-token strings claimed earlier. The embedded chat template contains those same markers. Phi-4's model card uses this ChatML-style prompt layout.6 So Ollama reads the vocabulary from the model file itself and uses it to turn my prompt into token IDs. The ordinary IDs used in this prompt are compatible with cl100k_base, while the GGUF adds Phi-4's special chat tokens. A Llama model's GGUF carries a different vocabulary, and Ollama uses that instead. Same runtime, different tokenizer per model, because the tokenizer travels with the model. Results Here is the Week 2 result in one place: Item Measured result Runtime Ollama 0.31.2 Main model Phi-4, 14.7B parameters, Q4_K_M Model layer 9,053,114,464 bytes Context settings tested 2,048 and 4,096 tokens Cold TTFT 6,034.1 to 6,260.8 ms Warm TTFT 151.9 to 196.1 ms Generation speed 23.4 to 23.5 tokens/sec Python deliverable Streaming command-line chat client Benchmark deliverable Cold/warm TTFT and throughput benchmark I also used every interface in the plan: interactive CLI, curl, and Python. The commands changed temperature and context size. The Python client used a system prompt, and the benchmark compared repeatable runs with measured timing. What surprised me The first surprise was the model size. Phi-4 has 14.7 billion parameters, but its quantized model layer is 9.1 GB instead of the roughly 29.4 GB needed for 16-bit weights alone. The bigger surprise was the cold start. Generation speed stayed near 23.5 tokens/sec, but the first token took about 6 seconds when Ollama had to load the model. Once warm, it arrived in about 0.15 to 0.20 seconds. Changing context capacity from 2,048 to 4,096 did not improve this short request. A larger limit gives the KV cache room for more tokens. It does not make an 18-token prompt larger or automatically faster. Mistakes and troubleshooting I first called the output of ollama show a manifest. That was wrong. Model metadata is a readable summary. The manifest is the JSON index that points to the model, template, license, and parameter layers. I also tried to estimate TTFT by adding load_duration and prompt_eval_duration from a non-streaming response. That is useful server timing, but it is not client-observed TTFT. The fixed benchmark streams the response and timestamps the first non-empty text chunk. One smaller issue came from Ollama's terminal-aware CLI. Its one-shot response was not visible through plain SSH capture, so I used ssh -tt to allocate a pseudo-terminal. The Python and curl clients did not need that workaround. Production implications A production service should keep commonly used models warm. Otherwise, a user can wait seconds for model loading before generation even starts. Streaming also matters because it lets the user see the first token instead of waiting for the whole reply. Context size is a capacity and memory decision. Setting it to the model maximum for every request can reserve more KV-cache memory than the workload needs. The right setting comes from measured prompt and conversation lengths. The system prompt is not an access-control system. If I expose Ollama beyond localhost, I still need a gateway with authentication, authorization, TLS, rate limits, request-size limits, and logging. Prompt instructions do not replace those controls. What I will learn next Week 3 moves from running a packaged model to opening a Hugging Face repository and inspecting its files. I will compare base and instruct models, read the model card and license, then inspect configuration, tokenizer, chat-template, and Safetensors files.7 Run it yourself The public Week 2 lab contains the chat client, benchmark, scripts, pinned requirements, raw results, and model-run record.8 Can llama.cpp reuse this Ollama model? This is optional reading. It is useful if you want to understand whether a model file belongs to one runtime or can move between compatible runtimes. For this Phi-4 download, yes. Ollama reports details.format = "gguf", and the independent file-header check agrees. llama.cpp loads local GGUF files, so it can use this same 9,053,114,464-byte model blob without downloading the weights again.5 The content matters, not the missing .gguf extension on Ollama's content-addressed filename. A file header alone does not prove another runtime can load the model and generate text, so I installed a pinned CUDA build of llama.cpp and pointed it at the exact model blob named by Ollama's manifest digest. It loaded the file and generated a reply: Check Result Model blob same file, Ollama manifest digest fd7b...df20 llama.cpp build pinned commit 571d0d54 Device used CUDA0 (NVIDIA GB10) GPU layers offloaded 41/41 Generated reply Shared model works. That is a concrete yes. The full investigation records the failed first check, CUDA toolkit discovery, pinned ARM64 build, scripts, commands, raw output, and the apparent terminal hang.2 One limit remains. The model blob is only one layer of the Ollama package. The manifest also lists a prompt template, license, and default parameters as separate layers. llama.cpp does not read those Ollama layers just because it can read the model blob. This Phi-4 GGUF carries its own tokenizer and chat template, but I still set the seed, temperature, context, and GPU layers explicitly. Same weights do not guarantee the same output from two runtimes. This result is specific to this Phi-4 file and this pinned llama.cpp commit. For another Ollama download, I would repeat the test because an older runtime may not support a newer model architecture. Official project docs for the alternatives named here: llama.cpp at https://github.com/ggml-org/llama.cpp, vLLM at https://docs.vllm.ai, NVIDIA TensorRT-LLM at https://nvidia.github.io/TensorRT-LLM/, and Hugging Face Transformers at https://huggingface.co/docs/transformers/index ↩ Full Ollama-to-llama.cpp investigation, including the pinned CUDA build and raw command output: https://github.com/dramasamy/from-api-to-gpu/blob/main/week-02-first-local-model/ollama-model-in-llamacpp.md ↩ Ollama API reference: https://docs.ollama.com/api/introduction ↩ tiktoken, OpenAI's tokenizer library, lists the standard vocabularies and which model each one belongs to. See the model-to-encoding table at https://github.com/openai/tiktoken ↩ GGUF is the model file format from the llama.cpp project. It stores the weights plus metadata such as the tokenizer and chat template. See https://github.com/ggml-org/ggml/blob/master/docs/gguf.md ↩ Microsoft Phi-4 model card, including its chat input format: https://huggingface.co/microsoft/phi-4#input-formats ↩ Week 3 roadmap: https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-03.md ↩ Week 2 companion lab: https://github.com/dramasamy/from-api-to-gpu/tree/main/week-02-first-local-model ↩