SealedBench: A Protocol for Freezing AI Benchmarks Before the Questions Are Known

A frozen model manifest, a generator sealed with drand, and an Arweave record anyone can use to rebuild the exam and check its scores.I knew the question types but not the actual questions On 2026-10-17, a bundle carrying my encrypted generator reached an Arweave block. The generator was sealed to a drand round that had not been published yet. I wrote that program, so I knew it would produce integer word problems, short Python programs, and shortest-path questions. I did not know which ones. The questions depended on a beacon the drand network would publish later, at round 33170212, on 2026-10-18T09:00:00Z.When the round arrived, the same beacon value unlocked the program and seeded it. Four open-weight models answered 360 completions each: 240 on newly generated questions and 120 on a frozen set of public GSM8K items. But the first two batches I ran should not be cited, and it is worth saying why, because the reason is the most useful thing in this article. The first batch, b1, was valid under its own release record. Its manifest bound the ciphertext with a SHA-256 hash, but it never bound the ciphertext's transaction ID. Calling that check a pass would be a mistake, and it is WEAK. The v2 protocol exists partly to close that gap. The second batch, b2, fixed the binding and then failed on the grader. The rule I sealed said an integer answer counts only when the last line is #### followed by nothing but the number. llama.cpp writes its end-of-text marker onto that same line, as #### 392 [end of text]. The registered grader threw the whole line away. Most of b2's arithmetic and GSM8K scores measure that rejection, not the rate at which models produced correct integers. The ciphertext was already sealed by then, and the grader could not be changed in place. That constraint is the entire point of the protocol, and it applies to me too. So I made b3: a new commitment, sealed before its own round, with the normalization written into the scorer the manifest hashes. So if you want a number from this project, take one from b3, and take the registered primary accuracy, which counts an answer as wrong when it cannot be parsed and wrong when the output was truncated. Its index transaction is vlVlUfyBn4IUmbXjkSX8079QuySP-Qqr8rN8TbivzTg. That index was published after the run. It is a convenient entry point, not evidence: the evidence that the generator existed before the round is the earlier ciphertext and manifest. What a reader needs in order to catch me Public benchmarks leak. Once the questions are online, a high score may only mean the model has seen them. A private test set looks like the fix, but it hands the same power to one person: whoever writes, stores, selects and reports the questions. That person includes the evaluator, which is me. If I can see the scores and then quietly change the seed, the prompt, the model file, or simply decide which batch to tell you about, I can aim the headline wherever I want. SealedBench tries to pin all of that down in advance. The generator, the model hashes, the prompts, the runtime settings and the scoring code are bound to one future beacon round. Unless drand's operators reach a threshold and collude to compute that round early, I cannot read the beacon before it is published. If the network stops, the batch never opens; that is the other half of the bargain. After the round, anyone can rebuild the questions from the Arweave record and re-score the answers I published. Here is every move I could make, and what I gave up in advance to make it visible. What I could change after seeing scores What was committed in advance What you still have to check The generator or the recipe Ciphertext and manifest on Arweave before the round Check L1 block time, not the upload receipt The seed The beacon round fixed in the manifest That the generator really uses that seed Questions, answers or the scorer, later Content hashes A hash cannot show which process wrote a file Reporting only the flattering batch Batches declared up front Encryption cannot force me to publish answers Freeze, seal, then wait for a block The manifest freezes the drand chain hash and public key, the release round, the seed rule, a SHA-256 for every GGUF, the llama.cpp commit and build flags, the prompts, the decoding settings, the grader and the reporting policy. Every batch uses quicknet chain 52db9ba70e0cc0f6eaf7803dd07447a1f5477735fd3f661792ba94600c84e971, genesis at unix 1692803367, period 3 seconds. The generator is sealed to that round with drand time-lock encryption (tlock). The ciphertext and the manifest go up through the Turbo bundler. A receipt is not a commitment. The commitment is the L1 block. Under the v2 rule a batch is valid only if the block holding its bundle is timestamped no later than unlock_unix − 600, the bundle ID appears in that block, and the data item's bytes hash to the committed SHA-256. Height, timestamp and block hash must agree on two independent gateways, and my own gateway ario.zigza.xyz does not count as either one. b1 used an earlier, simpler rule: manifest and ciphertext had to be in an L1 block before the release round, and a miss made the batch invalid, which I would still have had to report. After the round, the beacon signature is checked against quicknet's fixed public key. The same beacon both decrypts the generator and enters the seed. One value doing two jobs is not two independent draws, and I do not claim otherwise. b1's seed is a SHA-256 over a domain string, the chain hash, the round in decimal and the beacon randomness, with hex fields lowercased first. The exact sentence is in b1's manifest. v2 batches add a 32-byte secret stored inside the ciphertext, and hash length-prefixed fields: domain, chain hash, round, signature bytes and that secret. The JSON randomness field is not an input; it must still equal the SHA-256 of the signature. I chose the secret. It does not let me list the questions before the beacon exists. A successful tle -d only shows the ciphertext opens under the signature it was sealed to. The BLS check against the fixed public key is a separate step. There is one specific gap in v1 worth naming. The November capsule cap and b1 share the same plaintext tarball, SHA-256 e054f1ab7824fde33e6ac399252098cf25c29815ff4ffacabbd311da05eac9c2. Once b1 opened, that program was public. Until 2026-11-15 the unknown is the beacon, not the program. v2 still hides its questions, because the secret sits inside the ciphertext. Four small models, one CPU, three question families Every family has a program for an oracle, so grading is code and not a judge model. arith_word produces two-to-five-step integer word problems; the oracle is a function of the sampling parameters, and the prompt contains no answer line. code_output produces short Python programs with no imports and no input, run in an interpreter with a step budget; the oracle is their stdout. graph_path asks for the weight of a shortest path on a random connected graph with edge weights from 1 to 9, always between nodes at least two edges apart; the oracle is Dijkstra. The entropy check ran the generator over 1000 seeds and got 1000 distinct question sets, with 128,816 distinct prompts. That shows the seed changes the questions. It does not show the families are fair, or that they look unlike anything in training data. One caveat I want on the record: that check ran with a stand-in secret, not the sealed one, so it measures the generator's spread and not the exact bytes b2 and b3 committed. All four models are official Q8_0 GGUFs, hashed locally against the published LFS object IDs before sealing: Model Params GGUF SHA-256 (prefix) GSM8K-test first 50 (Phase 1) Gen tok/s Qwen3-4B-Instruct-2507 4B ae916ede1c01… 46/50 6.12 Gemma-3-4B-IT 4B 97b06383df48… 41/50 6.42 Ministral-3-3B-Instruct-2512 3B 70c6e5b77435… 39/50 7.00 Falcon3-1B-Instruct 1B dda447091571… 19/50 15.95 Falcon3-1B is weak on purpose. It is a low anchor that makes a difficulty shift easier to see. The runtime is llama.cpp v0.7.0, commit 620d2225cc436ab0b001e3200e590e3b286ef9ab, built CPU-only with AVX-512 on a Ryzen 7 7700 in a cgroup of about six CPUs and 18 GiB. Decoding is greedy: temperature 0, top-k 1, top-p 1, min-p 0, seed 42, -t 6 -b 2048 -ub 512. Thread count and batch size are in the manifest because they change the output bytes. Repeated runs at -t 6 were byte-identical for all four models. Dropping to -t 4 changed 2 of 10 Qwen outputs. Changing to -b 512 -ub 128 changed Qwen, Gemma and Falcon. Ministral matched in all three comparisons. After release I replayed Qwen's first ten exam items: 10/10 byte-identical on b1, 10/10 on b2, 10/10 on b3. That is one model on one machine. Generating and grading are ordinary deterministic programs. I am not claiming your GPU would produce the same bytes. b1 was valid, b2 graded the end marker, b3 is the correction Batch Protocol Round Unlock (UTC) Items b1 sealedbench/v1 33170212 2026-10-18T09:00:00Z 120 arith, 60 code, 60 graph, plus GSM8K-test 0..119 b2 sealedbench/v2 33188212 2026-10-19T00:00:00Z 80 arith, 30 code, 30 graph, plus GSM8K-test 0..79 b3 sealedbench/v2 33202612 2026-10-19T12:00:00Z 80 arith, 30 code, 30 graph, plus GSM8K-test 0..79 Question-set SHA-256: b1 a91b6dab74eee35c8aaba81f8323610a5604fc7d73529bc7229aa3c6e1668193, b2 473acec24390a708a3323d51fe8a1cd2265591bcf8a4039f4564f49ab1833bdb, b3 549ba12a3062df5ce40393408d907544bbb9a2b75339b9ef6183015485fc9df3. b1's release.json records batch_valid true. Its manifest 510sRNPxwIwK4D_xXkujpyih8M1pcww9DcZgOaLkkzU and ciphertext 57E6ydKUep6BKQ6L8wUSgrsDYkvq5r8pNmlUXzpxd7o entered block 2023266 at 2026-10-17T15:21:00Z, well before the unlock. Its index is AtNAZQtp_JqReahCoCpYcjP-b3P98jWtC_AJgVoBWdM. The v1 grader is not the v2 rule; it keeps a "take the last number" fallback. Arithmetic minus GSM8K, Wald 95%: Qwen 0.0083 [−0.0765, 0.0932], Gemma 0.1333 [0.0593, 0.2073], Ministral 0.1250 [0.0493, 0.2007], Falcon 0.4917 [0.3895, 0.5939]. Those two sets are not matched pairs. b2's registered grader is sealedbench/v2/grade.py, SHA-256 98b796af643192968126a5895561ec1d4701c71ccf9af76ae874fe1f0b3b0648. The integer rule is ^####[ \t]+(-?\d[\d,]*)[ \t]*$. Code questions could still parse, because fenced-output extraction does not require the closing fence to own its line. On that registered endpoint Qwen scored 0/80 on arithmetic, with all 80 invalid, and 3/80 on GSM8K, with 76 invalid. Gemma and Ministral scored 0/80 on both integer families. Falcon scored 1/80 on arithmetic, 79 invalid. The raw files often already contained the right number; [end of text] was stuck to it. The same scores.json carries a block called posthoc_sentinel_normalization, labelled POST-HOC with registered_primary false. It keeps the truncation rule on raw stdout and applies b3's normalization before the integer and code rules. There Qwen scores 73/80 on arithmetic and 77/80 on GSM8K. That is a reconstruction, not the score b2 promised to report. b3 pins sealedbench/v2/grade.py and sealedbench/v2/grade_b3.py. The grader SHA-256 eab22119c837f790f900989ce7d3c8700635ad7881204ffc90079d800854b290 is a hash over both files sorted by path, not the hash of grade_b3.py alone. The registered normalization strips trailing ASCII whitespace, removes one trailing [end of text] if present, then strips again. Truncation is decided before that, on raw stdout. b3 has no post-hoc block. Its manifest is _aZZOYmKQr9awEhsmKew9J4sAHyDnfZoo1Pq1NYXOVk and its ciphertext MU3l3e_pbboAuVH8dtiLJP4XMGKGaFR530w-TauzteM. v2 release files leave batch_valid unset. Their inclusion records put every sealed item's L1 time before that batch's own deadline. b2 and cap2 share bundle VWHJrEUcS-d9EXMzzJr9k6Qr2WKQLR9iwiMfm4nglgo, height 2023391, 2026-10-17T20:05:59Z. b3 and cap3 share J67Udct-eWBqCnNfoT9c_d0baY48io8tHlJ8uD_ZVys, height 2023739, 2026-10-18T08:11:02Z. The registered b3 scores Primary accuracy, with invalid and truncated answers counted as wrong: Model Arithmetic n=80 GSM8K n=80 Code n=30 Graph n=30 Qwen3-4B 71 correct, 0 invalid 77 correct, 0 invalid 20 correct, 5 invalid, 4 truncated 9 correct, 21 invalid, 21 truncated Gemma-3-4B 20 correct, 60 invalid 26 correct, 50 invalid, 2 truncated 17 correct, 8 invalid, 8 truncated 22 correct, 1 invalid, 1 truncated Ministral-3-3B 74 correct, 0 invalid 65 correct, 4 invalid, 1 truncated 7 correct, 0 invalid 20 correct, 2 invalid, 2 truncated Falcon3-1B 47 correct, 20 invalid 21 correct, 30 invalid, 1 truncated 2 correct, 0 invalid 1 correct, 13 invalid, 2 truncated The Wilson 95% intervals behind the two columns in the figure, rounded to three decimals from the wilson95 arrays: Qwen arithmetic [0.800, 0.940] and GSM8K [0.895, 0.987]; Gemma [0.168, 0.355] and [0.232, 0.434]; Ministral [0.846, 0.965] and [0.713, 0.883]; Falcon [0.478, 0.689] and [0.179, 0.368]. Arithmetic minus GSM8K, Newcombe 95%: Qwen −0.075 [−0.166, 0.010], Gemma −0.075 [−0.211, 0.065], Ministral 0.1125 [0.007, 0.219], Falcon 0.325 [0.173, 0.457]. At n=80 near one half, a Wilson interval is roughly 11 points wide. A gap inside that width is the difference between two unmatched sets of questions. The fresh and public items differ in wording, length and format. Calling that gap contamination would need a control this run does not have. One overlap should be stated plainly: Phase 1 had already run the models on GSM8K-test items 0..49, and b3's reference set covers 0..79, so the first 50 reference items were familiar ground before b3 was sealed. Nothing in this run separates that familiarity from the ordinary difficulty difference between two unmatched question sets. Two cells deserve more than a headline. Qwen's graph score is 9/30, and 21 of those 30 outputs were truncated; across that family it produced 15 distinct answers, against a uniform-guess baseline of 0.067. Gemma's arithmetic is 20/80 with 60 invalid, and all 20 that parsed were correct; it produced 67 distinct answers, against a uniform-guess baseline of 0.015. Figure 2 — b3 registered primary, fresh arith_word against GSM8K-test 0..79, with Wilson 95% intervals and n=80. article/fig/fig2_fresh_vs_public.png.One complete example, taken from the first line of results/batches/b3/items.jsonl rather than picked for its score. Item arith_word-0000, template buy_give_price, oracle 392: Gina buys 7 packs of balloons. Each pack has 9 balloons. Gina then gives 7 balloons to Kira. Each balloon is worth 7 dollars. How much are Gina's balloons worth in dollars? Qwen's stdout is 294 bytes, 94 generated tokens, return code 0, not truncated, SHA-256 e2f21c8f5596848bc8b9e7c4991dcc72c7ccf79a3c77bc035538bb8dc1878b24. It ends with #### 392 [end of text]. b3 strips the marker and parses 392, detail match. The same bytes are invalid under b2's registered grader, detail no #### line. The file is results/batches/b3/runs/qwen3-4b-instruct-2507/arith_word-0000.txt. "On Arweave" means a block, and blocks arrive late I measured four Turbo bundles. The 20 probes from the first day are one bundle, not 20 latency samples. Latency here is the L1 block timestamp minus the earliest receipt time in that bundle. Bundle Contents Earliest receipt L1 time Seconds ppP0MDrvyDX0uCeDTVww7FC4iGbkcbZiIazuQrcSIWE 20 probes 2026-10-17T09:12:54Z 2026-10-17T09:21:32Z 518 pzzVUc5WUQHkZMWHu3D12AseYrj6ZSwH1nx9lKurvxk b1 and cap 2026-10-17T15:05:36Z 2026-10-17T15:21:00Z 924 VWHJrEUcS-d9EXMzzJr9k6Qr2WKQLR9iwiMfm4nglgo b2 and cap2 2026-10-17T20:02:46Z 2026-10-17T20:05:59Z 193 J67Udct-eWBqCnNfoT9c_d0baY48io8tHlJ8uD_ZVys b3 and cap3 2026-10-18T08:04:26Z 2026-10-18T08:11:02Z 396 Sorted: 193, 396, 518, 924. With linear interpolation at index (n-1)*p, p50 is 457 seconds and p95 is 863 seconds; the maximum is 924. Nothing in this sample was still pending. Later results and index uploads are not in it, and their block times were not recorded the same way. The probes are 800 bytes each, sealed with tle v1.2.0 to rounds between 1176 and 1403 seconds after upload. Both the Go tool and tlock-js 0.9.0 refused to decrypt early, in 40 of 40 attempts. All 20 Turbo receipts verified. All 60 RFC 3161 timestamps from FreeTSA, DigiCert and Sectigo verified with openssl ts -verify. After the unlock, 20 of 20 probes decrypted in both tools and matched their plaintext hashes. FreeTSA's genTime landed a median 76.2 seconds after the Turbo receipt. A GraphQL poller pointed at arweave.net ran for three hours and got HTTP 429 every time, observing nothing. Serving bytes, being in a block and being indexed are three different states. The commitment is the block. When arweave.net returns 404, the honest output is UNKNOWN. Figure 3 — receipt-to-inclusion ECDF, one point per bundle, n=4. article/fig/fig3_inclusion_ecdf.png.Rebuilding b3 from the public repository The repository is https://github.com/zigengZ/sealedbench, released as v1.0.0. You need Python 3.12 and Node.js 22. Leave SEALEDBENCH_ROOT unset. git clone https://github.com/zigengZ/sealedbench.git cd sealedbench python3 -m venv .venv . .venv/bin/activate npm ci python3 -m sealedbench verify --batch b3 --txid vlVlUfyBn4IUmbXjkSX8079QuySP-Qqr8rN8TbivzTg Exit code 0 is PASS, 1 is a hard failure, and 2 is UNKNOWN, INVALID or WEAK. HTTP 404, 429 and timeouts all count as UNKNOWN. The command only reads gateways; it never uploads. It checks the published bytes, the inclusion deadline, the beacon signature and the grader hash, re-scores the published outputs, then decrypts the local ciphertext and rebuilds the question files. v2 requires https://ar-io.dev and https://turbo-gateway.com to agree. https://arweave.net takes no part in this check. On 2026-10-24 a fresh clone of the release candidate printed RESULT PASS, exit code 0, in 26.15 seconds. The registered file results/batches/b3/verify_seal.txt still ends RESULT UNKNOWN, because ar-io.dev returned HTTP 404 for the manifest that day. The later file verify_recheck_20261024.txt ends PASS. An earlier attempt the same day was UNKNOWN. I did not overwrite the registered file with the later result. b2's registered verify_seal.txt is UNKNOWN as well: the published scores and Ministral's completions returned 404. Its dated recheck is PASS. b1 was VALID in its 2026-10-18 release record, and the ciphertext-transaction-ID check is still WEAK. A b1 recheck on 2026-10-24 was never completed. From 2026-10-24T09:19:57Z every request this host made to arweave.net came back HTTP 429, including three attempts at b1's index. I stopped and did not restart it. Replaying the model is the heavier second path. It needs the GGUF and the frozen llama-completion binary: python3 -m sealedbench replay --batch b3 --model qwen3-4b-instruct-2507 --items 5 The registered replay directory is stamped 20261019T181448Z: 10 items compared, 10 byte-identical. The 2026-10-24 run compared 5 and matched 5, from arith_word-0000 to arith_word-0004. The weights were already on disk (qwen3-4b-instruct-2507-q8_0.gguf, 4,280,403,520 bytes). /usr/bin/time -v reported 1:05.87 wall clock, 337.29 seconds of user time, and a maximum resident set of 4,553,836 kB. During the process, the host's non-loopback interface counters rose by 185,159,022 bytes. That figure is for the whole host, not this command's downloads. SEALEDBENCH_ROOT was still pointing at another worktree, so the summary JSON was written there; the module that ran was the clone. Byte-identical output means this recipe, on this machine, produces that stdout. It does not mean the original file was produced this way. A weight hash is not an execution attestation. Three capsules, one November round Round 33980212 unlocks at 2026-11-15T12:00:00Z. Three separate objects are sealed to it: Capsule Protocol Manifest Ciphertext cap v1 lkdeMQV1Cit1HArMK5gPDQm1oFIReGhRmntjdW8VjF0 q5O5v8XwkxCJkAyuZHd9V1pBV6XCWEtmCU5_2bactb0 cap2 v2 v7VsVcmXb9WY2m6mSTm7to79gd9YvjVAqo2OaCESgGY dLSRuUQDZV8aXut9v0faXVwQjvIdw-juKlCWbOyK-qo cap3 v2 zVURiYYKd4afS6sWM0ahJ0rkC9vrWUP-6la6t9IgbS8 QUTZFXr03mYfwJ51wlrrxztLgL6VSbhQDZwMvI8YS9U cap verifies WEAK on 2026-10-24, which is v1's binding again, and both decryption tools refused it. cap2 and cap3 each carry their own 32-byte secret. Neither capsule's questions appear anywhere in the public source tree. Before 2026-10-24T13:04:39Z both tle and tlock-js exited 1 with zero bytes of plaintext; tle reported that round 33980212 was still ahead of the then-current round 33347872. The pre-unlock checks for cap2 and cap3 are PASS. Do not try to decrypt them yet. After the round I will publish the run and point back at these exact IDs. What I want left exactly as written drand is an assumption. A threshold of operators could collude to compute a future round early. If the network stopped, the capsules would never open. I know the question families. I know v2's secret. The entropy check shows new seeds give new questions. It cannot show the families are fair, or that the templates are absent from training data. Decryption does not compel me to publish model answers. Declaring batches in advance makes a missing batch visible. It does not force me to release the answers. The gap between fresh arithmetic and GSM8K is not a contamination measurement. b2's registered integer scores are an artifact of the end-marker rule. The post-hoc block is labelled. b3 is the correction made after registration. Closed models are outside this protocol. I cannot see their weights or their serving stack. Once a round is published, anyone can copy its questions. The next batch needs a new round. Where this sits next to other work Very little of the machinery here is new on its own. Commitments, beacons, encrypted test sets and generated variants all exist. What I think is new is binding them into one scheduled future release and shipping the evidence. Work What it already does What SealedBench adds PeerBench Commits to tests, evaluates each model once, retires tests, audits with public randomness A batch generated by a sealed generator at a scheduled future round OpenRoboto Seeds robot-policy evaluation tasks from drand The generator is sealed to the same round that seeds it Jacovi et al., 2023 Encrypts test data so crawlers cannot ingest it Time-triggered public reconstruction with no key holder GSM-Symbolic Generated math variants; measures score sensitivity A verifiable commitment and release record for the batch Scale SEAL Expert-written private evaluations Public reconstruction instead of trusting the holder; it does not replace expert questions LiveBench Regularly refreshed questions with objective grading Explicit pre-release commitments per batch Decentralized AI Needs Decentralizable Trust (2026-06-29) Rerunnable, contestable LLM evaluation A small implementation with on-chain artifacts Verifiable AI (VeriTrace, 2026-07-31) Tamper-evident receipts for AI decisions Pre-release handling of questions, not decision receipts Why AI Governance Needs a Decentralized Quality Assurance Layer (2026-09-01) Multiple evaluators and signed evidence A smaller first version whose artifacts any reader can check What I have not done yet Someone else rebuilding a batch on another machine, without me. Larger batches declared in advance, with failures published beside successes. A fixed reference model to track difficulty across batches, without calling it formal equating. Several authors committing a larger pool before a later public draw. A threat model for vendors submitting their own answers, with timing evidence from independent witnesses. A batch registry on an AO process, but only if it adds a property this design lacks. What remains is a checkable claim. For each published batch you can verify what was committed, when it was committed, which beacon released it, which questions it produced, and how the published answers score. Materials Code: https://github.com/zigengZ/sealedbench, release v1.0.0 b1 index: AtNAZQtp_JqReahCoCpYcjP-b3P98jWtC_AJgVoBWdM b2 index: OhlaaBEIuQrtAu_FvJz1zAvHknpHFMXktWO0iFleidU b3 index: vlVlUfyBn4IUmbXjkSX8079QuySP-Qqr8rN8TbivzTg Scores: results/batches/b1/scores.json, results/batches/b2/scores.json (models first, then posthoc_sentinel_normalization), results/batches/b3/scores.json Figures drawn by article/fig/make_figs.py: fig1_timeline.png, fig2_fresh_vs_public.png, fig3_inclusion_ecdf.png, cover cover_1400x600.png (1400×600) Phase 1 probe data: results/day1/