How far do free coding tokens go?

Full technical report

CPU-Only Code Generation: A 50-Task HumanEval Replication

Team 23N Labs Run 29-free-tokens-20261010-02 · 10 October 2026 Status: Working paper. (review.md, post/post.md)

Abstract. This local run recorded 35/50 HumanEval tasks passing (70.0%; approximate 95% Wilson interval 56.25–80.90%) and aggregate decoding throughput of 15.567 output tokens/s for Qwen2.5-Coder 1.5B on an Intel laptop CPU. The pipeline verified asset hashes and replayed all saved generated programs after the measurement. Kasauti independently checked the recorded arithmetic in a separate read-only session; it did not execute the benchmark. This is a hardware-specific replication; sustained capacity and hosted-service performance remain unmeasured. (results/problems.jsonl, results/measurement.json, evidence/check-table.md)

Scope and prior work. The source record identifies HumanEval as an existing 164-task Python benchmark and cites prior coding evaluations in the Qwen2.5-Coder technical report. The contribution is a local measurement and a dated comparison of differently denominated hosted allowances. The source record reports seven primary pages checked and two rejected search results; those activities were not repeated here. The earlier direction assessment explicitly lacked measurement evidence, whereas this revision includes the supplied result files. (sources.md, direction.md)

Methods. The implementation pins HumanEval to commit 6d43fb980f9fee3c892a914eda09951f772ad10d, selects sorted indices from random.Random(29029).sample(range(164),50), and generates one response per task sequentially. All 50 recorded IDs are distinct and match those indices. The hash-verified downloaded dataset remained outside Git; the local checker read it and replayed the tests. Kasauti's temporary review workspace did not contain the dataset, so that session could not independently confirm test contents. A fixed seed does not establish advance selection or exclude selective reporting. (code/study.py, results/problems.jsonl, evidence/team-run.jsonl)

The user-message prefix is:

> Complete this Python programming problem. Return only executable Python code implementing the requested function, with no markdown or explanation.

Generation uses Ollama’s local chat endpoint with temperature 0, seed 29029, context size 2,048, maximum generation 256 tokens, nonstreaming output, and a 30-minute keep-alive. The harness extracts fenced code when present and removes a leading fence. It concatenates the resulting code, benchmark tests, an entry-point check, and a randomized completion marker. Passing requires exit status zero and the marker as the final stdout line. (code/study.py)

The measurement used macOS sandbox-exec to deny networking, file writes, and process forks, with a five-second CPU limit, eight-second wall timeout, zero file-size limit, and randomized completion marker. After measurement, the checker replayed all 50 saved programs with a sanitized child environment; all recorded pass flags matched. No address-space or captured-output bound is implemented. Host reads remain allowed, and the marker shares the generated program’s execution context, so it is not tamper-proof verification. No obvious harness manipulation was observed in the supplied responses. (results/measurement.json, code/study.py, review.md)

Hardware, model, and provenance. Recorded hardware is an Intel Core i7-1068NG7 at 2.30 GHz, eight logical CPUs, and 32 GiB RAM, running macOS 26.5.2, Python 3.9.10, and Ollama 0.34.0. Ollama reported CPU-only execution. Its local metadata identifies qwen2.5-coder:1.5b, Q4_K_M quantization, and Apache License 2.0. The upstream model card states 1.54B parameters; the verified local model blobs total 986,062,089 bytes. HumanEval is MIT-licensed. No study-script license is established. (evidence/hardware.txt, evidence/model.txt, sources.md, evidence/team-run.jsonl)

The local fetch and claim-check stages verified these reproduction pins; Kasauti did not repeat the downloads:

The fetch script checks the dataset, manifest, and every referenced model blob before inference. The assets stay outside Git. (code/fetch.py, evidence/team-run.jsonl)

Results. Independent aggregation of the saved task records gives:

These values agree with the rounded measurement summary. The 15 failures comprise 13 assertion failures, one syntax error, and one index error; no test timeouts were recorded. The syntax-error response, HumanEval/32, reached the 256-token cap. (results/problems.jsonl, results/measurement.json)

Throughput is total output tokens divided by total Ollama decode duration, excluding prompt processing and testing. RSS is sampled approximately every 100 ms during generation and summed across matching Ollama/model-server processes. Its maximum is an observed process-memory peak, not total system memory or a guaranteed instantaneous maximum. (code/study.py)

The Wilson interval uses \(z=1.959963984540054\), without finite-population correction. It does not quantify repeat-run variability, contamination, or general coding ability. Results cover one model, quantization, machine, and task sample; context and generation limits may affect correctness.

Hypothetical daily output. Using the unrounded aggregate rate:

\[ \operatorname{round}\!\left(\frac{3816}{245.133792999}\times86400\right) =1{,}344{,}990. \]

This is hypothetical decode-only output under uninterrupted 24-hour generation, extrapolated from roughly four minutes of decoding. It excludes prompts, tests, idle time, electricity, and hardware wear. It establishes neither sustained daily capacity nor useful-code yield or cost-free operation. (results/problems.jsonl, results/measurement.json)

Hosted allowances. Saved excerpts dated 10 October 2026 record:

The research worker opened GitHub’s Copilot documentation and Amazon Q Developer pricing on 10 October 2026, and saved dated excerpts with response hashes. Kasauti and the read-only revision session did not browse; an excerpt and hash alone do not permit offline authentication of a whole response. No hosted calls or account experience were measured, and completions, agentic requests, and local tokens cannot be converted using this evidence. (evidence/hosted-source.json, evidence/hosted-source-aws.json)

Verification. The local study.py checks command verified pre-fetched asset hashes, replayed saved generated programs in the sandbox, recomputed all six claims, printed six PASS rows, and rewrote the check table. Each displayed claim.py command reruns those checks and prints its card's actual output. These are local replay and consistency checks, not independent proof of general correctness, a secure code sandbox, or hosted-service access. Kasauti separately recomputed record arithmetic without running the scripts. (evidence/check-table.md, code/claim.py, code/study.py, evidence/team-run.jsonl)

The report adopts these corrections to claim presentation:

- Cards 1–4 distinguish recorded outcomes from independently checked aggregate arithmetic. Card 2 now prints 56.25–80.90%. The unrounded lower endpoint is 56.2496%, which rounds to 56.2% at one decimal; Kasauti’s suggested 56.3% was a rounding error.

- Card 5 should label hypothetical decode-only output, identify use of the unrounded rate, and carry the uninterrupted-generation and operating-cost qualifications.

- Card 6 should say saved-source assertion, retain Amazon eligibility conditions, and state that Copilot’s reset period is unspecified.

The renderer checks for six saved PASS strings before generating SVGs, but does not itself validate raw results. The checker is the validation boundary. After review, the post and phone preview were regenerated with the precise interval, a visible decode-only caveat, source qualifications, and card commands whose printed outputs match the post. These editorial changes were not independently re-reviewed. (review.md, post/post.md, post/phone.png)

No sustained-load trial, energy measurement, throughput confidence interval, or independently authenticated full hosted response is available.

Evidence and limitations

Team 23N Labs Run 29-free-tokens-20261010-02

The recorded arithmetic supports a narrow local observation. Clearer verification claims and corrections to the methods and cards remain needed.

Required revisions:

- Qualify “CHECK PASS.” Separate independently checked arithmetic from recorded test replay and source assertions. claim.py reruns generated programs and rewrites the check table; it is not a read-only claim checker. render.py trusts saved PASS strings.

- Correct provenance and methods. The measurement explicitly distinguishes original execution from later replay with a sanitized environment. Document that distinction. Remove the unsupported “despite a code comment” statement and substantiate or soften “1.54 billion parameters” and “986 MB,” which the supplied model evidence does not establish.

- Repair card precision and scope. Card 2 displays 56.2% through double rounding; the unrounded lower bound rounds to 56.3% at one decimal. Card 5 displays a rounded-rate multiplication that does not reproduce its integer total; label use of the unrounded rate and emphasize hypothetical output.

- Clarify hosted allowances. State directly that the supplied Copilot excerpt establishes no reset period, and retain Amazon eligibility qualifications. Saved excerpts and response hashes alone do not authenticate unavailable source responses.

Reviewed only supplied files; independently recomputed aggregates without executing study scripts or generated code.

Sources and data licence

HumanEval: MIT; Qwen model: Apache-2.0. No benchmark prompts or solutions copied into the website package; source code and aggregate outputs only.

sources.md

Source record — read 2026-10-10 UTC

- HumanEval repository and MIT licence: 164 Python tasks in pinned data/HumanEval.jsonl.gz; test code is run only inside a restricted local subprocess. The original HumanEval paper establishes this as an existing benchmark, not a novel task.

- Qwen2.5-Coder 1.5B Instruct model card: Apache-2.0, 1.54B parameters; technical report already evaluates the series on coding tasks. The local Ollama 1.5B artifact is pinned by blob SHA-256 in the result, with about 986 MB downloaded.

- Copilot Free public documentation: says up to 2,000 code completions, read 2026-10-10 UTC; the exact excerpt and response hash are in evidence/hosted-source.json. This is a completion allowance, not a token allowance; account eligibility and actual experience were not measured. No account was created and no hosted call was made.

- Amazon Q Developer pricing: Free Tier lists 50 agentic requests per month for signed-in eligible users, read 2026-10-10 UTC; dated excerpt and response hash are in evidence/hosted-source-aws.json. This unit differs from Copilot completions and local tokens. No account or call was made.

- HumanEval execution warning: model output is untrusted and should be sandboxed. This study uses macOS sandbox-exec to deny network and file writes plus CPU and wall time limits. The local sandbox is not a general proof of safety.

Seven primary pages were opened and checked for these claims. Two search results were rejected as secondary or lacking a stable numeric allowance; they are not cited. Novelty: this is a local replication of code evaluation plus a hardware-specific throughput observation and a dated public-plan comparison. It does not establish a new model ranking or translate hosted completions into tokens.