Which language keeps an LLM honest?

Full technical report

Which Programming Language Keeps an LLM Honest?

Team 23N Labs · 16 October 2026

Abstract. When a language model writes code and a compiler is the referee, does the choice of language matter? We gave one small local coding model (Qwen2.5-Coder 1.5B, on a laptop CPU) the same 152 HumanEval tasks in seven setups: Python, Python checked by mypy, JavaScript, TypeScript in strict mode, Java, Go and Rust. Each setup's own compiler or checker judged every attempt. When it refused, only its error messages went back to the model, for at most two repair rounds. Hidden tests, never shown to the model, decided correctness. After feedback, the share of programs that were both accepted by the toolchain and correct was 61–62% for TypeScript and Python + mypy, 57–59% for Python, JavaScript and Java, 49% for Go and 39% for Rust. After a Holm correction, only the gaps to Rust are statistically clear. Counting only *semantic* mistakes (type, name and ownership errors, not syntax slips or missing imports), the toolchains flagged 56% of wrong first attempts in Rust, 45% in Java, 26% in Go, 10–17% in TypeScript, 6% with mypy and none in plain Python or JavaScript. Much of the apparent "recovery through compiler feedback" was repair of harness artifacts. 34 of TypeScript's 40 gains fixed syntax errors caused by the benchmark's TypeScript stop rule, which let the model run past the function. 14 of Go's 26 gains added imports that the completion prompt cannot add. When we re-ran TypeScript with JavaScript's stop rule, it matched JavaScript with no feedback (86 vs 86), and type-checker feedback added 7 tasks (93 vs 87, not significant). No toolchain kept the model fully honest: every setup still accepted 49–64 programs that failed the hidden tests. (results/table.md, results/summary.json)

1. Question

"Let the LLM write code and let the compiler keep it honest" is a common argument for typed languages in AI-assisted programming. GitHub's Octoverse 2025 reported TypeScript overtaking Python and JavaScript by monthly contributors and linked the shift partly to AI-assisted coding. Mündler et al. (PLDI 2025) showed that enforcing types *during* decoding more than halves compile errors in TypeScript. (sources.md)

We test the cheaper, black-box version anyone can run today: generate, compile, feed the diagnostics back, repeat. Three questions:

1. Catch: when the model's first attempt is wrong, how often does the language's toolchain notice before anything runs? 2. Recover: given only the toolchain's error messages, how often does the model fix the problem? 3. Trust: when the toolchain says yes, how often is the program actually right?

2. Method

Tasks. All 152 HumanEval problems that exist in every language of MultiPL-E (dataset revision 28441b60…, MIT). Python uses the original HumanEval prompts and tests (commit 6d43fb98…, MIT). The JavaScript, TypeScript, Java, Go and Rust prompts and tests are MultiPL-E's translations. File hashes are pinned in evidence/dataset-pins.json and checked before every run.

Model. qwen2.5-coder:1.5b (Q4_K_M, Apache-2.0) through Ollama 0.34.0 on an Intel Core i7-1068NG7 laptop, CPU only. Temperature 0, one sample per task, at most 384 new tokens. Toolchain versions are in evidence/environment.txt.

First attempt. Raw completion of the MultiPL-E prompt with that language's stop sequences, as in the MultiPL-E benchmark. The program is prompt + completion + hidden tests. The stop lists differ by language. JavaScript's is ["\nfunction ", "\n/*", "\n//", "\nconsole.log"], while TypeScript's is ["\nfunction ", "\n/*", "\n//", "\nclass"], with no console.log stop. Section 3.6 measures the effect.

The gate, per setup.

Each gate is a chosen configuration, not "the" compiler. Go's includes go vet, which goes beyond compilation. Mypy runs without --strict and checks only the solution, because the HumanEval test harness carries no types. tsc, javac, Go and rustc check the whole program, including the test file's call sites, as a real build would.

Repair. If the gate refuses, the model receives its own code and up to six sanitized diagnostic headlines, in a chat message asking for a corrected function with the signature unchanged. Echoed source lines are stripped from diagnostics. The function body, any helper functions and any imports in the reply are placed back into the original, unchanged prompt. At most two repair rounds. The first attempt is a raw completion and the repair is a chat prompt, so part of any repair gain may come from the change of prompting mode rather than from the diagnostics. A repair control with a generic "this failed to build" message was not run (section 5).

Scoring. Hidden tests run once on the first attempt and once on the final attempt. Untrusted code runs under macOS sandbox-exec with no network, file writes or process forks, plus CPU and wall-clock limits. The headline measure, accepted and correct, requires the gate to pass *and* the tests to pass. A gated workflow never ships a program its toolchain rejects, even when (as with TypeScript) the emitted JavaScript would have passed.

Rejection classes. Each first-try rejection is classed by its first diagnostic. Syntax covers parse errors (for example TS1xxx and Python SyntaxError). Import covers a standard-library package or module used without an import. Semantic covers everything else: type errors, unknown names and methods, ownership. The rules are in rejection_class in code/analyze.py.

Statistics. 95% Wilson intervals on single proportions. Exact McNemar tests on the same tasks across setups, for all 21 pairs, with a Holm correction.

3. Results

3.1 Headline

The TypeScript row's "no feedback" figure of 53 is depressed by the stop-rule artifact. With JavaScript's stop rule it is 86 (section 3.6).

Paired on identical tasks, with a Holm correction over all 21 pairs, five differences survive, and every one involves Rust. Python + mypy, TypeScript, JavaScript, Python and Java each beat Rust (corrected p ≤ 0.009). No other pair is clear. Go trails Python + mypy, TypeScript and Python (uncorrected p from 0.005 to 0.028), which is suggestive, but none of these gaps survives correction (Holm p ≥ 0.086). The top five are indistinguishable: Python vs TypeScript 20 vs 23 tasks, Python vs Python + mypy 1 vs 5, JavaScript vs TypeScript 11 vs 17, all with uncorrected p ≥ 0.2. For this model, Rust was the hardest language to get right, with or without a compiler. The full pair list is all_pairs_holm in results/summary.json.

3.2 Catch: does the toolchain notice a wrong first attempt?

Caution about the "all" column. For Java, Go and Rust a rejected program cannot run, so the design scores it as failing the tests. Every rejection therefore counts as a caught wrong attempt by construction, and false alarms (rejecting a program that would have passed) cannot be measured for these three languages. For Python, mypy, JavaScript and TypeScript we can still run the rejected program. Mypy refused 3 programs that would have passed (an unannotated empty dict, a "missing return", a list[float] vs list[int] return). Strict TypeScript also refused 3 such programs, in both runs.

What was caught.

- Go: 37 of 68 first-try rejections were standard-library packages used without an import. The completion-style prompt cannot add imports, and a developer's goimports would add them automatically. Neither these nor syntax slips are semantic mistakes a type system caught, which is why the semantic column is the one that answers the title question.

- TypeScript: 50 of 63 first-try rejections were syntax errors (TS1xxx). They came from the model running past the function into demo code, often opening a Markdown fence that tsc read as an unterminated template literal.

- Java: mostly unknown symbols and numeric type mismatches (long vs int).

- Rust: type mismatches (E0308), missing trait implementations (E0277), unknown methods (E0599) and use-after-move (E0382).

- Mypy: of its 10 catches, 3 were syntax errors that plain compile() already catches, so its incremental catch over Python is 7 of 63, of which 4 are semantic. Half of the 10 were undefined names. Three were missing imports such as math. Two were helper functions (is_prime, bucket_capacity) cut off by Python's "\ndef " stop sequence, which is another protocol artifact.

3.3 Recover: can the model use the error messages?

Only rejected programs are repaired, so on the headline measure repair cannot lose a task. Once syntax and import repairs are set aside, the semantic gains are small. Java gained 11, Go 11, TypeScript 6–7, mypy 4 and Rust 3. Rust is the outlier. Its compiler's messages were specific, but the 1.5B model usually returned the same code again: 63 of 96 Rust repair attempts were identical to the previous attempt (for example, still calling .len() on a char). Only 10 of 53 Rust rejections became green builds, and 43 tasks ended still rejected. With this model, a strict compiler that the model cannot satisfy turns would-be wrong answers into no answers. That is honest, but it is not productive.

3.4 Trust: when the build is green, is the program right?

This is the key limit. Even the strictest toolchains accepted 49–54 wrong programs out of 152, and a green build was correct only 55–66% of the time. A type checker tells you the program is *well-formed*; whether it does the right thing still depends on tests. The pattern fits Gao, Bird and Barr's estimate that static types would have caught about 15% of human-written JavaScript bugs: types catch one class of mistakes, not most mistakes.

3.5 Cost

Output tokens over all attempts, per accepted-and-correct task: Python 100, Python + mypy 114, JavaScript 124, TypeScript with JS stop rule 140, Java 329, TypeScript 335, Rust 410, Go 561 (tokens_per_shipped_correct). Repair rounds multiplied total output by about 2.4–3.2× for Java, Go and Rust. For TypeScript the factor was 1.3×, because its first attempts were already inflated by run-on output, and 1.2× with the JS stop rule. Mypy used 19% more output tokens in total than plain Python, or 14% more per accepted-and-correct task. The main study took about 5.6 hours of laptop time, and the sensitivity run about 1.6 hours.

3.6 Sensitivity: TypeScript with JavaScript's stop rule

The review of a first draft pointed out that the TypeScript stop list lacks JavaScript's "\nconsole.log" stop. In the main run, 110 of 152 TypeScript first completions ran on into demo console.log code, against 0 of 152 for JavaScript. We re-ran the TypeScript arm on all 152 tasks with the extra stop and changed nothing else (results/sensitivity-ts-jsstops.jsonl). No first completion ran on.

With the stop rule fixed, TypeScript matches JavaScript on first tries (12 tasks each way, p = 1.0). Strict type checking with feedback then adds 7 tasks. Against JavaScript the paired result is 18 vs 12 tasks, p = 0.36: a positive direction that this sample cannot confirm. TypeScript's final total is unchanged (93 in both runs, with 1 task differing each way). Repair recovered almost all of the run-on damage in the main run, at more than twice the tokens. So the main-run JavaScript-vs-TypeScript first-try gap (86 vs 53) was a harness difference, not a language difference. The fair estimate of what strict TypeScript adds over JavaScript for this model is +7 tasks (+6 over JavaScript's own feedback round), at 13% more tokens per accepted-and-correct task.

4. What this means

- For a small model, the language matters less than the loop. The top five setups finished within noise of each other, and only Rust is clearly behind after correction.

- Strict checkers catch real semantic mistakes, but how many depends on the type system. Rust and Java flagged about half of wrong first attempts for type, name or ownership errors. TypeScript and Go flagged a quarter or less. Mypy and the dynamic languages flagged almost none.

- Much of the visible "compiler recovery" fixes the harness, not the logic. TypeScript's +40 was mostly run-on syntax caused by a stop rule, and Go's +26 was half missing imports. Measured fairly, TypeScript's type checker added about 7 tasks over JavaScript, and Java's compiler added 11. Real-world setups that strip extra output and run goimports would never see these artifacts.

- For Python, mypy is a small, cheap upgrade. It added 7 incremental catches out of 63, of which 4 were semantic, and +8 tasks after feedback. The cost was 14% more tokens per accepted-and-correct task (19% in total) and three false alarms.

- A compiler the model cannot satisfy is a cost. Rust's diagnostics are excellent, but a 1.5B model rarely acted on them. Larger models, or feedback during generation (Mündler-Sasahara et al., 2026), may change this. This study cannot say.

- Tests remain the real referee. In every language, a third or more of green builds were wrong. Compiler feedback should be paired with test feedback.

5. Limitations

- One small model, one sample per task, temperature 0. Larger or frontier models may rank languages differently, and a stronger model will repair Rust far better. Results are not a ranking of languages for human developers.

- Benchmark shape. HumanEval tasks are short, self-contained functions with simple types, where type systems have the least to check. Repository-scale code, with real interfaces, is where types are expected to help most.

- Protocol choices affect languages differently.

- Stop lists are MultiPL-E's own and differ by language (section 2). In the main run, 110 of 152 TypeScript first completions ran on, against 0 of 152 for JavaScript, which produced 50 syntax-class TypeScript rejections. Section 3.6 removes the confound for TypeScript, and the main-run TypeScript first-try figure should be read through it. - Completion prompts cannot add imports, which hurts Go first tries. - Python's "\ndef " stop cuts off helper functions. - The repair step keeps only the function, helpers and imports from the reply.

- No generic-repair control. Repair switches from raw completion to a chat prompt, so some gain in any setup may come from the switch rather than from the diagnostics. A control round with a diagnostics-free "this failed to build" message is future work.

- Different test suites. Python uses the original HumanEval tests and the other languages use MultiPL-E translations. Test strength is not controlled.

- False alarms unmeasurable for Java, Go and Rust (section 3.2).

- Not fully deterministic. Ollama restarts may change some temperature-0 outputs. Rows that were re-run (below) were regenerated rather than replayed.

- Two harness bugs were found and fixed before analysis, both disclosed.

1. The first TypeScript run used --lib es2020, which hid Node's console from tsc and produced false rejections. The TypeScript arm was re-run with --lib es2020,dom. The 152 discarded rows are kept in evidence/typescript-lib-bug-runs.jsonl. 2. When a completion ended in a line comment with no newline, the first line of the tests was swallowed by the comment. Prompt, completion and tests are now always joined with a newline. The 72 affected rows (TypeScript 60, Java 8, Go 3, Rust 1) were re-run, and the discarded rows are in evidence/join-bug-runs.jsonl.

- Sandbox. macOS sandbox-exec denies network, writes and forks for executed code; compilation runs outside it. This is a research harness, not a hardened sandbox.

- Review status. An independent review of the first draft asked for major revisions. All nine required corrections were made, and the TypeScript arm was re-run (revision.md, review.md). This revised version was not independently re-reviewed.

6. Verification

code/checks.py does five things: 1. It verifies dataset hashes. 2. It recomputes every number in summary.json from the 1,064 raw rows plus the 152 sensitivity rows. 3. It confirms that every task is present in every arm. 4. It confirms that no hidden-test line appears verbatim in any diagnostic fed back or in any model reply. 5. It replays 49 seeded saved final programs (7 per setup) through the same gate and sandboxed tests.

The leak check is narrow. Compiler diagnostics can reveal line positions inside the test region: in the main run, 32 TypeScript and 2 Go fed-back diagnostics pointed there, and in two Go tasks a diagnostic named a test function. A scan for test-only strings, numbers and identifiers found only those test function names. No test inputs or expected outputs reached the model. All checks pass (evidence/check-table.md).

Reproduce: run python3 code/fetch.py, then python3 code/study.py measure. Run python3 code/study.py measure --arms typescript-jsstops --out results/sensitivity-ts-jsstops.jsonl for section 3.6. Then run python3 code/analyze.py, python3 code/checks.py and python3 code/render.py. Toolchains go in the git-ignored assets/tools/ folder (OPERATIONS.md).

Sources

See sources.md: MultiPL-E (Cassano et al.), HumanEval (Chen et al.), Type-Constrained Code Generation (Mündler et al., PLDI 2025), Generative Compilation (Mündler-Sasahara et al., 2026), CRUST-Bench (Khatry et al.), To Type or Not to Type (Gao, Bird and Barr, ICSE 2017), GitHub Octoverse 2025.

Evidence and limitations

Independent review

Reviewer: Kasauti · Team 23N Labs · 16 October 2026

Verdict

MAJOR REVISIONS. The arithmetic is sound: every headline number in report.md matches summary.json, and summary.json and table.md recompute identically from runs.jsonl. Hidden tests do not meaningfully leak. The two harness bugs are disclosed well. The problem is the paper's main interpretive claim, that strong checkers "mattered mainly as a recovery mechanism" with TypeScript as the lead example. That claim rests largely on a protocol asymmetry between JavaScript and TypeScript.

MultiPL-E's TypeScript stop sequences lack the "\nconsole.log" stop that the JavaScript set has. As a result, 110 of 152 TypeScript first completions ran on into demo console.log code, while 0 of 152 JavaScript completions did. All 50 syntax-class (TS1xxx) first-try rejections contain this run-on, and 32 contain a Markdown fence that opens an unterminated template literal. 34 of TypeScript's 40 feedback gains come from those 50 tasks, and on 33 of them the JavaScript arm's first attempt passed.

So most of the "TypeScript 53 → 93" recovery is repair of a truncation artifact. The type checker played no part in it, and a syntax-only gate such as node --check would have flagged the same programs. The limitation is disclosed in one line, but the abstract, sections 3.2 and 3.3 and section 4 still credit it to strict TypeScript. In the same way, the catch-rate comparison counts every Java, Go and Rust rejection as a "catch" by construction. Both issues have to be fixed in the text before the conclusions can stand. The data does not need to be re-collected.

Numbers checked

- I recomputed with python3 code/analyze.py. The regenerated results/summary.json and results/table.md are byte-identical to the saved copies.

- Headline table, section 3.1: accepted-and-correct counts with and without feedback, percentages and Wilson intervals all match summary.json (94/93/90/87/86/75/60; 86/53/89/86/75/49/57).

- Paired McNemar values in section 3.1 match paired_final: 20 vs 23, p = 0.76; 11 vs 17, p = 0.34; 1 vs 5, p = 0.22; 28 vs 13, p = 0.028; 43 vs 13, p < 0.001; 40 vs 7, p < 0.001. I also computed all 10 pairs among the top five. None has p < 0.26, so "top five indistinguishable" holds for every pair and not only the four reported. Pairs that were not reported but that I found: Python + mypy vs Go, 31 vs 12, p = 0.005; TypeScript vs Go, 32 vs 14, p = 0.011; Go vs Rust, 31 vs 16, p = 0.040.

- Catch table, section 3.2: 3/63, 10/63, 3/66, 60/96, 36/77, 68/103 and 53/95 match, as do their intervals.

- Rejection reasons:

- Go: 37 of 68 first rejections name an unimported standard-library package. Confirmed. - TypeScript: 50 of 63 are TS1xxx (TS1160 31, TS1005 8, TS1351 5, TS1002 5, TS1443 1). Confirmed. Note that summary.json first_rejection_kinds is truncated to six kinds and sums to 57, so the 50 cannot be read off summary.json alone. - Java: 17 "cannot find symbol" and 10 "incompatible types" out of 36. Consistent with "mostly". - Mypy and TypeScript each have 3 false alarms. Confirmed: mypy on tasks 9, 24 and 111; TypeScript on tasks 5, 37 and 69.

- Recovery table, section 3.3: 13→9, 63→56, 36→22, 68→45 and 53→10, with net gains +8, +40, +11, +26 and +3. All match.

- Trust table, section 3.4: silent-wrong counts 61/54/64/52/52/54/49 and correct-when-accepted rates 59.6–66.2% match.

- Cost, section 3.5: the per-task figures are output tokens divided by accepted-and-correct tasks. Python 100.4, mypy 114.4, JavaScript 123.8, Java 329.4, TypeScript 334.7, Rust 410.3, Go 561.0, which round to the reported values. summary.json tokens_per_final_pass divides by test passes instead, so it says 324.3 for TypeScript and 113.2 for mypy where the report says 335 and 114. That is a labelling mismatch, not an error.

- Runtime: the sum of elapsed_s is 5.56 h, against "about 5.6 hours". OK.

- Re-run evidence: join-bug rows are TypeScript 60, Java 8, Go 3 and Rust 1, which totals 72. The TypeScript lib-bug file has 152 rows. Both match.

- Rust repairs: 63 of 96 repair attempts returned the same completion as the previous attempt, which supports "usually returned the same code again".

- Mismatches found:

- (a) "Two differences are clear", but three are listed. - (b) "Repair rounds roughly triple the generation budget for the typed setups" does not hold for TypeScript. Total output divided by first-attempt output is 31,130 / 24,257 = 1.28× for TypeScript, against 2.66× for Java, 3.18× for Go and 2.44× for Rust. TypeScript's first attempts were already inflated by the run-on. - (c) "at about 14% more tokens" is per accepted-and-correct task. Total output is 19% higher (10,758 vs 9,039).

Required corrections

1. Attribute the TypeScript recovery to the stop-sequence artifact. - In the abstract, change "Strong checkers mattered mainly as a recovery mechanism: TypeScript went from 53 to 93 accepted-and-correct tasks, Go from 49 to 75." to state that 34 of TypeScript's 40 gains repaired syntax errors caused by the model running past the function. MultiPL-E's TypeScript stop list lacks the "\nconsole.log" stop that JavaScript has. Likewise, 14 of Go's 26 gains repaired missing imports that the completion prompt cannot add. - Make the same correction in section 4, bullet 2: "TypeScript, Go and Java turned many broken first attempts into working programs, using nothing but compiler output." 2. Quantify the asymmetry in Limitations. "MultiPL-E stop sequences let the chat-tuned model run past the function (hurting TypeScript first tries)" should state the actual stop lists: JS ['\nfunction ', '\n/*', '\n//', '\nconsole.log'] vs TS ['\nfunction ', '\n/*', '\n//', '\nclass']. It should also state the counts: 110/152 TypeScript vs 0/152 JavaScript run-on completions, and 50 resulting TS1xxx rejections. Then state plainly that the JavaScript-vs-TypeScript first-try gap (86 vs 53) and most of TypeScript's catch rate come from this harness difference, not from the languages. The cleanest fix is to re-run the TypeScript arm with the JavaScript stop list, or to post-truncate at \nconsole.log for both. If that is not done, the JavaScript-vs-TypeScript comparison must be labelled as confounded. 3. Fix the catch-rate definition for compiled languages. - For Java, Go and Rust a rejected program cannot run, so it is scored as failing. Every rejection therefore counts as a "caught wrong attempt": first_wrong_caught_by_gate equals first_gate_reject exactly, at 36, 68 and 53. False alarms cannot be measured for these languages (gate_runnable_when_rejected: false), whereas they were measured at 3 each for mypy and TypeScript. - Add this caveat under the section 3.2 table and next to the sentence "The statically typed languages flag half or more of the wrong first attempts." - Add that 37 of Go's 68 "catches" are missing imports and 50 of TypeScript's 60 are run-on syntax errors. Neither kind is a semantic mistake that a type system caught. 4. Section 3.1, multiplicity. "Two differences are clear. Python beat Go (28 vs 13, p = 0.028) and Rust (…), and TypeScript beat Rust (…)" needs two changes. Make the count match the list ("Three differences"). Downgrade Python vs Go: with seven uncorrected tests, p = 0.028 does not survive Holm or Bonferroni. Suggested wording: "Rust is clearly behind Python and TypeScript (p < 0.001); Python's lead over Go (p = 0.028, uncorrected) is suggestive." 5. Section 3.5. "Repair rounds roughly triple the generation budget for the typed setups" should read "roughly 2.4–3.2× for Java, Go and Rust; 1.3× for TypeScript, whose first attempts were already inflated by run-on output". 6. Section 4, bullet 3. "at about 14% more tokens" should be "about 14% more tokens per accepted-and-correct task (19% more in total)". Also note that 3 of mypy's 10 catches are syntax errors that plain Python's compile() already catches. The incremental catch is 7 of 63. 7. Section 3.2, mypy. In "mostly undefined names such as a missing import math", 5 of the 10 caught attempts are name-defined errors, which is half, not "mostly". Two of the five (is_prime, bucket_capacity) are helpers that were cut off by the "\ndef " stop sequence, which is another protocol artifact. Reword to "half were undefined names (missing imports, or helper functions cut off by the stop sequence)". 8. Section 6, wording of the leak check. "confirms that no hidden-test line appears verbatim" is accurate but narrow. Add that diagnostics can reveal line positions inside the test region and, in two Go tasks, the test function name. In this run, 32 TypeScript and 2 Go fed-back diagnostics point into the test region. A scan for test-only string, number and identifier tokens found only test function names. No test inputs or expected outputs reached the model. 9. Section 3.3. "No setup lost a correct task through repair" is true by construction for the headline measure, because only rejected programs are repaired. Say so, or drop the sentence.

Suggestions

- Report all 21 pairwise McNemar results, or at least all 10 among the top five, in an appendix with a Holm correction.

- Add a column to summary.json tokens_per_shipped_correct so that section 3.5 can be traced to a named field. Also stop truncating first_rejection_kinds to six entries.

- Add a "semantic catch" metric that excludes syntax and missing-import errors. That is the quantity the title question is actually about.

- Gate strength is a design choice that differs by arm. Go uses go vet, which goes beyond the compiler. Mypy runs without --strict and on the solution only, while tsc --strict checks test call sites. Rust and Java suppress warnings. Add one sentence acknowledging that each "compiler" is a chosen configuration.

- The repair step uses a chat prompt while the first attempt is a raw completion. Some of the gain for any arm may come from the change of prompting mode rather than from diagnostics. A control would help: a repair with a generic "this failed to build" message and no diagnostics.

- In sources.md, the MultiPL-E entry ("*mixed* conclusions about type annotations") goes beyond the abstract, which mentions only "language features". Mark it as from the paper body, since the file says abstract-level claims only.

Residual limitations

- One 1.5B quantised model, one greedy sample per task and 152 short HumanEval-style functions. Intervals of about ±8 points mean that differences under about 10 tasks are noise. Nothing here ranks languages for humans or for frontier models, and the report says so appropriately.

- Python uses the original HumanEval tests and the other languages use MultiPL-E translations. Translation quality and test strength differ by language and are not controlled.

- Temperature-0 outputs are not fully reproducible across Ollama restarts. The TypeScript arm and 72 other rows were regenerated rather than replayed.

- False-alarm rates for Java, Go and Rust remain unknown under this design.

- The sandbox is sandbox-exec for execution only. Compilation of untrusted code runs outside it. That is acceptable for a research harness and is disclosed.

Source characterisation was checked against the arXiv abstracts:

- Mündler et al. (PLDI 2025), "reduces compilation errors by more than half", TypeScript, HumanEval/MBPP: fair.

- Mündler-Sasahara et al. (arXiv 2607.13921), Rust, beats post-generation feedback, no numbers in the abstract: fair.

- CRUST-Bench (o1, 15 tasks single-shot) and Gao, Bird and Barr (about 15%): consistent with my knowledge of those papers.

Sources and data licence

HumanEval: MIT; MultiPL-E translations: MIT; Qwen2.5-Coder model: Apache-2.0. Website package carries code, aggregate outputs and figures only; no per-task prompts or generated programs.

sources.md

Sources and prior work — checked 16 October 2026

Each entry was opened or searched on 16 October 2026. Abstract-level claims only unless stated; full papers were not re-derived.

Not used / rejected: a secondary summary of a "PROBE" benchmark claim about Rust and compiler feedback could not be confirmed from the primary abstract (arXiv:2607.13820), so it is not cited as evidence. Search-engine summaries of CRUST-Bench self-repair gains were not confirmed against the abstract and are not used.

Novelty: no claim of novelty. This is a small, controlled local replication of a widely discussed engineering question with one small model.