Token counts across Indian languages

Full technical report

Token counts across six FLORES-200 languages: a local replication

Team 23N Labs · Revision · Run 28-paper4-20261010-02

Token counts vary substantially across the five pinned tokenizer files. On 300 aligned FLORES-200 devtest rows, all 25 non-English/English ratios exceed one, with every reported bootstrap interval entirely above one. This is a narrower local replication; no originality is claimed. The source record identifies prior work on Indic tokenization and context consequences.

Scope and method. The direction considered all 1,012 devtest rows; the measured JSON covers 300, sampled without replacement using seed 28004 to keep the local multi-stage paper run bounded. Identical sorted row indices are used for English (eng_Latn), Hindi (hin_Deva), Bengali (ben_Beng), Tamil (tam_Taml), Telugu (tel_Telu), and Marathi (mar_Deva). Findings describe this sample of aligned FLORES-200 sentences, not all 1,012 rows.

The measurement code loads local tokenizer JSON files, checks asset SHA-256 hashes, asserts 1,012 lines per language, and encodes with add_special_tokens=False. Padding and truncation are disabled. GPT-2 and Qwen2.5 use byte BPE, mBERT uses WordPiece, and XLM-R and UMT5 use SentencePiece Unigram.

Mean tokens per sentence are arithmetic means. Language/English ratios divide aggregate token counts, rather than averaging sentence-level ratios. Tokens per character divide aggregate tokens by Python string lengths, including spaces and punctuation; these are Unicode code-point counts, not grapheme counts. Summary values are rounded to four decimal places.

For each ratio, 2,000 bootstrap resamples draw 300 aligned rows with replacement, preserving language–English pairing. Seeds are 28004 + 100 × language_index + tokenizer_index, using the recorded ordering. Sorted ratios at zero-based indices 49 and 1949 supply the reported 95% endpoints.

Measured findings. Values below reproduce the JSON’s stored precision. The complete measured table supplies all 30 means, intervals, character rates, budget values, and unknown-token counts.

XLM-R has the lowest mean count and English-relative ratio for each sampled Indic language among these five assets. The smallest ratio is XLM-R Marathi, 1.1984, with interval 1.1778–1.2185; the largest is GPT-2 Tamil, 15.5755, with interval 15.2287–15.9183. Cross-tokenizer rankings are descriptive: no paired intervals for differences between tokenizers were calculated.

Unknown-token substitution qualifies these comparisons. Unknown IDs are derived from the pinned tokenizer assets and counted in encoded ID arrays. mBERT records 9 English, 86 Hindi, 412 Bengali, 4 Tamil, 37 Telugu, and 0 Marathi unknown tokens across the sampled sentences. An unknown token can replace multiple characters, reducing counts while losing distinctions in the input. Consequently, smaller counts do not necessarily indicate better text representation. The other four tokenizers record zero unknown IDs for all six languages.

GPT-2 Hindi averages 200.5767 tokens per sentence versus 27.0067 for English: ratio 7.4269, interval 7.2617–7.5918. As a matching-tokenizer illustration, the JSON’s first sampled row, zero-based index 2, contains 308 Hindi versus 31 English GPT-2 tokens. This individual example does not estimate the aggregate ratio.

Hypothetical token budget and claim checks. The budget calculation is floor(4096 / unrounded_mean_tokens_per_sentence). GPT-2 gives 20 Hindi versus 151 English average-length sentence equivalents, with 20/151 ≈ 0.132. This is a mean-based illustration, not actual sentence packing or evidence that any associated model supports a 4,096-token context. It excludes prompts and special-token overhead.

The saved check table records five passes against the thresholds in claim-check code:

PASS means the specified numerical threshold passed. It does not certify model quality, provenance, generalization, or publication readiness.

Provenance and execution record. The asset manifest records filenames, byte sizes, SHA-256 hashes, repository identities, licenses, and pinned URLs. All six dataset files use DGME/FLORES-200 revision 3cb4c84a41f461a82cb8ad61903590a7351c2ec8.

The JSON’s eleven asset hashes match the supplied manifest. The local fetch_paper4.py run verified downloaded byte sizes and SHA-256 hashes, and the final paper4_checks.py run checked them again before re-encoding every sampled sentence. The committed check-run metadata binds the five saved outcomes to SHA-256 digests of the results, checks, measurement/check code and manifest, and records Python, platform and Tokenizers versions.

The hardware record reports an Intel Core i7, four physical cores at 2.3 GHz, 32 GB RAM, and x86_64 macOS 26.5.2; it attributes the more specific i7-1068NG7 designation to the brief. Runtime metadata records Python 3.9.10 and Tokenizers 0.21.1. No GPU was used; no model inference or speed measurement was performed.

Execution command recorded for this run:

TOKENIZERS_PARALLELISM=false python3 paper4_team.py run --run-id 28-paper4-20261010-02

The structured run log records this command's fetch, measurement, checks, table, chart, and six separate bounded research-team stages under run 28-paper4-20261010-02, followed by logged revision stages. The measurement script's …-01 default was overridden by the command. The final phone capture and induced asset-integrity failure are recorded in phone evidence and failure evidence.

The assessment reported that raw-count arithmetic, seeded sampling, means, ratios, budget values, bootstrap endpoints, and saved claim outputs matched. This revision resolves the report’s omitted unknown-token caveat, run-identity ambiguity, inaccessible citation paths, unsupported news-domain wording, and unclear PASS interpretation. It adds manifest and hardware provenance and a GPT-2-specific example.

The claim-card post links to this report with the correct ../report.md path. Its GPT-2 claim uses GPT-2 means; a separate mBERT figure illustrates token pieces without claiming to explain GPT-2's aggregate ratio. Seven code-generated SVGs and the offline 390px screenshot are in post/.

Remaining limitations. Bootstrap intervals describe row-resampling variability only. They lack finite-population correction despite sampling 300/1,012 rows, do not account for possible document clustering, and have no simultaneous-comparison adjustment. They do not capture translation, domain, corpus, or tokenizer-version uncertainty. Token counts establish neither quality, latency, pricing, nor task performance.

The final validator run checked exact tokenizer and language identities, seeded row selection, file hashes, re-encoded token counts, unknown-token totals, character denominators, means, ratios, hypothetical budgets, and every paired bootstrap endpoint. The review sessions deliberately received only draft, code, and results, so their own provenance conclusions are limited by that isolation. The local validation and hash-bound check record supply the additional operational evidence.

The source record links the checked upstream FLORES card, public mirror, and two prior papers. Upstream FLORES-200 says CC BY-SA 4.0, while the mirror card says CC BY 4.0. The discrepancy remains unresolved; this private package retains the upstream share-alike obligation. Full dataset files stay out of Git. The results and mechanism visual retain five token pieces of one aligned English/Hindi row, attributed to FLORES-200 under CC BY-SA 4.0. Nothing was published, posted, hosted, or deployed.

Evidence and limitations

The stored arithmetic supports the narrow sample claims, but validation and provenance wording need correction.

- Verified independently: All 9,000 stored counts are positive integers. Seeded sampling, all 30 means, aggregate ratios, budget values, and paired bootstrap endpoints reproduce. All five saved claim outputs match their functions. Both check-run digests match the supplied files. XLM-R’s reported descriptive minima hold.

- Context calculation is correct: floor(4096 / unrounded_mean) gives GPT-2 Hindi 20, English 151, quotient 0.132. The report appropriately distinguishes this from actual packing or supported model context.

- Fix unknown-token validation: Measurement counts token strings matching a marker; validation trusts the marker supplied in results.json. This does not independently establish unknown-token identity, and string matching can miss unknown encodings whose token text preserves the input surface. Derive unknown IDs from tokenizer assets and count encoded IDs; reassess the four zero-unknown claims.

- Correct the dataset-text statement: The report says the package “does not commit dataset text,” yet results.json contains complete English and Hindi sentences under pieces. Correct this wording and account for those excerpts when addressing the acknowledged licensing discrepancy.

- Separate consistency from provenance: Matching hashes establish consistency between supplied artifacts, not proof of execution. Within the permitted files, I cannot verify source assets, character denominators, re-encoding, licenses, hardware, or the claimed pipeline history. Bind future validation records to code and environment versions as well as results.

- Uncertainty is adequately scoped: Pairing and aggregate ratios are appropriate. Clustering, finite-population treatment, and simultaneous inference remain limitations; the draft acknowledges them. Threshold PASS results do not establish novelty or generalization. Keep tokenizer rankings explicitly descriptive.

- Presentation needs a final check: The post separates the mBERT illustration from GPT-2 claims and explains PASS. Add the “not a supported model context” caveat directly to claim 5. Generated SVGs and phone captures were outside this review, so visual readability remains unverified.

Retain the work as a local replication; the unknown-token audit and dataset-text/provenance corrections remain incomplete.

Sources and data licence

FLORES-200: conservatively CC BY-SA 4.0 despite the mirror's CC BY claim. The mechanism excerpt is attributed to FLORES-200, shared under CC BY-SA 4.0; not the full dataset. Tokenizer licenses are recorded in the source package.

sources.md

Source and novelty check — Team 23N Labs

Checked 2026-10-10. FLORES-200's original card states aligned configurations, 1,012 devtest rows and CC BY-SA 4.0; it now requires account acceptance. The public DGME mirror supplies the same six named devtest files without a gate, but its card says CC BY 4.0. We conservatively retain the upstream CC BY-SA 4.0 obligation and do not commit the full dataset files. The evidence and mechanism visual include the first five token pieces of one sampled English/Hindi row, attributed here to FLORES-200 under CC BY-SA 4.0. File hashes and exact mirror revision are in paper4-assets.json.

Priyansh Srivastava's *The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages*, arXiv:2607.24276 (2026), already studies FLORES-200 with six tokenizers and fourteen Indian languages, including context consequences. Maharaj Brahma et al.'s *Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights*, Findings of ACL 2026, pp. 32614–32632, DOI:10.18653/v1/2026.findings-acl.1632, also examines Indic tokenizer fertility. This paper is a local replication and narrower comparison, not a novel discovery of an Indic tokenizer tax. It compares five specific public tokenizer files and supplies executable claim checks and a phone graphic. The repository-specific files and model cards named in paper4-assets.json provide tokenizer provenance and license metadata. Model weights were neither fetched nor run.

Sources checked: upstream FLORES card, DGME mirror, two prior papers, five model cards = 9. Rejected as primary evidence: gated upstream data files (1); RoBERTa tokenizer (1) because its sample counts duplicated GPT-2. This does not mean those sources are false.