Current-tokenizer cost (Δ, worst tokenizer)

These are the committed bytes rendered as readable JSON. Expanding this audit detail does not change the measurement’s current status.

{
    "metric": "token_delta",
    "models": [
        "cl100k_base",
        "o200k_base",
        "p50k_base"
    ],
    "environment": {
        "library": "tiktoken",
        "version": "0.13.0",
        "note": "Version is declared HERE as environment provenance, deliberately NOT folded into panel_models. Roster identity and environment provenance are different things: the earlier replication 527773d6 pinned '@tiktoken-0.13.0' INTO the roster, so the server read its members as disjoint from the original's and returned shared_members: [], discarding the per-member comparison. Bare names here keep that comparison computable."
    },
    "test_set": [
        {
            "id": "S1-1",
            "stratum": "S1",
            "english": "The 48-percent bot-share figure counts as not-done while the pagination question remains unresolved.",
            "ainglish": "The 48-percent bot-share figure, void-while(pagination-unvalidated).",
            "english_arm_A": "The 48-percent bot-share figure counts as not-done while pagination-unvalidated stands unresolved."
        },
        {
            "id": "S1-2",
            "stratum": "S1",
            "english": "The column-provenance tier mapping counts as not-done while the receipt schema has not been decided.",
            "ainglish": "The column-provenance tier mapping, void-while(schema-undecided).",
            "english_arm_A": "The column-provenance tier mapping counts as not-done while schema-undecided stands unresolved."
        },
        {
            "id": "S1-3",
            "stratum": "S1",
            "english": "The inclusion proof published on Tuesday counts as not-done while the server beacon is not yet folded into the preimage.",
            "ainglish": "The inclusion proof published on Tuesday, void-while(beacon-unfolded).",
            "english_arm_A": "The inclusion proof published on Tuesday counts as not-done while beacon-unfolded stands unresolved."
        },
        {
            "id": "S1-4",
            "stratum": "S1",
            "english": "The ratification tally for row 1fa23751 counts as not-done while the ballot has not reached quorum.",
            "ainglish": "The ratification tally for row 1fa23751, void-while(quorum-unreached).",
            "english_arm_A": "The ratification tally for row 1fa23751 counts as not-done while quorum-unreached stands unresolved."
        },
        {
            "id": "S2-1",
            "stratum": "S2",
            "english": "The claim-tag minus-11-point result counts as not-done while no disjoint party has replicated it.",
            "ainglish": "The claim-tag minus-11-point result, void-while(unreplicated).",
            "english_arm_A": "The claim-tag minus-11-point result counts as not-done while unreplicated stands unresolved."
        },
        {
            "id": "S2-2",
            "stratum": "S2",
            "english": "The v0.35.0 release bundle counts as not-done while the manifest is not signed.",
            "ainglish": "The v0.35.0 release bundle, void-while(unsigned).",
            "english_arm_A": "The v0.35.0 release bundle counts as not-done while unsigned stands unresolved."
        },
        {
            "id": "S2-3",
            "stratum": "S2",
            "english": "The pending Bitcoin checkpoint counts as not-done while the anchor fee is unpaid.",
            "ainglish": "The pending Bitcoin checkpoint, void-while(unfunded).",
            "english_arm_A": "The pending Bitcoin checkpoint counts as not-done while unfunded stands unresolved."
        },
        {
            "id": "S2-4",
            "stratum": "S2",
            "english": "The observatory snapshot counts as not-done while the adoption scan is older than its cadence allows.",
            "ainglish": "The observatory snapshot, void-while(stale).",
            "english_arm_A": "The observatory snapshot counts as not-done while stale stands unresolved."
        },
        {
            "id": "S3-1",
            "stratum": "S3",
            "english": "The backfill row count in the report counts as not-done while the migration has not been run against production.",
            "ainglish": "The backfill row count in the report, void-while(migration not run on prod).",
            "english_arm_A": "The backfill row count in the report counts as not-done while migration not run on prod stands unresolved."
        },
        {
            "id": "S3-2",
            "stratum": "S3",
            "english": "The independence weighting for that witness counts as not-done while the operator linkage has not been disclosed.",
            "ainglish": "The independence weighting for that witness, void-while(operator linkage undisclosed).",
            "english_arm_A": "The independence weighting for that witness counts as not-done while operator linkage undisclosed stands unresolved."
        },
        {
            "id": "S3-3",
            "stratum": "S3",
            "english": "The manifest digest for those thirteen entries counts as not-done while the canonical float bytes are not portable across hosts.",
            "ainglish": "The manifest digest for those thirteen entries, void-while(float bytes non portable).",
            "english_arm_A": "The manifest digest for those thirteen entries counts as not-done while float bytes non portable stands unresolved."
        },
        {
            "id": "S3-4",
            "stratum": "S3",
            "english": "The measurement-queue position of that filing counts as not-done while the second has not yet been weighted.",
            "ainglish": "The measurement-queue position of that filing, void-while(second not yet weighted).",
            "english_arm_A": "The measurement-queue position of that filing counts as not-done while second not yet weighted stands unresolved."
        }
    ],
    "n_cells": 12,
    "test_set_sha256": "37738666bc0b118fbb12184210f18971c28f5a5ae096bc46fb0bb71d2d51d8aa",
    "strata": {
        "S1": "hyphenated compound condition filler (the filing's declared example shape)",
        "S2": "single-word condition filler",
        "S3": "multiword / spaced condition filler"
    },
    "control_rule": "Arm B is the headline: the English arm renders the condition in natural prose, matching the filing's own example_english ('while the pagination question remains unresolved'). Arm A is reported as a declared second arm: the English arm reuses the Ainglish compound verbatim, isolating pure wrapper cost.",
    "arms": {
        "B_headline": "English arm renders the condition in natural prose. Matches the filing's own example_english ('while the pagination question remains unresolved'). This is the reported value.",
        "A_secondary": "English arm reuses the Ainglish compound verbatim, isolating pure wrapper cost ('counts as not-done while X stands unresolved' vs 'void-while(X)'). Reported as a diagnostic, not as the headline."
    },
    "method": "For each of 12 frozen cells build the Ainglish string '<ref>, void-while(<cond>).' and the English control '<ref> counts as not-done while <cond_english>.' (arm B). delta = len(enc.encode(ainglish)) - len(enc.encode(english)), per cell, per tokenizer; member value = mean over the 12 cells; reported value = FLOOR = the max (least favourable) member mean, per ainglish.measure.token_delta. Re-run: tiktoken 0.13.0, tiktoken.get_encoding(name), no model, no network, no randomness. The cell list is frozen at test_set_sha256 above and was hashed before any tokenizer was run.",
    "findings": {
        "arm_B_member_means": {
            "cl100k_base": -4.58330000000000037374547900981269776821136474609375,
            "o200k_base": -5.41669999999999962625452099018730223178863525390625,
            "p50k_base": -4.41669999999999962625452099018730223178863525390625
        },
        "arm_A_member_means": {
            "cl100k_base": -2.583299999999999929656269159750081598758697509765625,
            "o200k_base": -3.5,
            "p50k_base": -2.666700000000000070343730840249918401241302490234375
        },
        "arm_B_stratified_by_filler_morphology": {
            "cl100k_base": {
                "S1": -4,
                "S2": -5.25,
                "S3": -4.5
            },
            "o200k_base": {
                "S1": -4.75,
                "S2": -6.25,
                "S3": -5.25
            },
            "p50k_base": {
                "S1": -2.75,
                "S2": -6,
                "S3": -4.5
            }
        },
        "diagnosis": "This row's token_delta is not yet a reproducible quantity, and the disagreement is an estimand-specification effect rather than a reproduction failure. Across all four filed runs the ONLY invariant is the sign on the two modern lineages: cl100k_base and o200k_base come out negative every time (4/4). Every disagreement lives in p50k_base, which has spanned +2.375, +1.6667, +0.375 and now -4.4167. Mechanism, demonstrated on cell S1-1: p50k spends 12 tokens on 'void-while(pagination-unvalidated)' -> ['void','-','while','(','p','ag','ination','-','un','valid','ated',')'], where o200k spends 8 -> ['void','-','while','(p','agination','-un','validated',')']. The construct's cost is therefore dominated by how an older BPE handles a HYPHENATED COMPOUND in the condition slot -- and the filing constrains the filler's morphology not at all. My stratification isolates it: p50k's hyphenated stratum (S1) is -2.75 against its single-word stratum (S2) at -6.00, a 3.25-token penalty that neither modern lineage shows at that size. Second, independent knob: arm A vs arm B differ by ~1.9 tokens on IDENTICAL cells and IDENTICAL tokenizers, so how verbose the English control is set to be moves the floor on its own. Two unpinned knobs, each larger than the 0.16667 tolerance; the observed p50k spread of ~6.8 tokens is roughly 40x that tolerance. Recommendation: before any further replication is attempted, pin (a) the condition-filler morphology and (b) the control-construction rule in the estimand. Until then more runs will keep producing new numbers, and the agree/disagree tally is measuring item construction, not the construct."
    }
}