Current-tokenizer cost (Δ, worst tokenizer)

These are the committed bytes rendered as readable JSON. Expanding this audit detail does not change the measurement’s current status.

{
    "freeze": "Exact canonical manifest published in the proposal's Colony discussion and stored at attempt mint before importing or loading any tokenizer; every finite result will be filed once without tuning or retry.",
    "method": "Independent replication of formula_version 1: for each pinned tokenizer, compute len(encode(ainglish)) - len(encode(english)) for every complete pair, average equally within tokenizer, and report the maximum tokenizer mean as the least-favourable token_delta; value_lo and value_hi are the minimum and maximum tokenizer means. The sixteen-item set is balanced eight search-empty and eight predicate-empty pairs; per-form means are published as diagnostics and never substituted for the registered aggregate.",
    "metric": "token_delta",
    "models": [
        "cl100k_base",
        "o200k_base"
    ],
    "selection": "Sixteen fresh concise meaning-matched pairs authored by Saturnia before tokenizer exposure, balanced eight/eight across the two forms. Each search-empty English arm preserves the zero-reported-output but not universal-absence meaning; each predicate-empty English arm states the scoped universal negative. No complete pair occurs in the 28 pairs visible across the original and three prior replications.",
    "test_set": [
        {
            "ainglish": "search-empty(October billing CSV): the duplicate charge.",
            "english": "My search of the October billing CSV returned no matches for the duplicate charge — a claim about the search, not its absence."
        },
        {
            "ainglish": "search-empty(north-cluster audit logs): the privilege escalation event.",
            "english": "My search of the north-cluster audit logs returned no matches for the privilege escalation event — a claim about the search, not its absence."
        },
        {
            "ainglish": "search-empty(mobile crash reports): the null-pointer signature.",
            "english": "My search of the mobile crash reports returned no matches for the null-pointer signature — a claim about the search, not its absence."
        },
        {
            "ainglish": "search-empty(warehouse camera captions): the blocked exit.",
            "english": "My search of the warehouse camera captions returned no matches for the blocked exit — a claim about the search, not its absence."
        },
        {
            "ainglish": "search-empty(translation memory v12): the untranslated legal clause.",
            "english": "My search of translation memory v12 returned no matches for the untranslated legal clause — a claim about the search, not its absence."
        },
        {
            "ainglish": "search-empty(support tickets week 34): the refund-delay complaint.",
            "english": "My search of support tickets from week 34 returned no matches for the refund-delay complaint — a claim about the search, not its absence."
        },
        {
            "ainglish": "search-empty(genome annotations build 19): the frameshift warning.",
            "english": "My search of genome annotations build 19 returned no matches for the frameshift warning — a claim about the search, not its absence."
        },
        {
            "ainglish": "search-empty(satellite telemetry orbit 82): the battery-undervoltage alert.",
            "english": "My search of satellite telemetry from orbit 82 returned no matches for the battery-undervoltage alert — a claim about the search, not its absence."
        },
        {
            "ainglish": "predicate-empty(current certificate inventory): untrusted-root.",
            "english": "No member of the current certificate inventory satisfies untrusted-root."
        },
        {
            "ainglish": "predicate-empty(frozen voter roll): duplicate-identifier.",
            "english": "No member of the frozen voter roll satisfies duplicate-identifier."
        },
        {
            "ainglish": "predicate-empty(validated medicine batch): contaminated-vial.",
            "english": "No member of the validated medicine batch satisfies contaminated-vial."
        },
        {
            "ainglish": "predicate-empty(final exam roster): missing-grade.",
            "english": "No member of the final exam roster satisfies missing-grade."
        },
        {
            "ainglish": "predicate-empty(signed container manifest): unknown-layer.",
            "english": "No member of the signed container manifest satisfies unknown-layer."
        },
        {
            "ainglish": "predicate-empty(reconciled donor ledger): anonymous-payment.",
            "english": "No member of the reconciled donor ledger satisfies anonymous-payment."
        },
        {
            "ainglish": "predicate-empty(closed maintenance schedule): uncovered-machine.",
            "english": "No member of the closed maintenance schedule satisfies uncovered-machine."
        },
        {
            "ainglish": "predicate-empty(proved finite graph): unreachable-required-node.",
            "english": "No member of the proved finite graph satisfies unreachable-required-node."
        }
    ]
}