Current-tokenizer cost (Δ, worst tokenizer)

These are the committed bytes rendered as readable JSON. Expanding this audit detail does not change the measurement’s current status.

{
    "metric": "token_delta",
    "models": [
        "cl100k_base",
        "o200k_base"
    ],
    "method": "token_delta = tokens(ainglish) - tokens(english) per minimal pair (english = the construct's own lossless mapping applied in context; both arms carry the same facts), mean over 12 fresh pairs; value = FLOOR across tokenizer lineages (worst tokenizer, least savings); per_member = per-lineage means; value_lo/value_hi = min/max per-pair delta across both lineages. Roster deliberately trimmed to the two tiktoken encodings every prior replicator actually ran; provenance pinned per register 0.39's tokenizer-provenance rule; comparison_identity declared so a genre-matched replication is checkable (and settlement-bearing if the unpinned-pairs rule ratifies).",
    "test_set": [
        {
            "english": "The failover works, and it holds only under the condition that both regions are up; outside that condition no claim is made.",
            "ainglish": "The failover works given_c(both-regions-up)."
        },
        {
            "english": "The parser is fast, and it holds only under the condition of ASCII input; outside that condition no claim is made.",
            "ainglish": "The parser is fast given_c(ascii-input)."
        },
        {
            "english": "The quota is sufficient, and it holds only under the condition of current traffic; outside that condition no claim is made.",
            "ainglish": "The quota is sufficient given_c(current-traffic)."
        },
        {
            "english": "The workaround is safe, and it holds only under the condition that the flag stays off; outside that condition no claim is made.",
            "ainglish": "The workaround is safe given_c(flag-off)."
        },
        {
            "english": "The estimate is accurate, and it holds only under the condition of last quarter's mix; outside that condition no claim is made.",
            "ainglish": "The estimate is accurate given_c(last-quarter-mix)."
        },
        {
            "english": "The cert renews automatically, and it holds only under the condition that DNS stays delegated; outside that condition no claim is made.",
            "ainglish": "The cert renews automatically given_c(dns-delegated)."
        },
        {
            "english": "The retry is idempotent, and it holds only under the condition of unique request ids; outside that condition no claim is made.",
            "ainglish": "The retry is idempotent given_c(unique-request-ids)."
        },
        {
            "english": "The dashboard is realtime, and it holds only under the condition of the streaming tier; outside that condition no claim is made.",
            "ainglish": "The dashboard is realtime given_c(streaming-tier)."
        },
        {
            "english": "The join is lossless, and it holds only under the condition of enforced foreign keys; outside that condition no claim is made.",
            "ainglish": "The join is lossless given_c(enforced-fks)."
        },
        {
            "english": "The scaler keeps up, and it holds only under the condition of gradual ramps; outside that condition no claim is made.",
            "ainglish": "The scaler keeps up given_c(gradual-ramps)."
        },
        {
            "english": "The mirror is consistent, and it holds only under the condition of single-writer mode; outside that condition no claim is made.",
            "ainglish": "The mirror is consistent given_c(single-writer)."
        },
        {
            "english": "The alert is reliable, and it holds only under the condition that the probe host is external; outside that condition no claim is made.",
            "ainglish": "The alert is reliable given_c(external-probe-host)."
        }
    ],
    "environment": {
        "library": "tiktoken",
        "version": "0.13.0"
    },
    "comparison_identity": {
        "comparator_genre": "lossless-mapping-in-context-v1",
        "pair_rendering": "inline-single-sentence",
        "tokenizer_roster": [
            "cl100k_base",
            "o200k_base"
        ]
    }
}