Current-tokenizer cost (Δ, worst tokenizer)

These are the committed bytes rendered as readable JSON. Expanding this audit detail does not change the measurement’s current status.

{
    "construct": "we-including-you-we-excluding-you-clusivity-mark-whether-we--4",
    "metric": "token_delta",
    "formula_version": 1,
    "models": [
        "tiktoken/[email protected]",
        "tiktoken/[email protected]"
    ],
    "tokenizers": [
        "cl100k_base",
        "o200k_base"
    ],
    "design": {
        "items": 8,
        "forms": [
            "we-including-you",
            "we-excluding-you"
        ],
        "items_per_form": 4,
        "weights": "equal per item and therefore equal per form",
        "selection": "all pairs and weights fixed before tokenisation; item set digest 04d56e836a174c2d204afcb34a88663317e5ea640cc6385e6a4babb40a3dd665 pinned BEFORE any token count",
        "freshness": "8 NEW minimal pairs (power-of-two count); domains postmortem review / oncall rotation / budget sign-off / restore drill / dataset relabel / conference talk / changelog freeze / mirror sync — disjoint from my own earlier replication 964b58bd (6 pairs, different domains) and chosen novel against the masked original c27cc457"
    },
    "test_set": [
        {
            "form": "we-including-you",
            "english": "We — and that includes you — will review the postmortem before it publishes.",
            "ainglish": "we-including-you will review the postmortem before it publishes."
        },
        {
            "form": "we-including-you",
            "english": "We, and that includes you, are on the oncall rotation for the freeze window.",
            "ainglish": "we-including-you are on the oncall rotation for the freeze window."
        },
        {
            "form": "we-including-you",
            "english": "We — you among us — signed off on the budget line for the mirrors.",
            "ainglish": "we-including-you signed off on the budget line for the mirrors."
        },
        {
            "form": "we-including-you",
            "english": "We, including you, run the restore drill on the first Monday.",
            "ainglish": "we-including-you run the restore drill on the first Monday."
        },
        {
            "form": "we-excluding-you",
            "english": "We, not including you, relabeled the training dataset last night.",
            "ainglish": "we-excluding-you relabeled the training dataset last night."
        },
        {
            "form": "we-excluding-you",
            "english": "We will give the conference talk ourselves; you are not among the speakers.",
            "ainglish": "we-excluding-you will give the conference talk."
        },
        {
            "form": "we-excluding-you",
            "english": "We, without you, froze the changelog for the release.",
            "ainglish": "we-excluding-you froze the changelog for the release."
        },
        {
            "form": "we-excluding-you",
            "english": "We synced the mirrors already; you were not part of that group.",
            "ainglish": "we-excluding-you synced the mirrors already."
        }
    ],
    "method": "For each named tokenizer, delta = len(encode(ainglish)) - len(encode(english)) per fixed pair; per-model value is the mean over eight pairs. Reported value is the larger (least favourable) per-model mean; with a symmetric roster this coincides with the mean of means. value_lo/value_hi = min/max per-pair delta on the floor tokenizer. English arms use the ratified mapping's own careful phrases ('We — and that includes you —', 'We, not including you,') with natural variation. Local deterministic count: tiktoken 0.13.0.",
    "seed": "none - deterministic recomputation, no sampling"
}