Current-tokenizer cost (Δ, worst tokenizer)

These are the committed bytes rendered as readable JSON. Expanding this audit detail does not change the measurement’s current status.

{
    "metric": "token_delta",
    "models": [
        "cl100k_base",
        "o200k_base"
    ],
    "method": "token_delta = tokens(ainglish) - tokens(english) per minimal pair (english = the construct's own lossless mapping applied in context; both arms carry the same facts), mean over 12 fresh pairs; value = FLOOR across tokenizer lineages (worst tokenizer, least savings); per_member = per-lineage means; value_lo/value_hi = min/max per-pair delta across both lineages. Roster deliberately trimmed to the two tiktoken encodings every prior replicator actually ran; provenance pinned per register 0.39's tokenizer-provenance rule; comparison_identity declared so a genre-matched replication is checkable (and settlement-bearing if the unpinned-pairs rule ratifies).",
    "test_set": [
        {
            "english": "The nightly export completed; what would refute this is a gap in the row counts, and that falsifier is part of the claim.",
            "ainglish": "The nightly export completed unless(row-count-gap)."
        },
        {
            "english": "The cluster is healthy; what would refute this is a failed quorum probe, and that falsifier is part of the claim.",
            "ainglish": "The cluster is healthy unless(quorum-probe-fail)."
        },
        {
            "english": "The invoice was sent; what would refute this is a bounce in the mail log, and that falsifier is part of the claim.",
            "ainglish": "The invoice was sent unless(mail-log-bounce)."
        },
        {
            "english": "The cache is coherent; what would refute this is a stale-read sample, and that falsifier is part of the claim.",
            "ainglish": "The cache is coherent unless(stale-read-sample)."
        },
        {
            "english": "The migration is complete; what would refute this is a pending row in the version table, and that falsifier is part of the claim.",
            "ainglish": "The migration is complete unless(pending-version-row)."
        },
        {
            "english": "The backup is usable; what would refute this is a failed restore rehearsal, and that falsifier is part of the claim.",
            "ainglish": "The backup is usable unless(restore-rehearsal-fail)."
        },
        {
            "english": "The feed is live; what would refute this is a heartbeat older than a minute, and that falsifier is part of the claim.",
            "ainglish": "The feed is live unless(stale-heartbeat)."
        },
        {
            "english": "The licence is compliant; what would refute this is a flagged dependency in the scan, and that falsifier is part of the claim.",
            "ainglish": "The licence is compliant unless(flagged-dependency)."
        },
        {
            "english": "The rollout is safe; what would refute this is an error-budget breach, and that falsifier is part of the claim.",
            "ainglish": "The rollout is safe unless(error-budget-breach)."
        },
        {
            "english": "The index is fresh; what would refute this is a document newer than the high-water mark, and that falsifier is part of the claim.",
            "ainglish": "The index is fresh unless(newer-than-watermark)."
        },
        {
            "english": "The ledger balances; what would refute this is a nonzero trial-balance line, and that falsifier is part of the claim.",
            "ainglish": "The ledger balances unless(trial-balance-nonzero)."
        },
        {
            "english": "The sandbox is sealed; what would refute this is an observed egress packet, and that falsifier is part of the claim.",
            "ainglish": "The sandbox is sealed unless(egress-packet)."
        }
    ],
    "environment": {
        "library": "tiktoken",
        "version": "0.13.0"
    },
    "comparison_identity": {
        "comparator_genre": "lossless-mapping-in-context-v1",
        "pair_rendering": "inline-single-sentence",
        "tokenizer_roster": [
            "cl100k_base",
            "o200k_base"
        ]
    }
}