Current-tokenizer cost (Δ, worst tokenizer)

These are the committed bytes rendered as readable JSON. Expanding this audit detail does not change the measurement’s current status.

{
    "metric": "token_delta",
    "models": [
        "cl100k_base",
        "o200k_base"
    ],
    "test_set": [
        {
            "english": "Use one marker after a reported observation X that is being offered inside an argument for one reading against another.\n\n`X tells-apart(<R>)` = \"X, and the rival reading R predicts a different value for X, so X is an observation that separates R from the reading I am arguing for.\" The speaker commits to a checkable claim: someone can take R, derive what R predicts for X, and find it differs.\n\n`X fits-both(<R>)` = \"X, and R predicts X as well, so X does not separate the readings. I report it as context, not as support.\" This is the load-bearing half. It makes non-discriminating evidence sayable, so including it becomes a stated position rather than an implicature — without it the marker is droppable in exactly the way this register criticises English hedges for being.\n\nThe argument names the rival and is mandatory: an observation is not discriminating in the abstract, only with respect to some alternative. A bare `tells-apart` with no named rival is not the marker.\n\nThis is a distinct evidence axis. `obs/inf/rep/src` say how the evidence was obtained; `proxy(<M>)` says the measured quantity stands in for the claimed one; `ctl(<C>)` says the result was capable of being different; `caused-by/co-occurring` says whether a cause is asserted; `search-empty/predicate-empty` splits zero-found from nothing-exists; `[c=; ⊥ …]` names a future observation that would refute. None of them says whether an observation ALREADY CITED varies between the two readings on the table. They compose: `X fits-both(<R>) ctl(<C>) obs(<log>)` = \"I observed X directly, my check could have come out otherwise, and R predicts X too.\"",
            "ainglish": "X tells-apart(<rival reading>) | X fits-both(<rival reading>)"
        }
    ],
    "method": "tiktoken encode count difference between Ainglish form and English gloss",
    "environment": {
        "library": "tiktoken",
        "version": "0.14.0"
    }
}