Anchor, regions, deterministic decode
The container is a header, a manifest and a sequence of independently recoverable blocks. Every block records the codec, codec version, dictionary identity, dictionary hash, offsets and a checksum — enough for a decoder that has never seen the encoder.
Four layers
Semantic layer
Scalar values. Identity of the characters themselves.
Anchor layer
The normative, verifiable form every representation must return to.
Region layer
Per-block reversible representations chosen by cost, not by convention.
Execution layer
Storage, network, search, editor, model inference — each with its own weights.
Per-block codec candidates
The scheduler picks per block against a cost function, and choosing raw UTF-8 is a legitimate answer. The worst case for UTF-8X should be a UTF-8 container with a bounded header — not something unreadable.
Original UTF-8, escaped verbatim
Local dictionary and short codes for frequent characters and phrases
Unicode code-point differencing across adjacent scalars
General-purpose compression, optionally with a shared dictionary
General-purpose compression via system libzstd
Two kinds of reversibility, never conflated
Semantic round-trip requires well-formed input and guarantees the same Unicode scalar sequence back, with no undeclared normalisation, glyph substitution or variant merging.
Byte-exact round-trip is needed when the source contains malformed UTF-8, historical illegal bytes, file signatures or non-text segments. Preserving semantics is then not enough and raw byte escapes are required.
The container must declare which mode it is in. The two are different guarantees and the format does not let them blur.
There is no globally optimal representation
The scheduler minimises a weighted cost over encoded size, encode time, decode time, memory, random-access cost, header and dictionary overhead, and compatibility risk. Archival weights size; live chat weights decode latency and header cost; an editor weights random access.
Because the weights differ by task, no single code table can be permanently optimal. That is the argument for choosing per region rather than declaring a winner.
Stated failure modes
- Dictionary poisoning — external dictionaries must be identified by hash, never by name or path.
- Decompression exhaustion — the container must bound both decoded length and expansion ratio before decoding.
- State desynchronisation — evolving online dictionaries need generations, checkpoints and a fallback to raw UTF-8.
- Content side channels — secret and attacker-controlled text must not share a length-affecting adaptive dictionary.
- Prompt injection into the scheduler — analyser output is schema-constrained and re-checked deterministically; text is never treated as instructions.