research baseline v0.22
One semantic anchor. Many physical representations.
UTF-8 is an excellent interchange format. That is not the same claim as being the best representation for every computation over the same text. UTF-8X keeps UTF-8 as the normative, verifiable anchor and lets storage, transport, search, editing and model inference each use a reversible representation suited to their own costs.
The proposition
unified semantics ≠ unified physical representation
Traditional encodings fix one global relation between a character and its bytes. That is essential at system boundaries and became an assumption everywhere else — that the same semantic object should carry the same physical form in every region. UTF-8X denies only the extension, not UTF-8. Given a stable anchor and verifiable recovery, two representations may differ in bits while provably meaning the same sequence.
Why CJK first
Most common CJK characters take three UTF-8 bytes, and real documents concentrate heavily on a small set of characters, phrases, punctuation and domain terms. That concentration is what a local code table can exploit. But the design unit is not a national language — it is any observable, separable, measurable region of data.
What a region means here
Not geography. A region can be a language, a script, a document section, a domain vocabulary, a system tier (memory, disk, cache, network, index), a task (search, edit, model input, archive, stream), or a time band (hot conversation versus cold corpus).
What AI is not allowed to do
AI analyses the data and proposes a strategy: which blocks, which codec, which dictionary, which parameters. That strategy is then serialised, versioned and hashed. Decoding never calls a model.
This is a hard architectural line, not a preference. A format that needs a model to guess the original back is not lossless — different models, versions, inference environments and floating-point implementations would all be entitled to different answers. Once the manifest exists, the decoder reads a fixed spec and a verified hash. If the model that produced the strategy no longer exists, the data still decodes.
What this is not
- Not a replacement for UTF-8, and not a proposal to change any standard. UTF-8X is an anchored container above it.
- Not a claim to beat general-purpose compressors on ratio. Measured here, plain brotli is considerably smaller on the same corpus.
- Not a Unicode Consortium or IETF standard. The name is provisional and the format must never be labelled charset=UTF-8.
- Not performing normalisation. NFC, NFD, NFKC, NFKD, simplified/traditional conversion and variant merging are all outside the encoding layer.
- Not finished. v0.22 is a research engineering baseline, explicitly not a product.