Updating Mojibake to Unicode 18.0.0

I’ve updated Mojibake to Unicode 18. The beta period is over, and I don’t think there’s any need to wait for the latest summer fixes. My library generates all the files automatically, so I’m confident the generator script will not crash in September.

New emojis, like the U+1FADD (PICKLE), a meteor, and others, you say. But those are not a problem until you add something. When you change something is the problem. But fortunately, the Unicode standard is pretty backward compatible.

Step 1: analysis

The news can be found in the draft document at https://www.unicode.org/versions/Unicode18.0.0/ with my initial thoughts.

1. Unicode 18.0 adds 13,047 characters, for a total of 172,848 characters: not a problem. The library’s size will increase slightly.
2. Four new scripts: it’s ok; enums are automatically generated
3. JurchenSources.txt, SealSources.txt: not used right now. Will check later. See https://www.unicode.org/reports/tr60/tr60-2.html#Modifications
4. Other UTS have been updated accordingly: not a problem once the tests are ok.
5. Errata: it’s ok
6. Stability Policy Update: ok
7. New blocks: automatically generated
8. Conformance Changes: CONFORMANCE_REQUIREMENTS.md is already ok, just some tweaks at C4

Now it’s the time of UAXs, the most important in my opinion. I had only a couple of “let’s hope for the best” moments:

1. Rule LB12a was changed to disallow a break between BA and GL: pretty dangerous
2. Rule GB9c (“Do not break within certain combinations with Indic_Conjunct_Break (InCB)=Linker.”) has been revised: same

When you change a rule, it isn’t always a good idea.

Step 2: generating new data tables

I changed UNICODE_VERSION to draft on the generate script and tried to run make generate.

utils/generate/tables/case.ts:164
throw new Error(`Simple case folding mapping is too large to pack: ${row.codepoint}`);

Not a good start. What happened? It happened that Mojibake is trying to save as much data as possible, and it was too much of an optimization. Unicode added to the CaseFolding.txt.

1DF95; F; 0073 0073; # LATIN SMALL LIGATURE LONG S WITH DESCENDER S
1DF95; S; 00DF; # LATIN SMALL LIGATURE LONG S WITH DESCENDER S

What does this mean? That the U+1DF95, a character outside BMP, has been added to the special cases.

BMP is Plane 0; it is the Basic Multilingual Plane, AKA the first 16 bits that were used in the UCS2 encoding before we discovered that we had more than 65,535 characters.

How the old packing worked

Simple mappings were stored as one uint32_t:

bits 31........16 | bits 15.........0
source codepoint  | mapped codepoint

The generator did:

(row.codepoint << 16) | row.mapping

For example, the mapping U+1E9E -> U+00DF was packed as:

0x1E9E00DF
  ^^^^ source U+1E9E
      ^^^^ target U+00DF

One mapping occupies four bytes. However, each side only has 16 bits, so both values must be no greater than U+FFFF.

Why did U+1DF95 not fit?

Simple: U+1DF95 > U+FFFF. Packing it into the old format would require: 0x1DF95 << 16

That needs 37 bits before adding the mapping. A 32-bit value would silently lose the leading 1, decoding the source as U+DF95 instead of U+1DF95.

The old generator deliberately protected against this corruption:

if(row.codepoint > 0xFFFF || row.mapping > 0xFFFF) {
    throw new Error(...);
}

Always be redundant in throwing errors!

The new supplementary format

A Unicode codepoint needs at most 21 bits. Therefore, the new supplementary entry uses two 21-bit fields inside a uint64_t:

bits 41........21 | bits 20.........0
source codepoint  | mapped codepoint

The generator packs it with: (BigInt(row.codepoint) << 21n) | BigInt(row.mapping)

And generates a new C table with such a long name I choose to remember myself that Unicode is a convoluted standard:

static const uint64_t mjb_unicode_case_fold_simple_supplementary_mappings[] = {
    0x03BF2A000DFULL,
};

We have two tables now. I didn’t want to have only one; otherwise, I would have double the size. The new one has only one entry. Don’t be alone you, new codepoint from around the block (pun); I’m pretty sure you will have new friends in the near future.

Step 3: try the tests

The generation worked well. Time to test things. On the first run, I got 122,773 failures. [insert here a very sad emoji here]

So it’s time to apply the changes.

LB12a

[^SP BA HY HH] × GL is now [^SP HY HH] × GL. This was scaring me, but it was a one-line change.

U+1DF95

The infamous new codepoint needed to be added to the mjb_maybe_has_special_casing function together with the other eight.

GB9c

Finally, some simplifications here.

\p{InCB=Consonant} [ \p{InCB=Extend} \p{InCB=Linker} ]
* \p{InCB=Linker} [ \p{InCB=Extend} \p{InCB=Linker} ]*
× \p{InCB=Consonant}
become
\p{InCB=Linker} \p{InCB=Extend}* × \p{InCB=Consonant}

It has greatly simplified the code.

Jurchen and Seal were added to the table for computing implicit weights

I was wrong here; I thought it would have been automatic. Check the draft document to see the changes at https://www.unicode.org/reports/tr10/tr10-54.html#Reserved_Weights.

Before Unicode 18, I had a implicit_ranges static arrays, but now it’s not enough. Too boring as an argument to be part of a blog post. Check the PR for details if you are interested.

Step 4: final cleanup

The cleanup didn’t clean anything at all. All the URLs to a https://www.unicode.org/* needed to be updated, and it created a lot of noise.

Total number of tests passed from 1,610,123 to 1,600,176, so it lost 10k tests? But it makes sense:

1,610,123 −9,947 = 1,600,176

Step 5: no other steps

Unicode is not the actual standard, I know. Should I have been waiting? Nah. It was fun, and pretty exciting.

Until next time.