Mojibake library v0.8 is out, and I’ve stabilized the C/C++/WASM API, which won’t change going forward. Rust and Elixir support are now mature and usable.
Why add a new Unicode library for Rust and Elixir (and C++) when they already have very good Unicode support? A summary:
| Unicode feature | C++ (mjb::) | Rust (mojibake-rs) | Elixir (Mojibake) |
|---|---|---|---|
| UTF-8/16/32 validation and conversion, with malformed-input policies and byte offsets | ✚ (<codecvt> was deprecated in C++17 and removed in C++26) | ◐ std has UTF-8 and UTF-16 but no UTF-32, BOM detection, or skip/stop policies | ◐ :unicode converts, but has no detection and no stop/replace/skip choice |
| Normalization NFC/NFD/NFKC/NFKD (UAX #15) | ✚ | ✚ | — String.normalize/2 already does these |
NFKC_Casefold, quick check, is_normalized | ✚ | ✚ | ✚ |
| Full upper/lowercase | ✚ (toupper only maps one unit at a time) | — | — |
| Titlecase | ✚ | ✚ | ◐ (String.capitalize only changes the first grapheme) |
| Casing for any BCP 47 locale (tr, az, lt, …) | ✚ | ✚ | ◐ (only :turkic and :greek) |
| Case folding, full and simple | ✚ | ✚ | ◐ (:string.casefold gives full folding only) |
| Caseless matching (D144–D147) | ✚ | ✚ | ✚ |
| Grapheme clusters (UAX #29) | ✚ | ✚ | ◐ graphemes exist. Mojibake adds counts and prefixes |
| Word and sentence segmentation (UAX #29) | ✚ | ✚ | ✚ |
| Line breaking (UAX #14) | ✚ | ✚ | ✚ |
| Bidirectional algorithm (UAX #9) | ✚ | ✚ | — not wrapped yet |
| Terminal / East Asian width (UAX #11), truncate to width | ◐ (C++23 std::format only estimates width) | ✚ | ✚ |
| Character names | ✚ | ✚ | ✚ |
| General category, scripts, Script_Extensions, blocks, planes | ✚ | ✚ | ✚ |
| Combining class, bidi class, decomposition type, numeric values | ✚ | ✚ (Rust has is_numeric but no values) | ✚ |
| Any binary or enumerated UCD property | ✚ | ◐ (Rust has a few predicates such as is_alphabetic and is_whitespace) | ✚ |
| Emoji properties and sequences, RGI (UTS #51) | ✚ | ✚ | ✚ |
| Hangul composition, decomposition and syllable names | ✚ | ✚ | ✚ |
| Collation and sort keys (UTS #10, DUCET) | ✚ (std::collate depends on the platform locale and isn’t UCA) | ✚ | ✚ |
IDNA to_ascii / to_unicode (UTS #46, Punycode) | ✚ | ✚ | ✚ |
| Confusables, skeletons, mixed scripts (UTS #39) | ✚ | ✚ | ✚ (Elixir’s compiler checks these internally, but there’s no public API) |
| Identifiers, XID_Start/Continue (UAX #31) | ✚ | ✚ | ✚ (internal to the tokenizer, not public) |
| Strict BCP 47 language-tag parsing | ✚ (std::locale names depend on the platform) | ✚ | ✚ |
| Text filters (normalize, collapse spaces, …) | ✚ | ✚ | — not wrapped |
What I needed to do to support C++, WASM Rust, and Elixir
The best thing about C is that it is “easy” to call the exported functions from other languages/frameworks. That’s why I decided to add support for multiple languages. Let’s see what these frameworks/languages provide.
C++
C++ is an extension of C, but C is NOT a subset of C++. Most ordinary C code can be compiled as C++, but Mojibake wasn’t very easy. C++ doesn’t like out-of-place pointers, implicit conversions, designated initializers, or implicit function declarations (among other things).
I was lucky because months (years?) ago I added a make test-cpp to the Makefile that tried to compile the project in C++. That helped me fix a long list of warnings/errors.
What was unusable was the Attractor library that was using the _Generic C11 feature. I needed to add a C++-specific function, which wasn’t ideal, but I was forced to. I’ve created a C++-specific assertion.
template <typename T1, typename T2>inline unsigned int att_assert_cpp( T1 result, T2 expected, att_op op, const char *description, const char *file, unsigned int line) {}
This was interesting because it helped me keep up with the latest C++ additions.
std::is_enum_vstd::is_signed_vstd::att_is_wide_integerstd::is_same_v- And so on.
The C++ wrapper is very simple. An example of normalization:
// "Cafe" + U+0301 COMBINING ACUTE ACCENTstd::string input = "Cafe\xCC\x81";// Text now is "Caf\xC3\xA9"std::string text = mjb::nfc(input);
The mjb::nfc, or mjb::normalize functions call the mjb_normalize_into C function. Check the README.md for the C++ API.
WASM
Compiling the project with Emscripten was also very easy; it compiled with just a few fixes here and there, and it also provides a __EMSCRIPTEN__ macro to detect if you are working with WASM.
What the Mojibake module needed was only to locate the compiled mojibake.wasm WASM file. In a WASM Emscripten-generated module, you can call C functions in multiple ways. Let’s imagine a function that adds two integers called add().
extern "C" {int add(int a, int b) { return a + b;}}
From JavaScript:
// This runs the function.Module.ccall( 'add', 'number', ['number', 'number'], [10, 20]); // 30// This lets you run it multiple times.const add = Module.cwrap( 'add', 'number', ['number', 'number']);console.log(add(10, 20)); // 30
This wasn’t enough for me because I had a big API, and it was creating a lot of noise. Fortunately, Emscripten lets you “export” the functions you want. See the EXPORTED_FUNCTIONS in the CMakeLists.txt file and the MojibakeWasmModule, automatically generated with all the exported functions. This lets you do this; for example, with the mjb_is_rgi_emoji:
isRGIEmoji(input: MojibakeInput, options: TextInputOptions = {}): boolean { const wasmInput = this.copyInput(input, options.encoding); const textPtr = this.copyText(wasmInput); try { // The exported function can now be called. return this.module._mjb_is_rgi_emoji(textPtr) ? true : false; } finally { this.free(wasmInput.ptr); this.free(textPtr); }}
The JavaScript code calls the _mjb_is_rgi_emoji function. It has an underscore before the name so it doesn’t pollute the Emscripten namespace.
Important thing to consider when you use it with WASM. JavaScript uses UTF-16 by default for strings. Be sure to pass UTF_16 in the options, or if you want to use UTF-8, JavaScript provides a TextEncoder.encode() API. But usually it is not a good idea. Mojibake support UTF-8, UTF-16 and UTF-32.
So, a little summary:
const mojibake = await Mojibake.create({ locateFile: (path, prefix) => `${prefix}${path}`});// ☺️: true.mojibake.isRGIEmoji('\u263A\uFE0F');
Another thing I needed to change is the mjb_for_each_codepoint function. That’s a function that calls a callback for every codepoint of a string.
bool next_character( const mjb_character *character, mjb_character_position position, void *context) {}
You can’t pass a JavaScript function to a C function, or better, you can call JavaScript code from C/C++, but it was too slow. That’s why I’ve added a _mjbForEachCodepointCallback function.
From C:
EM_ASM_INT( { return _mjbForEachCodepointCallback($0, $1, $2); }, &character, first_character ? MJB_POSITION_FIRST : MJB_POSITION_NONE, context);
From JavaScript:
forEachCodepoint(input: MojibakeInput, options: TextInputOptions = {}): NextCharacter[] | null { const previousCallback = (globalThis as any)._mjbForEachCodepointCallback; // ... // We set the _mjbForEachCodepointCallback function on the global object, when needed. (globalThis as any)._mjbForEachCodepointCallback = (character: Pointer, type: number) => { // ... return true; }; try { this.module._mjb_for_each_codepoint(...); // ... } finally { if(previousCallback === undefined) { // We delete it. delete (globalThis as any)._mjbForEachCodepointCallback; } else { // We restore the previous callback. (globalThis as any)._mjbForEachCodepointCallback = previousCallback; } // ... }}
Probably not the best solution, but enough for us.
Rust
Rust also lets you call C functions with ease. It’s called the FFI (Foreign Function Interface). I chose to have a low-level “sys” library with raw FFI bindings and another “safe” one.
let mut build = cc::Build::new();build .files(&sources) .file(&shim) .include(&src) .warnings(false);if build.get_compiler().is_like_msvc() { build.flag("/std:c11");} else { build.flag("-std=c11");}for (cargo_feature, define) in FEATURES { let enabled = env::var_os(cargo_feature).is_some(); build.define(define, if enabled { "1" } else { "0" });}build.compile("mojibake");
Important things:
- The
shimis a C file used to add some test helpers &srcis a list of all C files- MSVC uses
/somethingkind of CLI arguments. I needed it to use C11 - Mojibake lets you strip away part of the library. Example:
MJB_FEATURE_CHARACTER_NAMES
After this, we declare an FFI binding:
#[must_use]pub fn mjb_normalization_quick_check( text: *const mjb_text, form: mjb_normalization, quick_check: *mut mjb_quick_check_result,) -> mjb_status;
In the safe library, we declare instead a Rust function that calls it:
pub fn quick_check(self, s: &str) -> Result<QuickCheck> { let text = ffi::text(s); let mut answer = sys::MJB_QC_NO; check( // Here we call the C function unsafe { sys::mjb_normalization_quick_check(&text, self.raw(), &mut answer) }, None, )?; Ok(match answer { sys::MJB_QC_YES => QuickCheck::Yes, sys::MJB_QC_MAYBE => QuickCheck::Maybe, _ => QuickCheck::No, })}
And to use that:
Form::Nfc.quick_check("abc").unwrap();
And that’s it, very simple. The only tedious thing is that I needed to recreate all the C enums and function declarations. Fortunately, a lot of files are automatically generated, such as src/rust/mojibake-sys/src/functions.rs.
Elixir
I’ve been studying Erlang and Elixir lately, which is why I’ve added Mojibake to it. Elixir has the best Unicode support I’ve seen, but it was missing something, not very important for Unicode support but interesting enough.
Elixir and Erlang have what are called NIFs (Native Implemented Functions). The NIFs of a module are compiled and linked into a dynamic loadable shared library (.so on Linux, .dll on Windows). To compile the Elixir package, you need a C compiler. Fortunately, Elixir lets you use precompiled libraries, and I have a GitHub job that does this so users can simply import and use them.
mix.exs adds :elixir_make to the project’s compilers, so mix compile runs a small Makefile. The Makefile compiles each library file plus the binding, c_src/mojibake_nif.c, as C11 against erl_nif.h and links them to mojibake_nif.so.
The Mojibake.Nif module then:
@on_load :load_nifdef load_nif do path = :filename.join(:code.priv_dir(:mojibake), ~c"mojibake_nif") :erlang.load_nif(path, 0)end
The load callback runs once; it creates the atoms that the binding returns. A very important thing is that Elixir uses the custom allocators Mojibake let you use:
static void *mjb_nif_alloc(void *context, size_t size) { return enif_alloc(size);}// mjb_nif_realloc// mjb_nif_freemjb_allocator allocator = { NULL, mjb_nif_alloc, mjb_nif_realloc, mjb_nif_free};mjb_set_allocator(&allocator);
This lets us use the enif_alloc function. This way, every allocation the library makes goes through the BEAM’s memory allocator.
Now we see an example of a NIF function declaration:
// quick_check(text, form) -> {:ok, :yes | :no | :maybe}static ERL_NIF_TERM mjb_nif_quick_check( ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) { ErlNifBinary binary; mjb_text text; mjb_normalization form; if( argc != 2 || !mjb_nif_get_text(env, argv[0], &binary, &text) || !mjb_nif_get_form(env, argv[1], &form) ) { return enif_make_badarg(env); } MJB_NIF_MAYBE_DIRTY(mjb_nif_quick_check, binary.size); mjb_quick_check_result quick_check = MJB_QC_NO; mjb_status status = mjb_normalization_quick_check(&text, form, &quick_check); if(status != MJB_STATUS_OK) { return mjb_nif_error(env, status, NULL); } if(quick_check == MJB_QC_YES) { return mjb_nif_ok(env, mjb_nif_atoms.yes); } return mjb_nif_ok(env, quick_check == MJB_QC_MAYBE ? mjb_nif_atoms.maybe : mjb_nif_atoms.no);}
All the mjb_nif_something functions you see are utilities that call enif_get_atom to create a Mojibake argument. Example:
static bool mjb_nif_atom_is( ErlNifEnv *env, ERL_NIF_TERM term, const char *name) { char buffer[32]; // This copies the string representation of the // atom into buffer. return enif_get_atom( env, term, buffer, sizeof(buffer), ERL_NIF_LATIN1) > 0 && strcmp(buffer, name) == 0;}// Later we check it.if(mjb_nif_atom_is(env, term, "nfd")) { return MJB_NORMALIZATION_NFD;}
Now we can call quick_check/2 this way:
iex > Mojibake.quick_check("abc", :nfc){:ok, :yes}
Using C from Elixir was much easier than I thought. I just needed to generate a list of all the functions and the enums (see src/elixir/lib/mojibake/unicode_data.ex).
Until next time.