Today’s topic is about intrinsics. Intrinsics are functions supplied by a compiler that map to hardware instructions. These fall into two buckets:

1. builtins

For example, the function __builtin_popcount(unsigned int) for GCC and Clang provide a population count function. These lower to special “compiler instructions”, e.g. llvm.ctpop for clang that are fast pathed to special hardware instructions (e.g. POPCNT for x86). Not all architectures have hardware instructions for every builtin, so sometimes these will be emulated in software depending on your target.

2. Hardware intrinsics

For example, _mm_popcnt_u32(unsigned int) for x86 on SSE4.2. This intrinsic looks like a function, and will lower to POPCNT in asm.

We’ll focus on the second bucket today.

Slate provides architecture support for x86_64, ARM, and RISC-V in its libc-shim. That means we also need to support intrinsics for those architectures. Now a quick google search shows that there are ~10000 intrinsics for x86 and arm and much more for RISC-V.

If I were to write out every signature by hand, I probably wouldn’t transcribe all the signatures before dying (of boredom). So we have to have another approach.

Thankfully, the compiler authors and hardware manufacturers have us covered.

Read the Manual

Intel provides an xml file that you can parse for every intrinsic, what its name is, its arguments, its return type, a doc string, what cpu features it uses, what header file it belongs in, etc. ARM does something similar, except in JSON, and RISC-V provides a python script that generates all of its headers, because it has a lot more intrinsics than either x86 or ARM.

You can use these to generate the header files themselves. In clang, the way this is done is by parsing the manual files, and then generating definitions in a DSL called tablegen. This is used all through clang, and it is a somewhat C looking language that is used for generating all kinds of things, include llvm instructions, Clang’s IR (which we’ll get to another time)

How this flows through clang looks like this:

┌─────────────────────────────┐
│ ISA manual                  │
│ (Arm ARM, RISC-V RVV spec,  │
│ Intel SDM)                  │
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│ hand-written TableGen       │
│ (arm_neon.td, arm_sve.td,   │
│ RISCVVectorBuiltins.td)     │
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│ clang-tblgen emitter        │
│ (NeonEmitter / SveEmitter / │
│ RVV intrinsic emitter,      │
│ `-gen-arm-neon` etc.)       │
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│ generated C header          │
│ (immintrin.h, arm_neon.h,   │
│  arm_sve.h, riscv_vector.h) │
│                             │
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│ used by CGBuiltin.cpp       │
│ + Sema to map builtin call  │
│ -> llvm.* intrinsic at      │
│ CIR/IR emission time        │
└─────────────────────────────┘

Sadly that last point means slate can’t just copy paste clang’s headers into libc-shim and call it a day. The Clang IR (CIR) that it reads turns into raw llvm intrinsics.

The power of Tablegen

Since Slate is pinned to a given Clang version (since Clang IR isn’t built by default for Clang yet), we can use their tablegen parser. This allows us to get all the information that clang records for each intrinsic, and then change its form to something slate can recognize.

To that end, there’s a support crate for slate called slate-intrinsic-gen that parses each tablegen and creates a struct for each entry that looks like:

IntrinsicSignature {
    name: "llvm.x86.sse42.crc32.32.8",
    overloaded: false,
    ret: Some("i32"),
    params: Some(&[
        IntrinsicParam {
            llvm_type: "i32",
            immarg: false,
        },
        IntrinsicParam {
            llvm_type: "i8",
            immarg: false,
        },
    ]),
    overloaded_positions: None,
}

This allows us to recover the whole name itself. For example, for this program:

__attribute__((target("sse4.2"))) static unsigned long long crc32_probe(void) {
  unsigned int crc = _mm_crc32_u8(0u, 0x12u);
  crc              = _mm_crc32_u16(crc, 0x3456u);
  crc              = _mm_crc32_u32(crc, 0x789abcdeu);
  return _mm_crc32_u64(crc, 0x123456789abcdef0ull);
}

Slate will generate something like this:

#[target_feature(enable = "sse4.2")]
unsafe fn _mm_crc32_u8(a: u32, b: u8) -> u32 {
    unsafe { __slate_intrinsic_x86_sse42_crc32_32_8_f27bf8581dad0801(a, b) }
}

#[target_feature(enable = "sse4.2")]
unsafe fn _mm_crc32_u16(a: u32, b: u16) -> u32 {
    unsafe { __slate_intrinsic_x86_sse42_crc32_32_16_658f6bf45a185a4a(a, b) }
}

#[target_feature(enable = "sse4.2")]
unsafe fn _mm_crc32_u32(a: u32, b: u32) -> u32 {
    unsafe { __slate_intrinsic_x86_sse42_crc32_32_32_f5e6b09e791bc818(a, b) }
}

#[target_feature(enable = "sse4.2")]
unsafe fn _mm_crc32_u64(a: u64, b: u64) -> u64 {
    unsafe { __slate_intrinsic_x86_sse42_crc32_64_64_a6b1e708219fb1bb(a, b) }
}

#[target_feature(enable = "popcnt,sse3,sse4.1,sse4.2,ssse3")]
unsafe fn crc32_probe() -> u64 {
    let v1: u32 = unsafe { _mm_crc32_u8(0, 18) };
    let v2: u32 = unsafe { _mm_crc32_u16(v1, 13398) };
    let v3: u32 = unsafe { _mm_crc32_u32(v2, 2023406814) };
    let v4: u64 = 1311768467463790320u64;
    unsafe { _mm_crc32_u64(v3 as u64, v4) }
}

unsafe extern "unadjusted" {
    #[link_name = "llvm.x86.sse42.crc32.32.8"]
    fn __slate_intrinsic_x86_sse42_crc32_32_8_f27bf8581dad0801(_0: u32, _1: u8) -> u32;
    #[link_name = "llvm.x86.sse42.crc32.32.16"]
    fn __slate_intrinsic_x86_sse42_crc32_32_16_658f6bf45a185a4a(_0: u32, _1: u16) -> u32;
    #[link_name = "llvm.x86.sse42.crc32.32.32"]
    fn __slate_intrinsic_x86_sse42_crc32_32_32_f5e6b09e791bc818(_0: u32, _1: u32) -> u32;
    #[link_name = "llvm.x86.sse42.crc32.64.64"]
    fn __slate_intrinsic_x86_sse42_crc32_64_64_a6b1e708219fb1bb(_0: u64, _1: u64) -> u64;
}

Now note that slate does all of its external linking itself, since not all intrinsics in clang are supported by rust.

But there’s one thing wrong here: the tablegen says crc32’s signature takes a i32 and i8 and returns an i32 but the intrinsic itself is all unsigned (u32 and u8 returning a u8). What gives?

Some thorns in our side (Overrides)

Note that actually some functions have different signatures, used to disambiguate what the actual function is:

So we keep a separate table that contains the overridden link name, its params and return type as well, so we can use this to generate what we want. For crc32.32.8, that means we override its parameters:

StdarchOverride {
    link_name: "llvm.x86.sse42.crc32.32.8",
    params: &["u32", "u8"],
    ret: Some("u32"),
},

Also, there’s another thorn in our side where rust’s target features don’t map to LLVM’s:

for example, crc32 functionality is gated behind the feature flag crc32 in clang. In Rust, this is gated behind the broader sse4.2 flag. Likewise bmi, bit manipulation instructions for haswell is different on clang than it is in rust, bmi1. So we have to disambiguate that as well.

Why bother?

If we pass the previous program to C2Rust, note that it’ll lean on rust to emit the right intrinsics:

#![feature(stdsimd)]
#[cfg(target_arch = "x86")]
pub use ::core::arch::x86::{_mm_crc32_u8, _mm_crc32_u16, _mm_crc32_u32};
#[cfg(target_arch = "x86_64")]
pub use ::core::arch::x86_64::{
    _mm_crc32_u8, _mm_crc32_u16, _mm_crc32_u32, _mm_crc32_u64,
};

unsafe extern "C" fn crc32_probe() -> ::core::ffi::c_ulonglong {
    let mut crc: ::core::ffi::c_uint = _mm_crc32_u8(
        0 as ::core::ffi::c_uint,
        0x12 as ::core::ffi::c_uchar,
    );
    crc = _mm_crc32_u16(crc, 0x3456 as ::core::ffi::c_ushort);
    crc = _mm_crc32_u32(crc, 0x789abcde as ::core::ffi::c_uint);
    return _mm_crc32_u64(
        crc as ::core::ffi::c_ulonglong,
        0x123456789abcdef0 as ::core::ffi::c_ulonglong,
    );
}

Which works, most of the time. Rust supports almost all intrinsics for x86 that return a value, however, it doesn’t support the intrinsics that are used for their effects:

For example, take this program that flushes the cache line(s) for the CPU:

#include <immintrin.h>
#include <stdio.h>

__attribute__((target("clflushopt"))) void flush(void *addr) {
    __builtin_ia32_clflushopt(addr);
    _mm_sfence();
}

int main(void) {
    int x = 42;
    flush(&x);
    printf("%d\n", x);
    return 0;
}

There is no intrinsic for this in the Rust stdlib, so the flush function gets deleted by C2Rust wholesale:

#[cfg(target_arch = "x86")]
pub use ::core::arch::x86::_mm_sfence;
#[cfg(target_arch = "x86_64")]
pub use ::core::arch::x86_64::_mm_sfence;
extern "C" {
    fn printf(__format: *const ::core::ffi::c_char, ...) -> ::core::ffi::c_int;
}
unsafe fn main_0() -> ::core::ffi::c_int {
    let mut x: ::core::ffi::c_int = 42 as ::core::ffi::c_int;
    flush(&raw mut x as *mut ::core::ffi::c_void); // flush called
    // without a definition
    printf(b"%d\n\0".as_ptr() as *const ::core::ffi::c_char, x);
    return 0 as ::core::ffi::c_int;
}
pub fn main() {
    unsafe { ::std::process::exit(main_0() as i32) }
}

By adding the mapping yourself, you can translate this program and get the same result as the C would.

#![feature(clflushopt_target_feature)]
#![feature(abi_unadjusted)]
#![feature(link_llvm_intrinsics)]
#![feature(c_variadic)]
#![allow(
    dead_code,
    unused,
    non_camel_case_types,
    non_snake_case,
    non_upper_case_globals,
    arithmetic_overflow,
    unconditional_panic,
    suspicious_runtime_symbol_definitions,
    unpredictable_function_pointer_comparisons,
    unused_comparisons
)]

unsafe extern "C" {
    fn printf(_0: *const core::ffi::c_char, ...) -> i32;
}

#[target_feature(enable = "clflushopt")]
unsafe fn flush(_v0: *mut core::ffi::c_void) {
    unsafe {
        unsafe { __slate_intrinsic_x86_clflushopt_2bf6694e0a93403a(_v0) };
    }
    unsafe {
        unsafe { __slate_intrinsic_x86_sse_sfence_f8b270d178b3d220() };
    }
    return;
}

fn main() {
    let mut x: i32 = 42;
    let _v2: *mut core::ffi::c_void = std::ptr::addr_of_mut!(x) as *mut core::ffi::c_void;
    unsafe { flush(_v2 as *mut core::ffi::c_void) };
    unsafe { printf(c"%d\n".as_ptr(), x) };
    std::process::exit(0 as i32);
}

unsafe extern "unadjusted" {
    #[link_name = "llvm.x86.clflushopt"]
    fn __slate_intrinsic_x86_clflushopt_2bf6694e0a93403a(_0: *mut core::ffi::c_void);
    #[link_name = "llvm.x86.sse.sfence"]
    fn __slate_intrinsic_x86_sse_sfence_f8b270d178b3d220();
}

In Sum

The whole flow looks like this:

┌─────────────────────────────┐
│ C source                    │
│ _mm_add_epi32(..)           │
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│ Clang + CIR                 │
│ (headers inlined, resolved  │
│ to LLVM intrinsic call)     │
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│ cir.call_llvm_intrinsic op  │
│ (llvm.x86.foo, concrete     │
│ operand/result types)       │
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐        ┌─────────────────────────────┐
│ intrinsics.rs               │        │  slate-intrinsic-gen        │
│ mangled_link_name /         │        │ (offline, once per LLVM     │
│ find_stdarch_override       │◀───────│ stdarch version — mines     │
│ against intrinsics_table.rs │        │ Intrinsics.h + stdarch      │
│                             │        │ #[link_name] attrs)         │
└──────────────┬──────────────┘        └─────────────────────────────┘
               │
               ▼
┌─────────────────────────────┐
│ extern "unadjusted"         │
│ #[link_name = "llvm.foo"]   │
│ fn decl (shim)              │
└──────────────┬──────────────┘
               │
               ▼
┌─────────────────────────────┐
│ generated Rust fn call site │
│ (compiled + run)            │
└─────────────────────────────┘

Today’s topic is about translating C’s switch to rust’s match, which shockingly has a decent amount of literature about it (unbeknownst to me).

We’ll start out with this example which demonstrates the main features of switch in C:

int score(int x) {
  int out = 0;
  switch (x) {
  case 1:
    out += 10;
  case 2:
    out += 20;
    break;
  case 3:
  case 4:
    out += 40;
    break;
  default:
    out += 90;
  }
  return out;
}

int main(void) {
  printf("%d %d %d %d %d\n", score(1), score(2), score(3), score(4), score(5));
  return 0;
}

• score(1) hits the first branch, where it adds 10 to out. However, there is no break after 1, so it also falls through into case 2, which adds 20 to out. Thus, out will be 30 and printed.
• score(2) increments out by 20 and then breaks out. So it prints out 20.
• score(3) hits the 3 or 4 branch, increments out by 40 and prints.
• score(4) does the same thing, printing 40.
• score(5) will just print out 90, since even without a break it falls through into nothing.

If we were to draw a graph of this, it would look like:

                    ┌─────────────┐
                    │  entry      │
                    │  out = 0    │
                    └──────┬──────┘
                           │
                           ▼
                    ┌─────────────┐
                    │  switch(x)  │
                    └──┬──┬──┬──┬─┘
          x==1         │  │  │  │ default
      ┌────────────────┘  │  │  └────────────────────┐
      │            x==2   │  │ x==3,4                │
      │            ┌──────┘  └────────┐              │
      │            │                  │              │
      ▼            ▼                  ▼              ▼
┌───────────┐   ┌───────────┐    ┌───────────┐ ┌───────────┐
│ case 1:   │   │ case 2:   │    │ case 3:   │ │ default:  │
│ out += 10 │──▶│ out += 20 │    │           │ │ out += 90 │
└───────────┘   └─────┬─────┘    └─────┬─────┘ └─────┬─────┘
                      │                │             │
                      │                ▼             │
                      │          ┌───────────┐       │
                      │          │ case 4:   │       │
                      │          │ out += 40 │       │
                      │          └─────┬─────┘       │
                      │                │             │
                      └──────┬─────────┴─────────────┘
                             ▼
                      ┌─────────────┐
                      │ return out  │
                      └─────────────┘

We could try to convert this to rust but sadly, rust’s match doesn’t have fallthrough. So we’ll have to be a bit more creative.

Recovering Structure

There’s a theorem from the 60s (the structured program theorem) saying any Control Flow Graph (CFG) can be represented with three control flow structures: sequence, selection (if/else), and iteration (a loop over a boolean condition).

There are many ways of recovering structure, so let’s look at three concrete ways of converting switch to match: how C2Rust’s relooper does it, and the classic “folk theorem” construction using a loop and a program counter, and then a bottom up traversal algorithm that generates the best results.

Relooper

C2Rust currently emits this for the switch case in C we looked at, using an algorithm called Relooper:

's_34: {
    match x {
        1 => {
            out += 10;
        }
        2 => {}
        3 | 4 => {
            out += 40;
            break 's_34;
        }
        _ => {
            out += 90;
            break 's_34;
        }
    }
    out += 20;
}

Relooper’s trick is when code is shared by multiple branches but some branches need to skip it, it wraps the branch in a labeled block and has those branches break out of the label before reaching the shared code. So:

• out += 20 (case 2’s body) is hoisted outside and after the match, since it’s the code shared by case 1’s fallthrough and entering case 2 directly.
• Every case that must not run that trailing code (3, 4, default) gets an explicit break 's_34, so it doesn’t run into the out += 20 code.
• Case 1 has no break, so it falls off the end of its arm into the shared trailer, which executes out += 20.
• Case 2’s own arm is deleted, because its body is the shared trailing code.

Basically, Relooper will add an outer loop with break label for every fallthrough case, minus 1. So in this case, let’s count the fallthroughs:

switch (x) {
case -500 ... -1: // 1
  out += 100;
case 0:
case 1: // 2
  out += 1;
case 2 ... 100: // 3
  out += 2;
  if (out > 50) {
    break;
  }
  out += 3;
case 101: // 4
  out += 4;
  if (x % 2 == 0) {
    break;
  }
default: // 5
  out += 5;
case 200 ... 500:
case 600:
case 700 ... 900:
  out += 6;
  break;
case 999: // 6
  out += 7;
case 1000: // 7
  out += 8;
}

Since there are 7 fallthroughs, there will be 6 wrapping loops generated, making this code hard to read:

's_73: {
    'c_720: {
        'c_700: {
            'c_698: {
                'c_695: {
                    'c_704: {
                        match x {
                            -500 => {
                                out += 100 ;
                            }
                            0 | 1 => {}
                            2 => {
                                break 'c_695;
                            }
                            101 => {
                                break 'c_698;
                            }
                            200 | 600 | 700 => {
                                break 'c_720;
                            }
                            999 => {
                                out += 7;
                                break 'c_704;
                            }
                            1000 => {
                                break 'c_704;
                            }
                            _ => {
                                break 'c_700;
                            }
                        }
                        out += 1;
                        break 'c_695;
                    }
                    out += 8;
                    break 's_73;
                }
                out += 2;
                if out > 50 {
                    break 's_73;
                } else {
                    out += 3;
                }
            }
            out += 4;
            if x % 2 == 0 {
                break 's_73;
            }
        }
        out += 5;
    }
    out += 6;
}

The Structured Programming Theorem: Folk version

The folk theorem of the structured programming theorem has a simple kernel: interpret the unstructured part as a subprogram and hand-write a tiny interpreter for it.

We create a value that’s our program counter (in this case, __switch_case0). Since the switch only checks x, we can assign this to x, and map the different case branches (1,2,3,4,default) to one number per distinct case (0,1,2,3,4).

Afterwards, we enter the interpreter and interpret the program. Any node that has a break, say 1, 3, 4 (which line up with the switches that have breaks in the C program break out of the loop immediately. To handle fall through, the cases set __switch_case0 to the next execution. In this case, since we only have fallthrough, it will always be the next block (0 sets __switch_case0 to 1), (2 sets __switch_case0 to 3). This program emulates the C’s switch code perfectly.

{
    let mut x = 1; // what we call the function with, in this case 1
    {
        // the mapping of switch cases to unique blocks:
        let mut __switch_case0: i32 = match x {
            1 => 0,
            2 => 1,
            3 => 2,
            4 => 3,
            _ => 4,
        };
        '__switch0: loop {
            match __switch_case0 {
                0 => {
                    out += 10;
                    // in the fallthrough case, we increment __switch_case by 1
                    // and then continue
                    __switch_case0 = 1;
                    continue '__switch0;
                }
                1 => {
                    out += 20;
                    // for a fallthrough case we generate a break. This
                    // can be elided, but it's left here to be explicit
                    break '__switch0;
                }
                2 => {
                    __switch_case0 = 3;
                    continue '__switch0;
                }
                3 => {
                    out += 40;
                    break '__switch0;
                }
                4 => {
                    out += 90;
                    break '__switch0;
                }
                _ => {
                    break '__switch0;
                }
            }
        }
    }
}

This version is worse than what relooper generates. However, for the 7 fallthrough version, note that we’re guaranteed only one outer loop for our match, because the dispatcher (__switch_case0) at the top handles control flow.

Personally, I find this much more readable than 7 nested levels, so I chose this as the default lowering representation of a switch.

{
    let mut out: i32 = 0;
    {
        let __switch_value0 = ...; // whatever the value is
        let mut __switch_case0: i32 = match __switch_value0 {
            -500..=-1 => 0,
            0 => 1,
            1 => 2,
            2..=100 => 3,
            101 => 4,
            200..=500 => 6,
            600 => 7,
            700..=900 => 8,
            999 => 9,
            1000 => 10,
            _ => 5,
        };
        '__switch0: loop {
            match __switch_case0 {
                0 => {
                    out += 100;
                    __switch_case0 = 1;
                    continue '__switch0;
                }
                1 => {
                    __switch_case0 = 2;
                    continue '__switch0;
                }
                2 => {
                    out += 1;
                    __switch_case0 = 3;
                    continue '__switch0;
                }
                3 => {
                    out += 2;
                    if out > 50 {
                        break '__switch0;
                    }
                    out += 3;
                    __switch_case0 = 4;
                    continue '__switch0;
                }
                4 => {
                    out += 4;
                    if x % 2 == 0 {
                        break '__switch0;
                    }
                    __switch_case0 = 5;
                    continue '__switch0;
                }
                5 => {
                    out += 5;
                    __switch_case0 = 6;
                    continue '__switch0;
                }
                6 => {
                    __switch_case0 = 7;
                    continue '__switch0;
                }
                7 => {
                    __switch_case0 = 8;
                    continue '__switch0;
                }
                8 => {
                    out += 6;
                    break '__switch0;
                }
                9 => {
                    out += 7;
                    __switch_case0 = 10;
                    continue '__switch0;
                }
                10 => {
                    out += 8;
                    break '__switch0;
                }
                _ => {
                    break '__switch0;
                }
            }
        }
    }
}

Now let’s do better. Note that the CFG only goes one direction (down). So each node in the CFG does the following:

1. Executes its block
2. Either (breaks to end or falls through to the next branch)

We can use this to optimize our representation of our match by taking the switch case bottom up. The rules are simple:

• default is translated to _. Keep in mind that default can fallthrough (_ in rust cannot) so we must always move the _ pattern to the end of the switch case and handle fallthrough.
• We traverse the switch case bottom-up, and collect the blocks we’ve seen in reverse order and the labels we’ve seen. On break, for each label we’ll write the blocks we’ve seen in reverse order (since we’re traversing in reverse, this will write out the blocks in the proper order) for each label we’ve seen. Then we’ll clear our cache of blocks and labels.
• We stop when we hit the start of the block and clear our current cache of blocks and labels.

Since this is a straight-line construction, we can prove this with induction:

Invariant: right before we process case i, the buffer holds exactly the code that runs when control jumps into case i + 1’s label and executes until the next break or the end of the switch.

Step: if case i has no break, prepending its block to the buffer gives exactly the code that runs from a jump into case i’s label, so the buffer now satisfies the invariant for i. If case i has a break, the buffer for i is just its own block, since nothing after it is ever reached from a jump to i.

So by induction, the buffer at the moment we flush label L always equals C’s fallthrough semantics for entering at L which is exactly what a match arm needs to contain.

So let’s run through this example:

That gets us this:

match x {
    1 => {
        out += 10;
        out += 20;
    }
    2 => {
        out += 20;
    }
    3 => {
        out += 40;
    }
    4 => {
        out += 40;
    }
    _ => {
        out += 90;
    }
}

We can converge identical blocks, so note 3 and 4 are the same, so the block can be deleted and the label for 3 can be replaced with either | for or or you can use a range (for types that can).

This gets us:

match x {
    1 => {
        out += 10;
        out += 20;
    }
    2 => {
        out += 20;
    }
    3 | 4 => {
        out += 40;
    }
    _ => {
        out += 90;
    }
}

Which is exactly what I would handwrite in a translation.

With a little range coalescing, this also handles the 7 fallthrough case:

match x {
    -500..=-1 => {
        out += 100;
        out += 1;
        out += 2;
        if out > 50 {
        } else {
            out += 3;
            out += 4;
            if x % 2 == 0 {
            } else {
                out += 5;
                out += 6;
            }
        }
    }
    0 | 1 => {
        out += 1;
        out += 2;
        if out > 50 {
        } else {
            out += 3;
            out += 4;
            if x % 2 == 0 {
            } else {
                out += 5;
                out += 6;
            }
        }
    }
    2..=100 => {
        out += 2;
        if out > 50 {
        } else {
            out += 3;
            out += 4;
            if x % 2 == 0 {
            } else {
                out += 5;
                out += 6;
            }
        }
    }
    101 => {
        out += 4;
        if x % 2 == 0 {
        } else {
            out += 5;
            out += 6;
        }
    }
    200..=500 | 600 | 700..=900 => {
        out += 6;
    }
    999 => {
        out += 7;
        out += 8;
    }
    1000 => {
        out += 8;
    }
    _ => {
        out += 5;
        out += 6;
    }
}

Some improvement could be made to hoist the if out > 50 {} else {} out, since there are lots of repeated sub blocks here, but that’s for another time.

Generality ain’t all it’s cracked up to be

Handling the general case is hard. For C2Rust, relooper is the way to go because the switch might have a stray goto out of it, and because its first pass is only transpilation with an optional post-processing pass.

Slate’s first lowering pass is pretty similar (I chose to implement the folk theorem in the lowering phase for its simplicity, even if it has worse output). However, Slate has an advantage in that it runs post-processing, so it can consume lowered output and recover structure from it. Thus, we can apply the bottom up algorithm conservatively and get the ideal match output from the conclusion for almost every switch.

When I first started out on slate, I didn’t link to a custom libc. Why would you, you install a compiler, it’s automatically pointed at the libc you have configured, and you can start compiling. The compiler has everything you need. This works for a single target setup. If you’re compiling code for only your target (I’m on x86_64-unknown-linux-gnu). However, C and Rust are both easily cross compiled for other targets.

There are a series of preprocessor macros that are defined per target, which can enable and disable code. For a very trivial example, here’s a program that prints differently based on if you’re on x86_64 or not:

#include <stdio.h>

int main() {
#if defined(__x86_64__)
    puts("hi from x86_64");
#else
    puts("hi from any other architecture");
#endif
}

When compiling the program, some C compilers like gcc and clang will define the __x86_64__ macro when compiling your code, setting it only for x86_64 systems, so it’s possible to test for it. Because the preprocessor can only choose one block here, the actual programs that the C compiler will see after preprocessing are either:

#include <stdio.h>

int main() {
    puts("hi from x86_64");
}

Or:

#include <stdio.h>

int main() {
    puts("hi from any other architecture");
}

This is fine and dandy. Rust also has a similar mechanism (#[cfg(...)]) blocks that let us do the same thing, so this is semantically the same program:

fn main() {
    if cfg!(target_arch = "x86_64") {
        println!("hi from x86_64");
    } else {
        println!("hi from any other architecture");
    }
}

Now the crux is that we want to take the C program and translate it to Rust. How do we do that? The C preprocessor plays a cruel trick on us – since it only preserves blocks that are enabled for you (either through -D... or through the compiler, you can’t actually see the full program, you’ll only see the program after preprocessing. So, we could say translate to:

fn main() {
    println!("hi from x86_64");
    // or this: println!("hi from any other architecture");
}

But we’d have lost cross compilation.

What’s even worse in a way is that this program is wrong – on a non-x86_64 architecture, you want to see hi from any other architecture, but you’ll see hi from x86_64 regardless of what target you compile for.

One half of the problem is how to splice the preprocessor blocks such that we can recover the original program. That’s for another time. However, C also allows us to configure our build differently based on different flags, which is really useful.

For an example here, I have a macro that I use for setting memory alignment:

#if defined(__STDC_VERSION__) && __STDC_VERSION__ >= 202311L
#define __aligned(X) [[aligned(X)]]
#elif defined(__STDC_VERSION__) && __STDC_VERSION__ >= 201112L
#define __aligned(X) _Alignas(X)
#elif defined(__GNUC__) || defined(__clang__)
#define __aligned(X) __attribute__((aligned(X)))
#elif defined(_MSC_VER)
#define __aligned(X) __declspec(align(X))
#else
#define __aligned(X)
#endif

Different compilers and different versions of the c standard library have different ways of expressing this concept. Since this was non-standard until C11, MSVC and clang chose different ways to express this concept. However, until C11, since this was not in the standard, compilers didn’t need to have a way to express it. In that case, aligned is defined as an empty string.

This is an important part of C, and without support for this, you can only support truly standard C that uses no architecture specific features or compiler features. Thus, to translate C, Slate needs to provide this faculty.

Thinking back to the start of this blog post though, our target only has a standard library for our target. So slate can’t rely on the user’s libc for platform code, and has to provide its own libc. To actually compile a whole matrix of libc code, slate defines target specific macros in its own libc:

#if defined(__SLATE_ARCH_X86_64) + defined(__SLATE_ARCH_X86) +                 \
        defined(__SLATE_ARCH_AARCH64) + defined(__SLATE_ARCH_ARM) +            \
        defined(__SLATE_ARCH_RISCV64) + defined(__SLATE_ARCH_RISCV32) !=       \
#error "Slate requires one supported target architecture."
#endif

Which are passed at build time based on the target that’s requested to be cross compiled:

FixtureFlavor::Macos => &[
    "-D_SLATE_LIBC",
    "-D__SLATE_ARCH_AARCH64",
    "-D__SLATE_VENDOR_APPLE",
    "-D__SLATE_KERNEL_DARWIN",
    "-D__SLATE_PLATFORM_MACOS",
    "-D__SLATE_LIBC_DARWIN",
    "-D__SLATE_OBJ_MACHO",
    "-D__SLATE_WORDSIZE_64",
    "-D__SLATE_ENDIAN_LITTLE",
],

This allows slate’s libc, libc-shim, to be compiled for different targets and still show one consistent libc for rust externs to link to.

the strerror_r incident

Here’s a question for you, dear reader. What’s the signature of strerror_r?

int strerror_r(int errnum, char *buf, size_t buflen);

or is it?

char *strerror_r(int errnum, char *buf, size_t buflen);

The answer is it depends based on which libc you’re using. The first definition is the standard one, and the second one is the gnu extended one.

This would be fine on systems with 32-bit pointers, but nowadays those are rarer, we live in the 64-bit world. So these have different ABIs. It’s not really that bad except for the fact that strerror_r has different behavior in the gnu version than it does for the standard one.

And so any C program translated to Rust on a glibc system vs a musl one where the signatures agree will have diverging behavior, and any where you compile on a 64-bit system will simply not recompile. Wonderful.

qsort_r has different signatures

What’s the signature of qsort_r in freebsd?

void qsort_r(
    void *base,
    size_t nmemb,
    size_t size,
    void *thunk,
    int (*cmp)(void *, const void *, const void *)
);

or is it?

void qsort_r(
    void *base,
    size_t nmemb,
    size_t size,
    int (*cmp)(const void *, const void *, void *),
    void *thunk
);

As you might expect the answer is both of them are valid. The first version is the standard POSIX version, and the second one is the old BSD version, which was different. Since the first version was standardized in 2024, freebsd decided to do a swap. So if you were to compile on a newer version of freebsd and send your transpiled code to an older version (or vice versa), your code would be incorrect. C side this is fine because the linker will choose the correct symbol, but the externs in rust won’t line up if translated through slate.

Both of these problems (and others) can be handled if you control the libc itself – you can link older versions of code for compatibility, or choose the right version to match the target and use #[cfg]s to have cross target and cross version code work just fine.

If you’ve never used long double, you may breathe a sigh of relief. If you use either f64 (double) or f128s (_Float128/__float128) then you’re absolved of sin and free to go. The heathens who wish to learn suffering may stay till the rest of the post to learn more.

long double has been around since C89. The standard just says that it must be as precise as double (f64). Because of this, long double can be pretty much any bit-size from 64-bits to infinite bits. In practice, there are a few dominant ones:

1. f64. MSVC does this for x86.
2. f80. GCC + Clang do this for x86. This is padded to 12-bytes or 16-bytes.
3. double-double. Two f64s glued together. For older PowerPC and IBM architectures.
4. f128. Arm64, PowerPC, RISC-V.

f64 and f128 are not hard to support. On a target where long double == f64/f128, you would just swap out every long double you see for the appropriate type in rust, and when you pass it to extern C code, it’ll be bit aligned and properly handled.

f80 is the real problem, and what the rest of the blog will explain. Afaik, C2Rust doesn’t support long double properly because it translates to f128. There have been efforts in the past to add f80 support, but this’ll probably take a while.

Representing f80

With long double, we need to be able to do anything you would normally be able to do with the type in C. That means the arithmetic ops, casting to and from, passing to functions as value or pointer, and being able to use it in callbacks.

A first try is this:

#[repr(C, align(16))]
#[derive(Clone, Copy)]
pub struct LongDouble { bytes: [u8; 10] };

With this we fulfill the requirements of its size and alignment. We could then even emulate the f80 math itself in rust, so when you call any long double function that has been translated to rust, and call a non-translated C function, the bits are all perfectly legal.

Sadly, long doubles are passed in floating point registers, whereas these chars are passed in general purpose registers, so trying to implement this as is will cause corruption.

We can have the rust type be plain bytes and implement every operation as asm:

pub unsafe fn add(self, rhs: Self) -> Self {
    let mut out = LongDouble { bytes: [0; 10] };

    unsafe {
        core::arch::asm!(
            "fld tbyte ptr [{a}]",
            "fld tbyte ptr [{b}]",
            "faddp st(1), st(0)",
            "fstp tbyte ptr [{out}]",

            a   = in(reg) self.bytes.as_ptr(),
            b   = in(reg) rhs.bytes.as_ptr(),
            out = in(reg) out.bytes.as_mut_ptr(),

            out("st(0)") _,
            out("st(1)") _,
            out("st(2)") _,
            out("st(3)") _,
            out("st(4)") _,
            out("st(5)") _,
            out("st(6)") _,
            out("st(7)") _,
        );
    }

    out
}

This works, but you would have to implement every primitive long double op yourself in handwritten assembly. (If you heard a groan, that was me).

Instead, here’s a nice trick: you can shim the calls and let libc do the rest.

So on the rust side:

#[repr(C, align(16))]
#[derive(Clone, Copy)]
pub struct LongDouble(pub [u8; 10]);

unsafe extern "C" {
    fn __slate_ld_add(a: LongDouble, b: LongDouble) -> LongDouble;
    // whatever other ops you want to provide
}

Use the same storage on the C side:

typedef struct {
    _Alignas(16) unsigned char bytes[10];
} slate_f80;

And then write two functions, one to convert chars to long double and one back:

static inline long double unpack(slate_f80 v) {
    long double x = 0.0L;

    memcpy(&x, v.bytes, 10);
    return x;
}

static inline slate_f80 pack(long double x) {
    slate_f80 v = {0};

    memcpy(v.bytes, &x, 10);
    return v;
}

slate_f80 __slate_ld_add(slate_f80 a, slate_f80 b) { return pack(unpack(a) + unpack(b)); }
slate_f80 __slate_ld_sub(slate_f80 a, slate_f80 b) { return pack(unpack(a) - unpack(b)); }
slate_f80 __slate_ld_mul(slate_f80 a, slate_f80 b) { return pack(unpack(a) * unpack(b)); }
slate_f80 __slate_ld_div(slate_f80 a, slate_f80 b) { return pack(unpack(a) / unpack(b)); }
slate_f80 __slate_ld_neg(slate_f80 a) { return pack(-unpack(a)); }

You could also implement casting, relops in the same way:

int __slate_ld_eq(slate_f80 a, slate_f80 b) { return unpack(a) == unpack(b); }
int32_t __slate_ld_to_i32(slate_f80 a) { return (int32_t)unpack(a); }
slate_f80 __slate_ld_from_i32(int32_t x) { return pack((long double)x); }

Likewise, for any arbitrary function:

long double custom(long double, int);

You would replace it with a call to:

slate_f80 __slate_custom(slate_f80 a, int b) { return pack(custom(unpack(a), b)); }

This makes it so we don’t have to know anything about the underlying long double type at all (no assembly writing for me) and keeps our shim code fairly light.

A couple of years ago, the 1 Billion Row Challenge got popular. The challenge is to write a program that reads a text file with the format of: $CITY_NAME;$TEMP\n of some size, and then for each city, the program should print out the min, mean, and max values per city, alphabetically ordered.

So if you have:

Hamburg;12.0
Hamburg;34.2

You would condense this down to:

Hamburg;12.0;23.1;34.2
   ^      ^    ^    ^
 city    min  mean max

And print that out.

I decided to do this in Rust, and this post is what I learned.

Preliminaries

Since I knew I wanted to try out a few different methods, to start off with I defined a trait so I would only have to rewrite the core logic.

pub type Temp = i16;

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct StationResult {
    pub station: String,
    pub min: Temp,
    pub mean: Temp,
    pub max: Temp,
}

pub trait Solver: Default {
    fn process(&mut self, station: &str, temp: Temp);
    fn process_bytes(&mut self, station: &[u8], temp: Temp) -> io::Result<()> {
        let station = std::str::from_utf8(station).map_err(|error| {
            io::Error::new(
                io::ErrorKind::InvalidData,
                format!("invalid station name UTF-8: {error}"),
            )
        })?;
        self.process(station, temp);
        Ok(())
    }

    fn finish(self) -> Vec<StationResult>;
}

As well, I needed to keep the stats for each station:

pub type Temp = i16;
pub type TempSum = i64;
pub type Count = u32;

#[derive(Default, Clone, Copy)]
pub(crate) struct Stats {
    pub(crate) sum: TempSum,
    pub(crate) count: Count,
    pub(crate) min: Temp,
    pub(crate) max: Temp,
}

impl Stats {
    pub(crate) fn add(&mut self, temp: Temp) {
        if self.count == 0 {
            self.min = temp;
            self.max = temp;
        } else {
            self.min = self.min.min(temp);
            self.max = self.max.max(temp);
        }

        self.sum += TempSum::from(temp);
        self.count += 1;
    }

    pub(crate) fn into_result(self, station: String) -> StationResult {
        StationResult {
            station,
            min: self.min,
            mean: round_mean_tenths(self.sum, self.count),
            max: self.max,
        }
    }
}

With a helper for reading input:

pub fn process_measurements<R, S>(reader: R, solver: &mut S) -> io::Result<()>
where
    R: BufRead,
    S: Solver,
{
    for line in reader.lines() {
        let line = line?;
        let Some((station, temp)) = line.split_once(';') else {
            continue;
        };

        solver.process(station, parse_temperature_tenths(temp)?);
    }

    Ok(())
}

A helper for writing output:

pub fn write_results_to_path(path: Option<&str>, results: &[StationResult]) -> io::Result<()> {
    match path {
        Some(path) => {
            let file = File::create(path)?;
            let writer = BufWriter::new(file);
            write_results(writer, results)
        }
        None => {
            let stdout = io::stdout();
            let writer = stdout.lock();
            write_results(writer, results)
        }
    }
}

And finally, running the solver:

pub fn run_solver<S>(input_path: &str, output_path: Option<&str>) -> io::Result<()>
where
    S: Solver,
{
    let file = File::open(input_path).map_err(|error| not_found_error(input_path, error))?;
    let reader = BufReader::with_capacity(8 * 1024 * 1024, file);
    let mut solver = S::default();

    input::process_measurements(reader, &mut solver)?;
    output::write_results_to_path(output_path, &solver.finish())
}

As well, I wanted to create a couple of things. First off, I needed cargo flamegraph to see where time was being spent, and I wrote a script so I could generate input of different sizes. The 1 billion rows would be wasteful to do my target runs on, so I took the weather data from the one billion rows repo and generated a million row sample to run on.

The Easiest Solution (BTreeMap)

First off, let’s start off with the basics. The simplest solution I could think of was to:

1. For each line, split at the ‘;’ character.
   • The left side is the station name.
   • The right side is the temperature.
   • Take the temperature, and then record the current total, current count, current minimum, and current maximum.
2. Sort the cities.
3. Calculate the mean by doing sum / count;
4. Print out “{city};{min};{mean};{max}” for each city.

You know what’s better than sorting at the end? Having a data structure do it for you. So at first, I used a BTreeMap. With the Solver trait, I just needed to implement process and finish.

#[derive(Default)]
pub struct BasicBTreeMapSolver {
    stats_by_station: BTreeMap<String, Stats>,
}

impl Solver for BasicBTreeMapSolver {
    fn process(&mut self, station: &str, temp: Temp) {
        self.stats_by_station
            .entry(station.to_string())
            .or_default()
            .add(temp);
    }

    fn finish(self) -> Vec<StationResult> {
        self.stats_by_station
            .into_iter()
            .map(|(station, stats)| stats.into_result(station))
            .collect()
    }
}

This solution crunches through one million rows in 218ms on my Ryzen 5 7600 CPU.

Not bad but we could do better. Taking a look at the flamegraph here, there’s not too much to say, but it looks like the btreemap logic itself is taking the longest amount of time, so we should tackle that first.

The Next Solution (HashMap)

BTreeMap is slow for this solution because we don’t need a sorted order all the time, just at the end. To do that, we can fall back to good old HashMap and sort once at the end.

And with that:

impl Solver for BasicHashMapSolver {
    fn process(&mut self, station: &str, temp: Temp) {
        if let Some(stats) = self.stats_by_station.get_mut(station) {
            stats.add(temp);
        } else {
            let mut stats = Stats::default();
            stats.add(temp);
            self.stats_by_station.insert(station.to_string(), stats);
        }
    }

    fn finish(self) -> Vec<StationResult> {
        let mut results: Vec<_> = self
            .stats_by_station
            .into_iter()
            .map(|(station, stats)| stats.into_result(station))
            .collect();

        results.sort_unstable_by(|left, right| left.station.cmp(&right.station));
        results
    }
}

This about halves the run time, to 110ms. I was expecting more of a modest improvement, but this is pretty nice. The next chunk I see taking a lot of time is read_line, so we can work on that next.

Memory Mapping

The next bottleneck looked to be I/O: BufReader was taking the most amount of time. The Bufreader already reads in chunks, but we want to avoid going back to the kernel for reads. Memory mapping is one way to do that.

All we have to do is change our input hook to memory map our input file.

pub fn run_solver_mmap<S: Solver>(input_path: &str, output_path: Option<&str>) -> io::Result<()> {
    let file = File::open(input_path)?;
    let mmap = unsafe { MmapOptions::new().map(&file)? };
    let mut solver = S::default();
    input::process_measurements_bytes(&mmap, &mut solver)?;
    output::write_results_to_path(output_path, &solver.finish())
}

This gets us an even faster runtime of 79ms.

More changes

At this point I was pretty satisfied. Memory mapping the input file + using a HashMap was already pretty good. But the flamegraph does show that parse_temperature_tenths_bytes is pretty slow (32% of runtime), so let’s dig into that.

The current implementation is:

fn parse_temperature_tenths_bytes(value: &[u8]) -> io::Result<Temp> {
    let negative = value.first() == Some(&b'-');
    let digits = if negative { &value[1..] } else { value };
                                                                                                                              
    if digits.len() < 3 || digits[digits.len() - 2] != b'.' {
        return Err(invalid_temperature_bytes(value));
    }
                                                                                                                       
    let whole = &digits[..digits.len() - 2];
    let fraction = digits[digits.len() - 1];
    if whole.is_empty()
        || !whole.iter().all(|byte| byte.is_ascii_digit())
        || !fraction.is_ascii_digit()
    {
        return Err(invalid_temperature_bytes(value));
    }
                                                                                                                       
    let mut temp = Temp::from(fraction - b'0');
    let mut multiplier = 10;
    for &digit in whole.iter().rev() {
        temp += Temp::from(digit - b'0') * multiplier;
        multiplier *= 10;
    }
                                                                                                                       
    Ok(if negative { -temp } else { temp })
}

This has a few branches and error checking. We can make this branchless by exploiting the fact that there are only 4 formats. We just need to check if the first character is the negative sign, then grab the fractional part from the end, and otherwise grab the number itself. We can then recreate the value itself that way, fully branchless.

fn parse_temperature_tenths_bytes(value: &[u8]) -> io::Result<Temp> {
    let neg = (value[0] == b'-') as usize;
    let frac = (value[value.len() - 1] - b'0') as Temp;
    let ones = (value[value.len() - 3] - b'0') as Temp;
    let has_tens = (value.len() - neg > 3) as Temp;
    let tens = has_tens * (value[neg] - b'0') as Temp;
    let sign = 1 - 2 * neg as Temp;
    Ok(sign * (tens * 100 + ones * 10 + frac))
}

This has two advantages: one, dropping the runtime to 72ms, a nice improvement, but also dropping the variance between runs. We used to have about 5-8ms variance, but now this is less than <1ms between runs, since branchless solutions are more consistent.

Handle the bytes

Why bother having strings at all? Why not index a city by its bytes? We can do that and combined with a faster hashmap (Fnv), I got a 60ms runtime on my computer.

pub struct FnvBytesSolver {
    stats_by_station: FnvHashMap<Vec<u8>, Stats>,
}

impl Solver for FnvBytesSolver {
    fn process_bytes(&mut self, station: &[u8], temp: Temp) -> io::Result<()> {
        if let Some(stats) = self.stats_by_station.get_mut(station) {
            stats.add(temp);
        } else {
            let mut stats = Stats::default();
            stats.add(temp);
            self.stats_by_station.insert(station.to_vec(), stats);
        }
        Ok(())
    }

    fn finish(self) -> Vec<StationResult> {
        let mut results = Vec::with_capacity(self.stats_by_station.len());
        for (station, stats) in self.stats_by_station {
            let station = String::from_utf8(station).expect("valid UTF-8");
            results.push(stats.into_result(station));
        }
        results.sort_unstable_by(|l, r| l.station.cmp(&r.station));
        results
    }
}

A Dead end

At this point, I had about hit wits end. About 48% of runtime was in FnvHasher’s get_mut, and otherwise about 17% of runtime was in float_to_decimal_common_exact.

So the next two things are to remove the hashmap and fix the output formatting.

The biggest impact would be to rip out the hashmap itself and come up with a custom flat hashmap.

fn fingerprint(key: &[u8]) -> u64 {
    let mut buf = [0u8; 8];
    let n = key.len().min(8);
    buf[..n].copy_from_slice(&key[..n]);
    u64::from_le_bytes(buf)
}

fn table_index(fp: u64, key_len: usize) -> usize {
    let h = fp ^ (key_len as u64).wrapping_shl(17);
    let h = h ^ (h >> 30);
    let h = h.wrapping_mul(0xbf58476d1ce4e5b9);
    let h = h ^ (h >> 27);
    let h = h.wrapping_mul(0x94d049bb133111eb);
    let h = h ^ (h >> 31);
    (h as usize) & TABLE_MASK
}

const TABLE_SIZE: usize = 1 << 17; 

struct Slot<'a> {
    fingerprint: u64,
    key: &'a [u8],
    stats: Stats,
}

struct FlatTable<'a> {
    slots: Box<[Slot<'a>]>,
}

impl<'a> FlatTable<'a> {
    fn update(&mut self, key: &'a [u8], temp: Temp) {
        let fp = fingerprint(key);
        let mut idx = table_index(fp, key.len());
        loop {
            let slot = &mut self.slots[idx];
            if slot.key.is_empty() {               
                slot.fingerprint = fp;
                slot.key = key;
                slot.stats = Stats::default();
                slot.stats.add(temp);
                return;
            }
            if slot.fingerprint == fp && slot.key == key {
                slot.stats.add(temp);
                return;
            }
            idx = (idx + 1) & TABLE_MASK; 
        }
    }
}

The second thing is to fix up the output formatting:

pub fn format_temperature_tenths(temp: Temp) -> String {
    format!("{:.1}", temp as f64 / 10.0)
}

Into this: avoids the 17% of runtime in outputting the float:

pub fn format_temperature_tenths(temp: Temp) -> String {
    let neg = temp < 0;
    let abs = temp.unsigned_abs();
    if neg {
        format!("-{}.{}", abs / 10, abs % 10)
    } else {
        format!("{}.{}", abs / 10, abs % 10)
    }
}

With both of those fixes, we get a dramatic improvement to 36ms runtime.

Parallelization

After this, it’s pretty unclear how to improve this single-threaded solution any more. I decided to drop –no-inline from my cargo flamegraphs since it said 90% of runtime was in process_bytes and the rest was inlined into it. But looking at the flamegraph while counting inlining, it seems like it’s doing about the minimal amount of work required.

A simple way to use parallelism is to cut the input into equal sized chunks by the number of threads, and then join at the end.

To chunk:

fn chunks_for_threads(bytes: &[u8], threads: usize) -> Vec<(usize, usize)> {
    let len = bytes.len();
    let threads = threads.min(len);
    let mut chunks = Vec::with_capacity(threads);
    let mut start = 0;
    for index in 0..threads {
        if start >= len { break; }
        let end = if index + 1 == threads {
            len
        } else {
            align_end_to_line(bytes, len * (index + 1) / threads)
        };
        if start < end { chunks.push((start, end)); }
        start = end;
    }
    chunks
}

fn align_end_to_line(bytes: &[u8], pos: usize) -> usize {
    memchr(b'\n', &bytes[pos..]).map_or(bytes.len(), |offset| pos + offset + 1)
}

To run in parallel:

pub fn run_parallel_borrowed_mmap(input_path: &str, output_path: Option<&str>, threads: usize) -> io::Result<()> {
    let file = File::open(input_path)?;
    let mmap = unsafe { MmapOptions::new().map(&file)? };
    let chunks = chunks_for_threads(&mmap, threads.max(1));

    let local_tables = thread::scope(|scope| {
        let handles: Vec<_> = chunks.iter().map(|&(start, end)| {
            let bytes = &mmap[start..end];
            scope.spawn(move || {
                let mut table = FlatTable::new();
                process_bytes(bytes, &mut table);
                table
            })
        }).collect();

        handles.into_iter()
            .map(|h| h.join().expect("worker thread panicked"))
            .collect::<Vec<_>>()
    });

    let mut table = FlatTable::new();
    for local in local_tables {
        table.merge_from(&local);
    }
    // sort and write
}

To merge:

fn merge_from(&mut self, other: &FlatTable<'a>) {
    for &slot in other.slots.iter() {
        if slot.key.is_empty() { continue; }
        let mut idx = table_index(slot.fingerprint, slot.key.len());
        loop {
            let target = &mut self.slots[idx];
            if target.key.is_empty() {
                *target = slot;
                break;
            }
            if target.fingerprint == slot.fingerprint && target.key == slot.key {
                target.stats.sum += slot.stats.sum;
                target.stats.count += slot.stats.count;
                target.stats.min = target.stats.min.min(slot.stats.min);
                target.stats.max = target.stats.max.max(slot.stats.max);
                break;
            }
            idx = (idx + 1) & TABLE_MASK;
        }
    }
}

With that, we get a runtime of about 21ms.

Conclusion

In the end, we started out with about a 220ms runtime and got it down all the way to 21ms, about a 10x improvement. The first improvement of just using a hashmap did most of the work, down to 110ms, and each respective improvement took more and more work to do for less and less gain.

Cargo flamegraph made it easy to find hotspots, but it doesn’t tell you how to improve your solution – that requires hard thought, but it’s useful to figure out where there’s room to improve.

Let’s start off with a u128 type. That would look something like this:

#[derive(Debug, Clone, Copy)]
pub struct Uint128 {
    pub h: u64, 
    pub l: u64, 
}

For addition, subtraction, and multiplication thankfully nightly rust has overflowing_*, widening_* and wrapping_* methods on integers that allow us a simple implementation.

impl std::ops::Add for Uint128 {
    type Output = Self;

    #[inline(never)]
    fn add(self, rhs: Self) -> Self::Output {
        let (l, carry) = self.l.overflowing_add(rhs.l);
        let h = self.h.wrapping_add(rhs.h).wrapping_add(carry as u64);

        Self { l, h }
    }
}

impl std::ops::Sub for Uint128 {
    type Output = Self;

    #[inline(never)]
    fn sub(self, rhs: Self) -> Self::Output {
        let (l, borrow) = self.l.overflowing_sub(rhs.l);
        let h = self.h.wrapping_sub(rhs.h).wrapping_sub(borrow as u64);

        Self { l, h }
    }
}

impl std::ops::Mul for Uint128 {
    fn mul(self, rhs: Self) -> Self::Output {
        let (p0_lo, p0_hi) = self.l.widening_mul(rhs.l);

        let t1_lo = self.l.wrapping_mul(rhs.h);
        let t2_lo = self.h.wrapping_mul(rhs.l);
        let h = p0_hi.wrapping_add(t1_lo).wrapping_add(t2_lo);
        Self { h, l: p0_lo }
    }
}

For division we have to handwrite and lean on the u128 type, but we get a pretty nice result in the end:

impl std::ops::Div for Uint128 {
    fn div(self, rhs: Self) -> Self::Output {
        let n = (self.h as u128) << 64 | self.l as u128;
        let d = (rhs.h as u128) << 64 | rhs.l as u128;
        let q = n / d;
        Self {
            l: q as u64,
            h: (q >> 64) as u64,
        }
    }
}

The assembly delegates to a compiler builtin called __udivti3, so our code is pretty much loads that up and all the hairy work is done for us.

<bigints::u128::Uint128 as core::ops::arith::Div>::div:
    push rax
    mov rax, rdx
    or rax, rcx
    je .LBB_2
    call qword ptr [rip + __udivti3@GOTPCREL]
    pop rcx
    ret
.LBB_2:
    lea rdi, [rip + .Lanon.12]
    call qword ptr [rip + core::panicking::panic_const::panic_const_div_by_zero@GOTPCREL]

And that’s it, right?

Sadly I wrote a bug in the beginning for my platform (x86_64). The bigints struct has to flip its members since this is a little-endian platform.

#[derive(Debug, Clone, Copy)]
pub struct Uint128 {
    pub l: u64, 
    pub h: u64, 
}

Otherwise the multiplication incurs two extra movs at the start of the function to load the members into the right place.

mov rax, rdx
mov rdx, rcx
mulx r8, rdx, rsi
imul rax, rsi
imul rdi, rcx
add rax, rdi
add rax, r8

If you write the struct the other way, i.e:

#[derive(Debug, Clone, Copy)]
#[cfg(target_endian = "little")]
pub struct Uint128 {
    pub l: u64, // bits 0-63 (lower address)
    pub h: u64, // bits 64-127 (higher address)
}

#[derive(Debug, Clone, Copy)]
#[cfg(target_endian = "big")]
pub struct Uint128 {
    pub h: u64, // bits 64-127 (lower address)
    pub l: u64, // bits 0-63 (higher address)
}

Then the two movs disappear from the x86_64 generated codegen.

mulx r8, rdx, rsi
imul rax, rsi
imul rdi, rcx
add rax, rdi
add rax, r8

Now I needed a way to make sure all my operations were closer to native, so I wrote up a test harness that used cargo-show-asm to check the generated assembly on a variety of platforms (including a big-endian one, s390x) to make sure that there were no unnecessary moves being produced.

I used this and compared my implementations to native to see if they were being properly optimized and found out that for aarch64, using overflowing_sub and wrapping_sub nets this:

<bigints::u128::Uint128 as core::ops::arith::Sub>::sub:                                                                                                                                                                                                                                                                                                                                
    subs x0, x0, x2                                                                                                                                                                                                                                                                                                                                                                    
    sub x8, x1, x3                                                                                                                                                                                                                                                                                                                                                                     
    cset w9, lo                                                                                                                                                                                                                                                                                                                                                                        
    sub x1, x8, x9                                                                                                                                                                                                                                                                                                                                                                     
    ret                                                                                                                                                                                                                                                                                                                                                                                

Whereas the native version looks like so:

<bigints::u128::Uint128 as core::ops::arith::Sub>::sub:                                                                                                                                                                                                                                                                                                                                
  subs x0, x0, x2
  sbc  x1, x1, x3
  ret

While for x86_64, it’s properly optimized:

<bigints::u128::Uint128 as core::ops::arith::Sub>::sub:
    mov rax, rdi
    sub rax, rdx
    sbb rsi, rcx
    mov rdx, rsi
    ret

As well, for multiplication, there’s an extra instruction due to scheduling:

<bigints::u128::Uint128 as core::ops::arith::Mul>::mul:
    mul x9, x2, x1
    mul x8, x2, x0
    umulh x10, x2, x0
    madd x9, x3, x0, x9
    mov x0, x8
    add x1, x9, x10
    ret

<bigints::u128::Uint128 as core::ops::arith::Mul>::mul:
    umulh x10, x0, x2    
    mul x9, x1, x2       
    madd x9, x3, x0, x9  
    mul x0, x0, x2       
    add x1, x9, x10      
    ret

I assume this is an LLVM backend issue somewhere, but it’s hard to fault them, I doubt anyone would write this code anyway since the u128 types exist.

The resulting code is here: Bigints

Now the actual plan was as follows: write an assembly language spec, write an interpreter, and write an encoder/decoder (so I could test out how different design decisions affected code density in the binary). My first order of business was to decide on the size of my opcodes. I decided to go with 16-bit like the m68k – I thought it would be spartan enough to not endlessly iterate and be compact yet give me enough space to design instructions. My first decision was to reverse one design decision I think hampered the m68k, splitting registers into address and data registers.

On the m68k, there are 16 32-bit registers. This is for a CPU that came out in 1979; compare it to a similar Intel CPU at the time, the 8086. The 8086 came with 8 16-bit registers. You get 2 times the registers with 2 times the bit-width. Sadly, with the data register/address register split, the 8 data registers store the usual data and the 8 address registers are for indirect accessing of memory. I assume this was chosen because this made registers take 3-bits (2^3 = 8) instead of 4-bits (2^4 = 16), and because the m68k had support for 8 effective addressing modes on its two operands, which took up 6 bits in total (EA + Reg). Given a 4-bit group, you’d be left with another 3 bits for the register for a binary operator, or to do what you wanted.

I wanted to see if it was possible to support 16 registers, so I bit the 4-bit bullet per register. Given that an instruction set needs to have at least 30 must have instructions, you also need to reserve at least 5 bits for every instruction so you know which opcode to use. I decided to go with a scheme that would allow me to have up to 64 instructions – a 4-bit major group, with a further 2-bits subdividing each group with 4 instructions each, so I would have 16 groups with 4 subgroups in each group. This left me with 10-bits. You see the problem. If each register costs you 4-bits, and you have 2 of them, you’re left with 2 bits per instruction. But wait, there’s more. Each instruction must also carry bits for its size. I wanted 16 64-bit registers, so each instruction had to support an 8-bit, 16-bit, 32-bit, and 64-bit variant (I called these byte, short, long, word). This is 4 choices, so this takes up 2-bits.

So there you have it. Each binary instruction has 4-bits for group, 4-bits for subgroup, 2-bits for size, 4-bits for register 1 (which is the target) and 4-bits for register 2. This works out for most instructions, like arithmetic ones, but you cannot do an arithmetic operation + change the addressing mode.

To do so, I added three instructions as full 12-bit instructions. lea, (load effective address), a load and store instruction. These took up 4 instructions a pop (since they take up a whole group), but I was able to cram in a 2-bit effective address for each of these.

The effective addresses are:

1. register direct, as is: %rX
2. register indirect, the address of the register, (%rX)
3. immediate, $10
4. offset indirect -4(%sp)

Since you can only place this on either the src or the target, this is a bit barebones, but it works out fine.

I decided not to go with a condition flag design (like m68k, arm, x86) and take a flagless design – in exchange though, you have to implement a lot of branching operations. For the branching instructions:

1. Branch if Equal
2. Branch if not equal
3. Branch less than
4. Branch less than equal
5. Branch if Zero
6. Branch if not zero

I could’ve gone with a sparser set, but I figured it’d be ok to use the instructions since branching is quite commonplace. For the cmp side, I decided to go with a sparser set, explicitly requiring a not or a flip of operands to get <=, >, and >=.

1. Cmp equal
2. Cmp not equal
3. Cmp less than equal
4. Cmp less than

I also wanted compact operations that used immediates a lot, so I provided immediate verisons of add, sub, mul, mov. If I dropped the size, I could have 6-bits left over. So I can have immediates from 0..63. However, I really wanted to be able to use 64. It turns out you actually can support the range of 0..64 for these ops:

For add and sub with 0, this does nothing, so you can read this code and emit a no-op instead. For mul and mov, it zeroes out the target register. You can do the same with xor %rX, %rX, which zeroes out the target register. Thus, in the textual assembly format, You can support 65 values (which requires 7-bits) by rewriting the last case to something supported. You can also do this with div (rewrite div %rX, $0 with trap), or mod as well.

I decided to add a few extra assembly directives:

• .byte to include a byte array,
• .ascii to include an ascii string,
• .asciz to include an ascii string with a null terminator
• .include to include the file copy pasted as in C, to link multiple assembly files together.

After hooking up syscall write on linux, you can write assembly pretty well:

        movi %r0, $1         # syscall write 
        movi %r1, $1         # stdout
        load.l %r2, msg      # r2 = &msg
        movi %r3, $12        # length
        trap                 # perform write
        ret

msg:
        .asciz "hello world\n"

And recursive fibonacci isn’t so bad:

# Calculate fib(10)
        movi %r1, $10        # n = 10
        call fib
        jmp done

fib:
        # Prologue: save caller-saved registers
        push.w %r2
        push.w %r3

        # Base case: if n == 0, return 0
        brz.l %r1, base_case

        # Base case: if n == 1, return 1
        mov.l %r3, %r1
        subi.l %r3, $1
        brz.l %r3, base_case

        # Recursive case: fib(n-1)
        subi  %r1, $1
        push.w %r1            # Save n-1
        call fib
        mov.l %r2, %r0        # r2 = fib(n-1)
        pop.w %r1             # Restore n-1

        # fib(n-2)
        subi.l %r1, $1
        push.w %r1
        call fib
        mov.l %r3, %r0        # r3 = fib(n-2)
        pop.w %r1

        # Return fib(n-1) + fib(n-2)
        mov.l %r0, %r2
        add.l %r0, %r3
        jmp end_fib

base_case:
        mov.l %r0, %r1        # Return n

end_fib:
        # Epilogue: restore registers
        pop.w %r3
        pop.w %r2
        ret

done:
        nop

I really enjoyed this project – it was interesting to design an assembly language, since it made me value what instructions could be supported with such little space. I still won’t be writing assembly, but at least I can understand why it’s so hard to design a good ISA!

First things first, we had to compile some m68k ELF binaries. I fired up my copy of crosstool and made myself an m68k-unknown-linux-gnu-gcc toolchain.

To decode a binary, you have to first read the relevant ELF sections of a binary. To do so, I used the goblin library, which took care of that. The rest was to decode each instruction, and then execute it on a virtual CPU.

So, I started writing out a file with a new m68k assembly instruction, compiled it, and then tried to decode it with my program. This would fail, so I would go to work decoding it.

I did this until I implemented the whole m68000 set, which wasn’t too bad, given that the m68k instruction set is pretty compact. After decoding, I had to write a CPU that would emulate everything, and I was good to go. This took a bit longer than plain decoding, but after I got this to work out. Perfect! A fun little weekend project, right?

Unfortunately I wanted more. And this is where the pain really started. I wanted to compile C code for the m68000 and run it on my little interpreter. Couldn’t be so bad, right?

Unfortunately not. I wrote cat in c and then compiled it and tried to run it on my interpreter – decoding error. It turned out that the ELF binary had smuggled in an fmove (floating point register move). Why does cat, a program which does not have any floating point usage, need to move a floating point register?

No matter what flags you use to compile your code, glibc for the m68k target actually is compiled with floating point support. Thus, it’s not possible to use glibc as your libc for compiling m68000. Bummer. Sadly this took me a day of searching to find out why floating point registers were being smuggled into my user code. The -mcpu=X flag was useless, even using -msoft-float was useless. Anyway, I decided to go back to crosstool and try to compile a musl toolchain. Musl would never betray me, would it?

Sadly it too requires floating point registers, and thus does not support the 68000, on setjmp, to clear registers. I was losing hope. But crosstool supports one last libc. uclibc-ng. And uclibc delivered. It actually did support non-floating point m68k and we were in business. I compiled for the m68020 to play it safe (each compiler takes about 30 minutes to compile on my computer), and was back at my interpreter, ready to compile some C.

To get cat to work though, I needed to do some syscall passthrough. I needed to map the m68000’s pointers from guest to host, and also map the syscall numbers from m68k linux to x86_64 linux. But once that was done, I could finally write C, compile to m68000 linux elf, and run it on my x86_64 linux elf machine.

Of course, there was a bit more to do – I had to implement the rest of the m68010 and m68020 instruction set, which I discovered I didn’t complete by using fuzzing to generate random C programs with csmith, so I implemented those instructions, and was off to the races – all I had to do was implement syscalls.

In the end, I decided to translate about 200 or so syscalls, wrote tests for them, and tested them out against qemu-m68k-static for compatibility. There’s plenty more syscalls left to do, but I figured these would be a good chunk of syscalls, so I was done with that. Maybe I’ll come back and implement the rest of them, who knows. The remaining ones are really tough, so I skipped most of those.

You could write code for these emulators in assembly or C, due to there being a gcc toolchain for them: https://tho-otto.de/crossmint.php. I downloaded gcc, hooked up binutils and libraries, and started to write some code.

I went to go write some simple coreutils for the emulator, but compiling my first binary (rm), my binary was >100KB. Given that the atari ST was supposed to run on 1.44MB floppies, eating up a little over 1/15th of your storage on a 50-line C program is a little absurd.

It turns out that the libc that the Atari mint toolchain ships with links to a library called mintlib, which is a pretty complete libc, but its binaries are huge. There’s an alternative libc which isn’t as complete, called libcmini. I linked to that libc instead and saw the familiar file size of 4KB. Nice.

However, the binaries are still pretty large if you think about it. An assembly version of this would probably fit in a few hundred bytes, so C in this case does have a pretty large penalty.

I decided to start writing my own minimalistic libc and incrementally build out some parts of it. The first part was the C runtime, crt0.S. Afterwards, writing initializers and finalizers, setting up the stack, env vars, and jumping to start. All of this was pretty tricky but doable. Some interesting bits were the system calls: mintbind.h where, given this isn’t a Linux OS, a lot of the system calls are pretty interesting – there’s even a “malloc” so you can delegate memory management to the kernel.

While writing the libc was fun to see what was under the hood, I wanted some higher-level code. So I decided I wanted to write Rust. First things first – I downloaded the a.out toolchain from Thorsten Otto’s website. I needed the ELF one, since that was what rust’s toolchain supported. Next, I had to write up a target.json file, describing the properties of the architecture I was targeting.

{
  "arch": "m68k",
  "atomic-cas": false,
  "cpu": "M68040",
  "crt-objects-fallback": "false",
  "crt-static-default": false,
  "crt-static-respected": false,
  "code-model": "large",
  "data-layout": "E-m:e-p:32:16:32-i8:8:8-i16:16:16-i32:16:32-n8:16:32-a:0:16-S16",
  "dynamic-linking": true,
  "eh-frame-header": false,
  "executables": true,
  "has-rpath": true,
  "has-thread-local": true,
  "linker": "m68k-atari-mintelf-gcc",
  "llvm-target": "m68k-unknown-elf",
  "linker-is-gnu": true,
  "linker-flavor": "gcc",
  "pre-link-args": {
    "gcc": [
      "-nostartfiles",
      "-nostdlib",
      "-m68000",
      "-no-pie"
    ]
  },
  "late-link-args": {
    "gcc": [
      "-Wl,--gc-sections",
      "/home/takashi/current/mintbox/build/objs/crt0.o",
      "-Wl,--start-group",
      "-L/home/takashi/current/mintbox/build",
      "-lcbox",
      "-Wl,--end-group",
      "-lgcc"
    ]
  },
  "max-atomic-width": 32,
  "os": "mint",
  "panic-strategy": "abort",
  "position-independent-executables": false,
  "relocation-model": "static",
  "static-position-independent-executables": false,
  "target-endian": "big",
  "target-mcount": "_mcount",
  "target-pointer-width": 32,
  "vendor": "atari"
}

After quite a bit of fumbling around, reading docs online, reading the forums I was able to compile some code for rust. However, I ran into the same problem where my binaries were 100KB. I couldn’t link to libcmini, however, since I wanted to link to alloc in my no_std Rust code. To do so, I needed an allocator, which I wanted to delegate to the libc for. However, this requires posix_memalign, which libcmini does not provide. I implemented it in my libc, and was off to the races.

However, Rust turned out to be a pain in a very different way than writing C. If you try to implement printing with Rust’s fmt helpers when you try to print a type that can’t be “inlined” by the compiler, it’ll pull in compiler builtins, and bundle a 300KB runtime, bloating your binary by that much. If you for example, try to debug print an integer, that’s 300KB. There might be some workaround by using args.as_str() and failing, but this didn’t work in my case. So I decided to stop here. But it was a fun experiment.

Nowadays, on linux, it’s recommended to use systemd to manage your daemons. sshd and other services are registered through systemd. We’re going to go old school in this post for fun.

A daemon shouldn’t be reachable from the outside world, and it should do a task on some activity or change. You can run one or as many of them as you want to – maybe you want just one like a singleton, or many splitting up a task, or you can emulate a single writer many readers pattern.

Regardless, here’s the checklist of “how to daemon”

1. Fork + exit from the parent.
2. setsid() after the first fork to get a new progress group id.
3. disable SIGHUP to prevent the terminal from killing the daemon.
4. Fork again to make sure the daemon process cannot get a terminal.
5. umask(027) to set permissions for files it owns
6. chdir() to the directory you want
7. Close all FDs that aren’t 0/1/2, and redirect 0/1/2 to log files.
8. Get the pidfile, write your pid, keep the file open while alive.
9. Install signal handling, SIGTERM, SIGINT for shutdown, SIGHUP for reloading.
10. Open any files required for the task.
11. Do the task.
12. Install code on exit, to close resources used + unlink the pidfile.

That’s a lot of work. Luckily, the bsd folks have us (the linux folks) covered. They have libdaemon, which has a set of helpers to write daemons. If you choose a higher level language like Rust or Python, setting up a daemon is 15 lines of code.

In C, the setup code is closer to 40 lines, but it’s not that bad regardless with libdaemon – it sets up steps 1 to 7 for us. That leaves just a few steps. We set the path for the pidfile, and register it with daemon_pid_file_proc for it to be set up in the daemon_pid_file_create call later, returning if we couldn’t create it.

Next, there’s signal handlers installed for SIGTERM and SIGINT, and then we do our main loop.

Finally, we’ll do our work, which in this case, reads the request_path, checks how many bytes it has, and then every 5 seconds, logs that to another file.

Pretty simple, but does the trick in showcasing how a daemon works.

First, we have our includes.

#include <errno.h>
#include <fcntl.h>
#include <libdaemon/daemon.h>
#include <libdaemon/dpid.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>

Next, our actual work, which will print to the fd we provide.

static void log_file_size(const char *request_path, int result_fd) {
  struct stat st;
  if (stat(request_path, &st) < 0) {
    if (errno == ENOENT)
      dprintf(result_fd, "# %s missing\n", request_path);
    else
      dprintf(result_fd, "# error reading %s: %s\n", request_path,
              strerror(errno));
    return;
  }
  dprintf(result_fd, "# %s has %ld bytes\n", request_path, st.st_size);
}

Next, setting up the variables we need for libdaemon.

static volatile sig_atomic_t stop_flag = 0;
static void on_stop(int _sig) { stop_flag = 1; }

static const char *pidfile_path = "/tmp/byte_count_daemon.pid";
static const char *request_path = "/tmp/byte_count_requests.txt";
static const char *result_path = "/tmp/byte_count_results.txt";

static const char *pidfile_proc(void) { return pidfile_path; }

And then the main function:

1. You can pass it a -f flag as the first argument, if you want to run in the foreground for testing.
2. Next, the daemon identifier comes from the name of the binary, and then the pidfile path is set afterwards.
3. The daemon call is made only if it’s not foregrounded – so we can kill the process.
4. Signal handlers are registered.
5. Pidfile is created – if it can’t be exclusively owned, return here.
6. Open up an fd we need exclusively to write our results to. If we can’t, we return.
7. The main loop, along with sleeping for 5 seconds between tasks.
8. Stopping and cleaning up.

int main(int argc, char *argv[]) {
  // 1.
  int foreground = (argc > 1 && strcmp(argv[1], "-f") == 0);

  // 2.
  daemon_pid_file_ident = daemon_ident_from_argv0(argv[0]);
  daemon_pid_file_proc = pidfile_proc;

  // 3.
  if (!foreground) {
    if (daemon(0, 0) < 0)
      return 1;
  }

  // 4.
  struct sigaction sa = {0};
  sa.sa_handler = on_stop;
  sigaction(SIGINT, &sa, NULL);
  sigaction(SIGTERM, &sa, NULL);

  // 5.
  if (daemon_pid_file_create() < 0)
    return 1;

  // 6.
  int results_fd = open(result_path, O_WRONLY | O_CREAT | O_APPEND, 0644);
  if (results_fd < 0)
    return 1;

  // 7.
  dprintf(results_fd, "# byte count request daemon started pid=%d\n", getpid());
  while (!stop_flag) {
    log_file_size(request_path, results_fd);
    nanosleep(&(struct timespec){5, 0}, NULL); 
  }

  // 8.
  dprintf(results_fd, "# stopping\n");
  daemon_pid_file_remove();
  close(results_fd);
  return 0;
}