Nova's blog

Three-State Logic for FMA

2026-09-20T00:00:00+00:00

TL;DR: WebAssembly is weird. Whether it has true fused multiply-add is not known at compile time, and basing algorithm decisions around any assumption has the potential to break things.

You would think that whether or not a backend has true native Fused Multiply-Add (FMA) instructions would be pretty cut and dry, right? Until recently, Thermite types indicated if they have true native FMA instructions via a HAS_TRUE_FMA associated const bool. However, that was insufficient, and broken at worst.

# Why Indicate True FMA

It's quite helpful to have algorithms decide on certain paths or optimizations, at compile-time, based on if true FMA is available. For one, it's often as fast as a single multiply while doing both multiply and addition, so that alone accelerates many functions. It also only rounds once, so it can bypass extra effort required for that, such as some compensated arithmetic.

if const { V::HAS_TRUE_FMA } { /* ... */ }

was used all over Thermite's math libraries for these kind of optimizations.

It's also worth knowing Thermite has two flavors of FMA, which are important:

mul_add and family are always correctly rounded fused multiply-add, regardless of backend. If there is no hardware FMA, it falls back to a bit-identical vectorized polyfill, which can be quite expensive.
mul_adde and family are the opportunistic forms. They are fused when FMA is a single instruction, but plain a * b + c multiply+add where it isn't. Always fast, allowed to differ between backends.

mul_adde and family are used almost everywhere. Might as well opt for the fastest form available.

# WebAssembly

The WASM relaxed-simd proposal added relaxed_madd and relaxed_nmadd instructions. "Relaxed" meaning they act like mul_adde/nmul_adde, where it will use true FMA if available but otherwise fallback to a*b+c for performance. Nothing is guaranteed about which one you get, the engine is free to do whatever it wants. The only promise is that the choice is fixed for the lifetime of the module instance, which is what makes detecting it once at startup sound.

However, one change I made recently was to opportunistically enable true FMA on WASM if it's detected at startup. Using some WASM tricks, I inject a bit of code that runs before main(), updating a static mut global variable:

#[used]
#[unsafe(link_section = ".init_array")]
static INIT_WASM_RELAXED_FMA: extern "C" fn() = {
    extern "C" fn init() { detect_relaxed_fma(); }
    init
};

The linker collects .init_array entries into __wasm_call_ctors, which runs before user code.

mul_add and family then branch on that flag at runtime to opt into true FMA. Both arms are bit-identical, so that only trades speed.

Fun fact: relaxed_madd and relaxed_nmadd are separate instructions, and the spec allows an engine to fuse one and not the other. The init checks both independently.

Another fun fact: Using a static mut was required because LLVM won't optimize loops or collapse redundant branches if using atomic accesses, even in single-threaded environments such as WASM. Polynomial evaluation would instead query the flag every single FMA call. With static mut that query is hoisted above and branched on once, massively improving the codegen.

# What Broke

mul_add was fine, that was always guaranteed to be the correctly rounded fused multiply-add. However, mul_adde behaves differently. The default behavior of mul_adde was to check V::HAS_TRUE_FMA and either call mul_add, or do a*b+c, determined at compile time.

So when mul_adde for WebAssembly specifically was overridden to ignore V::HAS_TRUE_FMA and lower straight to relaxed_madd, leaving the engine to decide, we had cases where math algorithms chose the V::HAS_TRUE_FMA == false path but mul_adde used true FMA internally.

Normally that's fine, but notably "sum/difference of products" breaks entirely.

For example, multiplying a quaternion by its own conjugate should give exactly $(0, 0, 0, |q|^2)$ . The fast quaternion product computes each lane as a sum of four products, two of them folded into mul_addes:

let sum12 = Self::mul_adde(x, rhs_x_signed, Self::mul(w, rhs));
let sum34 = Self::mul_adde(z, rhs_z_signed, Self::mul(y, rhs_y_signed));

For $q \cdot \bar{q}$ the vector lanes are made of pairs like $t - t$ . If every product is rounded, that's $\mathrm{fl}(t) - \mathrm{fl}(t) = 0$ , exact, for free. If mul_adde fuses, one side of the pair is the exact product and the other is rounded, and what survives is the rounding error:

q = (0.1, 0.2, 0.3, 0.927...)
q * conj(q) = [1.5188877683131223e-19, 0.0, -5.551115123125783e-17, 1.0]

The quaternion product has a fast arm and an accurate arm that compensates for this. On backends with no FMA, the accurate path routed back to the fast arm, on the reasoning that all-rounded products already cancel exactly. Correct on SSE2. On WASM, with HAS_TRUE_FMA = false and an engine that fuses, it produced exactly the garbage above, in the path whose documentation promises exact zeros. The 4x4 determinant and inverse minors had the same defect.

In effect, HAS_TRUE_FMA could not be trusted. Not because the value was wrong, but because bool has no way to say "I don't know".

# Tribool

Many years ago I created the rust tribool crate, based off of Boost's Tribool class. It's basically just an enum:

pub enum Tribool {
    False,
    True,
    Indeterminate, // Unknown
}

with various three-state logic methods on it. Well, what would you call the state in which we don't know if FMA is present or not, other than Indeterminate?

The crate fit perfectly, so I set out and replaced V::HAS_TRUE_FMA with

const HAS_NATIVE_FMA: Tribool;

Backend	`HAS_NATIVE_FMA`
x86 SSE2 / SSE4.2	`False`
x86 AVX2+FMA, AVX-512	`True`
NEON (aarch64)	`True`
Scalar	whatever the baseline target features say
WASM	`Indeterminate`

The funny part is after the migration, no algorithm in Thermite mentions Indeterminate, instead preferring one of:

// "Is FMA cheap here? i.e.: dedicated instruction"
if const { matches!(V::HAS_NATIVE_FMA, tribool::True) } { /* ... */ }

// "Could two products round differently from each other?"
// Note the boolean NOT
if const { !matches!(V::HAS_NATIVE_FMA, tribool::False) } { /* ... */ }

As it turns out, the old bool const was close, but not exactly what was needed for informing algorithms about the FMA behavior. Depending on the situation we need to know if FMA is cheap (i.e., a single instruction), or just if we can rely on certain rounding behaviors or not.

# Composites

One more place the old bool was wrong was for composite types, such as Dual or Complex numbers, which reported HAS_TRUE_FMA = false unconditionally. However, if the question is "Is mul_adde faster than a*b+c?", that's more interesting.

Nearly every consumer of HAS_TRUE_FMA cared because of performance reasons, not because of the single rounding. Even then, making use of FMA for Dual/Complex's own mul_adde does improve precision some. Therefore, these now simply forward the inner type's HAS_NATIVE_FMA.

# Conclusion

In the end, most algorithms didn't really care, but when !false != true, making use of Tribool is both useful and cool.

Compensated Arithmetic in Core

2026-09-19T00:00:00+00:00

TL;DR: Opting into algebraic optimizations for scalars silently broke a significant portion of Thermite's math library. Here's how I fixed it.

Recent changes have exposed the raw compensated arithmetic primitives like two_sum and two_prod in Thermite core, whereas they were previously only in the thermite-compensated crate. Why? Turns out it's pretty essential to keep the basic compensated ops around at the very low level.

# What is Compensated Arithmetic?

Compensated arithmetic primitives, more formally known as Error-Free Transforms, are a way of recovering the error lost by rounding when applying basic operators such as addition, subtraction, multiplication, and division. Almost everything else falls out from that.

a + b in an f64 isn't a + b. The exact sum can need far more bits than the mantissa has (1e300 + 1e-300 would take about two thousand), so the hardware rounds it to the nearest representable value and drops the rest. What gets dropped is the interesting part, because for addition it's always exactly representable in the same format. Knuth's 2Sum recovers it in six adds, no branches, no extra precision.

\begin{aligned}
s &= \mathrm{fl}(a + b) \\
e &= (a + b) - s \\
s + e &= a + b
\end{aligned}

$s$ is the result you would have gotten anyway, and $e$ is exactly what the rounding lost, so the pair $(s, e)$ holds the exact sum in two words. Nothing is lost, the value is just re-encoded across two floats.

Multiplication works out the same way. The exact product of two $p$ -bit values needs $2p$ bits, and the residual left after rounding fits in $p$ of them, so it's recoverable as well. With a true fused multiply-add that's two instructions:

let p = a * b;
let e = a.mul_sub(b, p); // fma(a, b, -p)

Without FMA, you need Dekker's algorithm, which cuts both operands into half-width pieces with Veltkamp splitting and sums the four cross products in the right order. That's 17 operations instead of 2, so many algorithms prefer other methods when native FMA isn't available.

Everything else is built on those two. Double-double arithmetic (Compensated<f64>, around 106 bits of significand) is a two_sum/two_prod pair plus renormalization. Kahan and Neumaier summation are two_sum with the error fed back into the next iteration. A correctly-rounded mul_add on hardware without an FMA instruction is Boldo and Melquiond's construction, built out of two_prod and two_sum. Interval arithmetic needs the residual to know which way to nudge an endpoint. It's pretty handy all things considered.

One catch is that this buys precision, not range. The exponent is untouched, so anything that overflows or flushes to zero is still gone, and Veltkamp splitting has its own threshold ( $2^{115}$ for f32) past which operands have to be rebalanced before they can be split at all.

# `algebraic_scalar`

Thermite has the algebraic_scalar Cargo feature to enable the "Algebraic" float ops (algebraic_* on f32/f64), which allow LLVM to casually reassociate operations to improve performance or improve autovectorization opportunities. With this feature enabled, all basic operations (+,-,*,/) are set to algebraic ops at the lowest library level, so everything inherits those everywhere. This is honestly great for most scalar code. If you are not using or cannot use SIMD for one reason or another, making the scalar fallbacks as fast as possible is ideal.

Summation with a division in it is the clearest example:

// NOTE: "Scalars" in Thermite are still wrapped in `Vector`
let mut sum: Vector<f64> = Vector::splat(0.0);
for &x in xs {
    sum += x / Vector::splat(3.0);
}

Strict, at -O -C target-cpu=x86-64-v3, the inner loop is unrolled 8x and every copy looks like this:

...
vmovsd  (%rax), %xmm2
vdivsd  %xmm1, %xmm2, %xmm2   # xmm1 = 3.0
vaddsd  %xmm2, %xmm0, %xmm0
# repeated 8 times

Enabling the algebraic_scalar feature gives this instead:

vbroadcastsd __real@3fd5555555555555(%rip), %ymm1   # 1/3
vfmadd231pd  (%rcx,%r10,8), %ymm1, %ymm0
vfmadd231pd  32(%rcx,%r10,8), %ymm1, %ymm2
vfmadd231pd  64(%rcx,%r10,8), %ymm1, %ymm3
vfmadd231pd  96(%rcx,%r10,8), %ymm1, %ymm4

The division is gone entirely, replaced by a multiply against a rounded $1/3$ , which is less precise but "close enough" for most use cases. The add contracted into that multiply, and the serial accumulator turned into four independent ones at four doubles apiece, so each iteration retires 16 elements in 4 instructions where the strict version needed 32. With a runtime divisor it does the same thing and just hoists a single vdivsd out of the loop. This is a great trade for most code.

However, taking a look at two_sum:

fn two_sum(a: Self, b: Self) -> (Self, Self) {
    let s = a + b;
    let v = s - a;
    let e = (a - (s - v)) + (b - v);
    (s, e)
}

As real-number algebra, with $s = a + b$ and $v = s - a$ , reassociation collapses $v = (a + b) - a = b$ , and with it the whole error term becomes zero:

\begin{aligned}
e &= \bigl(a - (s - v)\bigr) + (b - v) \\
  &= (a - a) + (b - b) = 0
\end{aligned}

LLVM is free to do this in multiple places, including common Cody-Waite style range reductions that occur in almost everything.

For example, average-precision sin/cos reduces its argument modulo $\pi$ by splitting the constant across several floats, $\pi \approx \pi_A + \pi_B + \pi_C$ , where $\pi_A$ has enough low mantissa bits zeroed that $n \pi_A$ is exact for every $n$ in range, and each following term picks up the bits the previous one couldn't hold:

let n = (x * FRAC_1_PI).round();
let r = ((x - n * PI_A) - n * PI_B) - n * PI_C;

The subtractions are deliberately sequenced so that each one cancels against a quantity of its own magnitude, keeping $r$ accurate to full precision even when $x$ is large and $n \pi$ is nowhere near representable. Algebraically, though:

\begin{aligned}
r &= x - n(\pi_A + \pi_B + \pi_C) \\
  &= x - n\pi
\end{aligned}

so LLVM is free to fold the three constants into one rounded $\pi$ and emit a single $x - n\pi$ . The extra $\approx 50$ bits of the constant vanish, and the reduction degrades from full precision to whatever $x - n\pi$ happens to give. Not good.

exp and ln have similar range reduction. algebraic_scalar would silently break that when using the math library with scalars. The effects of it spread quite far.

For that reason, thermite-compensated previously would emit a compile_error! if used with the algebraic_scalar feature enabled. It would have been simply broken.

# The Solution is more than the `two_sum` of its parts

Fixing all of these weak points was simply a matter of moving two_sum/two_diff/two_prod/two_quot from thermite-compensated and putting them on FloatRegister/FloatVectorWithBits.

That way they can bypass the algebraic math intrinsics in the scalar backend implementation, using strict operators for compensated arithmetic, while also allowing anyone to use them anywhere that FloatVectorWithBits is available. That's actually an important distinction, because if it were only FloatVector, two_sum would have to be available on Complex or Dual composite types that implement FloatVector. Compensated arithmetic on those only makes sense as an implementation detail, not on the public interface.

Furthermore, FMA, even mul_adde (opportunistic FMA: fused where the hardware has it, separate mul+add where not) and family, inherently use strict operators as well so they cannot be reassociated.

Although not taking advantage of algebraic reassociation, LLVM is smart enough to see that a.two_sum(b).0 (accessing only the sum, ignoring the error) can ignore the error calculation entirely, returning a + b, but with a strict operator. In a way it provides an escape hatch for algorithms to ensure that the sum is exactly that, not mixed in with nearby values. Same with the others.

This pattern and just using two_sum/two_prod deep in the math implementations helped improve or stabilize precision across a wide variety of functions, and fixed a few latent bugs that would have crippled the emulated FMA paths. Range reduction, accumulators, etc., are now fixed and stable under the feature.

Originally, this change was just because I wanted thermite-compensated to compile regardless of what crate features users used. However, I've since found quite a few places in Thermite's math libraries where either strict operators or compensated arithmetic significantly improve precision, nearly for free in many cases, and the autovectorizer can do an even better job everywhere if that feature is enabled. Win-win.