Floats Don't Agree with Themselves
docs.merca.earthNot sure if this was written with AI assistance of not, but I've become allergic to linguistic triples as LLMs use them so much, reading "Same code. Same input. Different answer" makes me not want to read the rest.
a tricolon crescens ^^ https://latinaadvitam.blogspot.com/2007/02/poetry-device-of-...
I think the problem is the mismatch between the intended evocative tone of tricolon crescens and the triviality of a 'computer quirk' in the grand scheme of things. "Use figures of speech, but don't sling them around like monkey shit" - my literature teacher.
Its weird that almost every AI generated blurb tends to have that. Usually followed by "Its not just X, its Y". I wonder where they picked that up from.
My guess is LinkedIn posts and YouTube transcriptions. I notice the pattern almost exclusively being used there.
It sounds authoritative so people would respond positively to it when presented with it during training.
It's clearly AI written as are quite a few of the posts on the front-page now.
We seem to have lost the war for people to find their own voice, the AI articles are getting widely upvoted, and people pointing out they are AI are no longer getting traction, people don't seem to care.
Same. I now tend to simply abandon writing when I see those tell tale signs
Yeah, they often call this to mind: "Better ingredients. Better pizza. Papa John's."
I also got turned off after the very first paragraph. Too bad, because the content itself might be fine, but the AI language has become repulsive.
> It was IEEE 754 working as advertised. The standard pins down the storage format. It does not pin down behavior.
IEEE 754 absolutely does pin down behavior.
This article is full of LLM-induced inconsistency.
That being said, many hardware instruction does not follow strict IEEE 754 behavior.
You don't understand. In the modern AI-infested World, it is more important to make grandioese claims with em-dashes or periods working as hype-catalyst, than it is to be correct.
Once I understood that floating point arithmetic was just a fast useful approximation of actual numbers, it changed my thinking of when to use them.
Visualizing of floats as a arbitrary sample of numbers along the number line that you were allowed to choose from is a good way to figure out whether or not you should be using them.
Much like financial calculations, for this use case, exact representations of inputs are needed. An approximation of what the input might be isn’t useful.
One thing if you want point out though: there is a difference between reproduceability and accuracy.
The Java Virtual Machine (>=17, or strictfp) on every processor arch, OS, glibc, etc guarantees strict reproduceability for basic operations on floats.
some operations Math (pow, cos, log, etc) package are allowed to differ within a tiny precision window.
If you need absolute reproduceability, you can use StrictMath, which gives us an interesting property for a library like this: you could use floats, and it would be a bit reproducible on every platform and software stack combination and be deterministic, everywhere.
It would not have absolute integer math accuracy however. Whether that is still useful is up to you.
I wish the blog would reveal the values of the 3 floats that make their
cross_sign(A, B, C)
At least the Rust compiler (TFA's project is written in Rust) tries to configure LLVM specifically to avoid these discrepancies, and to treat all basic floating-point operations exactly as written with round-to-nearest behavior [0]. It does not have any of the -ffast-math options that the author('s LLM) is panicking about.
The main caveat is that on x86 targets without SSE2, LLVM is deeply wired to use the x87 instructions without attempting to emulate the IEEE overflow/underflow behavior [1]. So perhaps it could be possible to exhibit a discrepancy, but only by compiling for i586 and an ancient target-cpu. It's very doubtful that this was the cause of the original client vs. server issue in TFA's introduction.
[0] https://rust-lang.github.io/rfcs/3514-float-semantics.html
If the original code was written in Rust, then I don't think the Rust compiler is allowed to do any of these "optimizations" of rewriting floating point expressions.
The C compiler also isn't allowed to do all of them. However some people use fast-math or compilers that default to fast-math to break the rules. Some older targets also may use the 80 bit fpu, which is its own mess and any sane compiler will default to properly sized SSE instructions instead.
On x86, LLVM does not follow strict IEEE 754 behavior all the time.
IIRC it also depends on the CPU's storage and capabilities.
I love seeing a Shewchuk citation other than my ML background of learning conjugate gradient! He is truly a great educator!
Yeah, in general, this is a problem that people have spent a lot of time thinking about; while floating-point numbers can be finicky, they're what you have to work with if you have inputs at multiple scales.
(Meanwhile, I wonder why it's a fair bit harder to look up Ozaki et al.'s optimized version [0] compared to Shewchuk's original paper [1], unless perhaps later authors have found it to be no improvement at all.)
[0] https://www.tuhh.de/ti3/paper/rump/OzBueOgOiRu15.pdf
[1] https://people.eecs.berkeley.edu/~jrs/papers/robust-predicat...
his predicates paper opens with "Computational geometers despise floating-point arithmetic" same trick as the CG title: write the sentence a frustrated reader would write, then aren it.. if you like those the Triangle paper is the third one in the same key
Makes you wish everyone agreed on extended precision!
careful what you wish for, x87's extended precision is what wrote the original bug: 80 bits in registers,64 on a spill, so the value depended on register pressure
ARM and WASM dropped it for a reason, and more bits would not help anyway, a sign is one bit, any rounding step that disagrees at the last bit flips it
> It was IEEE 754 working as advertised. The standard pins down the storage format. It does not pin down behavior.
Huh? This is just false, isn't it? AFAIK, it pins down behavior, and the differences come from things like changing the order of operations (i.e. accumulating rounding error in a different order). I'm not an expert, though.
It does not pin down the behaviour.. just a storage format. See the NaNs (many representations, unspecified bits), the zeros (both), subnormals, the multiple rounding rules (no behaviour required).
This seems to be AI bullshit. You can easily get deterministic floats that give the same results everywhere. Just don't use -ffast-math.
Alas, compilers have historically fudged the behavior on some 32-bit targets, e.g., x86 targets without SSE2 [0], so you'd have to go all the way to soft-float implementations if you really want guarantees. But these targets are rare, and very unlikely to be the cause of the original issue.
(And NaN payloads in general are a mess across platforms and ABIs, but that can be solved by treating all NaNs as identical.)
> -ffast-math default, often on without intent
Ah yes -ffast-math, also known as -fincorrect-math
It makes sense in certain contexts, but it is batshit insane that it is enabled on a program (including any linked stuff) level. It would have made sense as numbers with separate types or perhaps even if you marked a region in code and it applied only to code in that region. Then the consequences of enabling it could be understood in ways that could be safe and usable (eg enabling it for a hot loop where the numerical ranges are well defined and safe in context).
But no, instead it is a fat shotgun.
That's why they always teach: "never compare floats for equality."
Or maybe they don't teach that anymore, I dunno.
See link for the Fundamental Axiom of Floating Point Arithmetic: All floating point arithmetic operations are exact up to a relative error of epsilon_machine.
I certainly teach that. When we work problems involving money, I always recommend students use integers for cents and only convert to dollars and cents when they have to print them.
On the other hand, if you store a small integer in a float it is generally reliable to compare to it. E.g., setting a float to zero and comparing whether the float is zero.