Floating point numbers, and why they suck
riskledger.comThe problem is that we're still stuck with only binary floating point types in our CPUs and compilers and runtime environments, over a decade after ieee754 released their decimal float spec [1].
Once we finally move away from binary floats, these nasty binary-exponent-decimal-exponent lossy conversions will end (as will the horribly complicated nasty scanf and printf algorithms).
This is a solved problem. Our actual problem now is momentum of adoption.
I've anticipated that this will eventually happen (hopefully sooner rather than later), and proactively marked binary floats as "legacy formats" [2].
[1] https://en.wikipedia.org/wiki/Decimal64_floating-point_forma...
[2] https://github.com/kstenerud/concise-encoding/blob/master/ce...
I predict IPv4 will be widely considered legacy before decimal floating point numbers are common place, let alone binary floats being considered legacy.
From what I've seen, there's just simply no demand for decimal floating point; there's basically one or two people who ask for it, and this carries up all the way through to C committee standardization. Far more people care about things like non-default rounding mode or exception support than care about decimal floating point, and that's an area of floating point that itself is pretty damn low on people's priority lists.
The advantage of decimal floating points is that it fixes the rounding error when converting to/from decimal strings... and that's it. It doesn't fix non-associativity, it doesn't fix gradual precision loss, it doesn't fix portability issues because different libraries have different precision targets. And you get this for the slight cost of making every single floating point operation slower.
And here's the other truism about floating-point: most people who care about it value speed over correctness. There's strong demand for operations like approximate division that don't do the refinement steps, and consequently throw away half the bits. Fast-math flags are similarly pretty common, because users already aren't expecting bit-precise results, so the resulting changes in precision are quite often acceptable for them.
Many compilers and runtimes support them just fine. CPUs don’t really support them, with the exception of POWER. The dirty little secret (use case measurement), however, is that Intel’s software library on x86-64 is faster than POWER’s instructions. I don’t believe there is any roadmap (e.g., 5 years out) to add hardware support mainly because customers don’t need them or want them (globally speaking) and software is fast enough for majority of use cases.
I agree that's cool. It's usually sold as for finance applications, for compliance with rounding rules from the pre-binary era etc. But really, any number that lived as a decimal digit string somewhere in its lifetime should probably have been decimal throughout.
But I don't think it would have mattered here anyway. With finite precision floating point addition is not going to be associative, decimal or not.
In practice they don't need to be perfectly associative across all possible values, because most real-world calculations tend to stay within a few orders of magnitude, and don't require more than 10-15 significant digits (even for Earth orbital calculations). Once you need more than that, you'd have to use a multi-word numeric format regardless (same as if your operations started overflowing your largest int type).
Decimal floating point still doesn’t have associative addition or multiplication.
But it can be shaped by rules to be associative enough to be useful for the common case. All you need to do is ensure that your calculations are in no danger of blowing past the max significant digits (much like you'd guard against overflow on integer types). Once you need more range, you switch to a multi-word numeric format (or impose rounding rules).
> All you need to do is ensure that your calculations are in no danger of blowing past the max significant digits
If you have a lot of control of the scale of the numbers you’re computing with, you could use fixed point instead of floating point.
The situations that make it useful to have a floating point (rather than fixed point) are also the situations where addition isn’t associative.
> If you have a lot of control of the scale of the numbers you’re computing with, you could use fixed point instead of floating point.
You could, but why would you go to the extra complication (and ensuing bugs) of adding an implied fixed point to integers when a decimal float type gives it to you for free?
As I said, it's not a silver bullet; it's just for the majority case (much like 32 and 64-bit integers are for the majority case when dealing with whole numbers).
I'm very annoyed by this title. It's a beginners mistake and FP numbers just are what they are -- a tradeoff between precision, range and efficiency. If you use FP, you should be familiar with its intricacies in the same way a C programmer needs to be aware of unsigned overflows.
I agree. Enough already. Isn't this covered in something like week 2 of CS 101? If you've missed that, maybe the literally decades of "ermagerd, floats!" articles would give you a hint.
As someone who's worked in science and finance (modeling, not accounting), floats work just fine, thank you very much. The modeling/accounting split in finance is a legit point of confusion, though.
IEEE 754 floating-point format has been very carefully and deliberately designed by the top experts in the field. When someone "it sucks", that is a guarantee that whatever design he might come up with would suck ten times as much.
Floating point feels to me like a very well designed jack-of-all-trades. They are decent for most tasks, but if you know exactly what you want to do then you can find something that does that particular job better.
This. Sure, floating point numbers come with a footgun, but the title is so clickbaity, would it really hurt to name it: "What I wish I knew about floating numbers before relying on them?"
I think it's common to just not be taught floating points need to be handled a lot differently than whole numbers, so the common experience is getting bit by it. I don't remember ever being taught that in my traditional schooling, personally.
This is a good link to send to people: https://floating-point-gui.de/
I remember covering how floating point were stored, but I don’t remember going all that deep into how that affected arithmetic.
Nooo unsigned integers suck, if you subtract 5 from 3 what would you get? Then I had a stroke of genius and figured it out, I have no idea what I'm doing.
The problem is that the creeping imprecision actually comes from the exponential mismatch, not the actual float itself. If we were inputting floats in binary format, most of our problems would disappear. But since we think in decimal, we input in decimal, and then magical complicated algorithms convert those "decimal" values to the "nearest" binary representation. And then the trouble ensues when we try to use them in calculations and get weird results.
Decimal floats would operate just as we're used to, and would solve 95% of our floating point problems.
> Decimal floats would operate just as we're used to, and would solve 95% of our floating point problems.
It does not in fact solve the problem identified in this blog post (a non-associative issue). It does not solve any of the problems I generally see mentioned in topics like multiplayer video game desyncs (math libraries on different platforms don't return the same results). It's a pretty bold assertion that "95% of our floating point problems" would be solved.
Every single one of the problems in the article are due to binary-exponent / decimal-exponent mismatch and the inevitable differences in how different platforms do the conversions. With decimal floats, that problem disappears. There is no converted-it-to-90.34326374227855; all implementations would correctly have 90.34326 because there's no lossy conversion-to-binary-exponent to make; it's just stored in decimal with a decimal exponent, the end.
Multiply 90.34326 by 0.1 and then 0.1 again? Once again, all platforms will correctly and EXACTLY give the result 0.9034326. No loss of precision at all. Or do it 0.1 times 90.34326 times 0.1. Same result (exactly the same). Do that with binary floats and you're in for a world of hurt.
> Every single one of the problems in the article are due to binary-exponent / decimal-exponent mismatch and the inevitable differences in how different platforms do the conversions.
The article itself actually started with your assumption... and found it wasn't true. The type conversion wasn't the issue--it was coming back as the same value on all the different systems.
The flaw was that the sum came out wrong depending on the order that it was done. And the input numbers were (if I'm understanding correctly) computed as a / b, for some integer values a and b, which are not generally perfectly precise in decimal or binary floating point.
I think I need to do a blog post on this. I get the same kind of reaction from pretty much anyone I talk to about this, and it takes hours of explanations & diagrams to get them to understand. It's a real shame too, because a correct understanding would have a LOT more people putting pressure on implementors to support decimal floats :/
I wish that fixed point numbers (as in Qx.y) was a first class citizen in programming languages. As well as saturation arithmetic.
Actually I wonder if anybody did a performance/power comparison of using floating point vs fixed point math in some common tasks using modern CPUs with extensive FP support.
They are a first class citizen in Ada, you even get two (three) varieties: http://www.ada-auth.org/standards/12rm/html/RM-3-5-9.html / http://www.ada-auth.org/standards/12rm/html/RM-3-5-7.html
Julia has this https://github.com/JuliaMath/FixedPointNumbers.jl. In general, fixed point is a little slower, but it's not awful. Fixed point is really good for small bit-widths in hardware though.
Floating point behaviors are well understood. This type of problem shouldn’t happen if your developers are qualified.
In Julia, there is the `isapprox` function to inexactly compare 2 numbers. (you can use it in infix form: `x≈y`. By default, two numbers are approximately equal if their relative tolerance is less than `sprt(eps(typeof(x))` (around 1e-8 for 64bit Floating point numbers)
Using equality to compare 2 floating point numbers doesn't get you anywhere, specially if you are using mathematical functions (the implementation of `sin` or `exp` could vary with operating systems or software versions)
Relative tolerance becomes problematic though if your numbers are very close to zero. (https://randomascii.wordpress.com/2012/02/25/comparing-float...)
Which is why it also lets you specify an absolute tolerance. That's not done by default because there isn't a good way to choose one (unlike with relative tolerance).
yes, it is a common problem, it is acknowledged in the function's documentation: https://docs.julialang.org/en/v1/base/math/#Base.isapprox
This is an issue that has bitten people before. Indeed, there's probably been some discussion of it and might even be some common mitigations taught as "best practice" in some places.
Do we ever need to store floats? using them in flight is one thing, but stored data so often needs to be some fixed precision.
If you want to type pun floats, store hexadecimal literals. Those will be accurate.
Asking basic questions about floating-point in interviews remains to this day a good way to distinguish good and bad programmers.
You'd think everyone would know the basics, but even in stuff like finance, people routinely fuck up decimals.
And that's why you user "currency" type of style. Both as calculation in your programming language (Go in this article) and in DB (PostgreSQL in this article). Fixed width "floating" number which is actually your largest integer type (64 bits these days, but there are libraries for 128 bits aplenty as well) is your friend.
Floating point arithmetic doesn't «suck», it's the code that treats floats as decimal numbers that sucks.
> We are sending the data to the warehouse as a float
There is your problem
Any vector addition done concurrently will result in indeterminate roundoff errors (to be precise there might be O(N!) different possible outputs depending on the order of calculation), this is literally the point of the field of numerical analysis. But that's really the tip of the iceberg. Beware any mathematical operation done with numbers of vastly different orders of magnitude. Beware spinning your own matmul even if you can't guarantee some normalcy to the numbers in your matrices.
Calculating an average is fairly painless, you can create an upper bound on the possible error that isn't usually dangerous. This isn't the case for all computations.
The vast majority of problems with floating point numbers are simply due to people not understanding number bases. Only a tiny subset is caused by people not understanding floats proper.
Consider the number 1/3. In ternary (base 3), you write that as 0.1, whereas in decimal (base 10) you write it as 0.3333... recurring. If you try to represent that number with a fixed number of decimal places, you have precision issues. E.g. decimal 0.333 converts to 0.02222220210 in ternary.
Now, the thing with that example is that we treat decimal as a special, privileged representation, so we accept that 1/3 doesn't have a finite representation in decimal as an entirely natural fact of life, while we treat that same problem converting between decimal and binary as a fundamental deficiency of binary.
Let's talk about why some numbers have finite representations while others don't. 53/100 is written as 0.53 in decimal. The general rule is that if the denominator is a power of ten, you just write the numerator, then put the decimal mark have as many digits from the right as the power of ten. If the denominator is not a power of ten, you make it one. 1/2 turns into 5/10, which is 5 with one decimal place, or 0.5. Obviously, you can't actually do this for 1/3. There's no integer `a` where 3a is a power of ten. The general rule here is that, if the denominator has any prime factors not present in the base, you don't have a finite representation. 4/25 = 16/100 = 0.16 has a finite representation, but 5/7, 13/3 don't.
Now, because 3 is coprime with 10, no number with a finite ternary representation can have a finite decimal representation (and vice versa), and we're used to it. Where binary vs decimal becomes tricky and confuses people is that 10 = 2 * 5, so numbers that can be expressed with a power-of-two denominator have finite representations in both binary and decimal, so you can convert some numbers back and forth with no loss of precision. Numbers with a factor of 5 somewhere in their denominator can have finite decimal represntations (1/5 = 2/10 = 0.2), but can't have finite representations in binary. 0.1 = 1/10 = 1/(2 * 5) and you can't get rid of that five. And that's why everybody gets bitten by 0.1 seeming to be broken.
Ancient COBOL had binary coded decimal.
Every database and most programing languages have decimal floating point types. They usually don't use BCD, and instead use some more efficient representation. But anyway, the representation is irrelevant.
By the way, integral BCD (what COBOL did most of the time) is useless.
Yep, floats should be considered dangerous.
$ python3 -c 'print(.1 + .2)'
0.30000000000000004Everyone knows this from using calculators in school, even non programmers knows this.
Obligatory: What Every Computer Scientist Should Know About Floating-Point Arithmetic
https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.h...