The Epochalypse Project
epochalypse-project.org197 points by maxeda a month ago
197 points by maxeda a month ago
> And what’s worse, malicious threat actors can manipulate time synchronization protocols in many cases to trigger this vulnerability at the time of their choosing.
If you switch to 64 bit timestamps, and the network protocol supports dates > 2038, can you then just trigger the rollover bugs by pretending it's 2*64 - 1 seconds after epoch start?
Also, if the actions are potentially so severe, and NTP (or whatever is used) so vulnerable, why haven't we seen many such attacks in the wild?
Update: to be clear I'm not arguing that there isn't a problem, I've already run into it myself. I'm trying to understand how severe it is, how exploitable, and how robust a solution could be.
I think this effort would benefit from trying to qualify what “unpredictable ways” actually means. If anyone is testing devices, a catalog of test results describing the actual failure modes that were revealed would help make this whole thing more concrete.
I think many software engineers know that if you want to make any organization care about this type of issue, you need to be ready to demonstrate the severity and impact.
Worst case scenario, it bricks your device: https://issuetracker.google.com/issues/36928638?pli=1
Even if the system boots properly, there's various critical systems that depend upon having the correct time. Say goodbye to things like HTTPS and SSL/TLS certificates.
Will the root-certificates still be trusted in 12 years? Will we largely use the same TLS versions? And if systems can be updated to account for that, shouldn't they also be able to be updated to deal with the timestamps limitation?
For comparison I've revived a decade old Axis PTZ camera recently and it can't be used with HTTPS because it only supports TLS 1.0 which is deprecated across the board these days lmao. The UI is so bugged out it's not possible to change the default username and password anymore.
There's two kinds of internet connected devices these days, those that keep getting updated and those that drift into incompatibility and die as the rest of the ecosystem evolves around them. If these supposed critical devices will still be in use in 12 years without any maintenance then they're unlikely to have any actual importance.
It means 'anywhere between being bricked and no problem at all, and we can't give you any idea of how severe or how likely any of those possiblities are'. The only way you can really know is to thorougly audit your system and/or test it. Preferably both.
So that means the best tool is a 2038 test environment - which people then install their application(s) and test it e2e to see what the impacts are.
However, I’m not sure how you make a 2038 test environment
It assumes that the OS/Kernel etc… are defacto frozen to 2025 or whatever increment until 2038
What was the y2k solution for the people that implemented those fixes in the 90s?
libfaketime is useful for testing these kinds of things.
It intercepts system calls to get the time and reports a fake time to the application.
Demonstrate? Or just scaremonger?
Y2K showed that you don't need details beyond vague threats of "medication administered at the wrong time" and "planes falling out of the air" to get organizations and the public to care. No idea how that's going to tie into the conspiracy-heavy media landscape we inhabit now.
(Note I do think this is a serious issue that needs to be addressed. And I'd love to see specific examples. I'm just pushing back against the idea that examples would make much difference to advocacy efforts)
What? Y2K did have many demonstrable problems... Having a 2 digit year did obviously cause problems. The reason nothing happened is because a shit ton of time and money was spent making sure it didn't.
Agreed. My point is that the orgs paying for all these updates were mostly motivated by the vague claims of experts rather than concrete examples
It was pretty easy for orgs with affected systems to produce concrete examples for themselves. Maybe to everyone else it seemed vague, but for the people who had to deal with it, it was taken pretty seriously from top to bottom.
It was thankless work that is still glossed over and waved away today, but it was all a very big deal throughout the late 90s.
> It was thankless work that is still glossed over and waved away today, but it was all a very big deal throughout the late 90s.
Mike Judge even made a movie about it! Office Space might be the most recognition turn of the millennium programmers will receive.
All I can say is that at my level in an org, if I want to say “instead of developing X this quarter, I want to test the effects of 2038 on Y,” that’s a far easier conversation if I can say something like “in similar embedded devices they crashed and wouldn’t even respond to OTA updates” vs “something bad could happen. Not sure.”
The latter is just a ripe plumb, left to rot in the backlog.
That's nonsense. Orgs spent time and resources on it because they grabbed a test server and demonstrated it caused problems. It's not some weird ethereal untestable bug. They set the dev server to a minute before midnight and went "oh shit".
That and quite often the problems started showing up years before 2000 itself. "Hey, the scheduler is giving me a meeting 80 years ago" type weirdness when it crossed the boundry.
“Planes falling pit of the sky” still gets used both as an example of overblown Y2k fear-mongering AND the reason why all those quiet preparations were necessary.
Is there any plausible mechanism for Y2K bugs to cause planes to fall from the sky?
Why not? From five years ago, on the most modern type:
https://www.theregister.com/2020/04/02/boeing_787_power_cycl...
Seems to be unrelated at first look, but considering the complexities of systems, and cascading failure modes?
I fixed a recent Y38 bug in some classic ASP code. The bug was nothing more than a simple `Date() + 5000` computation (adding 5000 days to the current date) as a sort of expiry date applied to something; I don't recall the exact details. VB6 did not take kindly to computing any date value beyond the Y38 max and threw an error. In practice this error ended up denying service to everyone even though the Y38 max date was 14 years in the future. You never know what little bugs like that are lurking in such legacy code.
"Epochalypse" is a nice coinage. Not a new one either, this 2017 article attributes it to Mikko Hypponen (of F-Secure). https://www.tomsguide.com/us/2038-bug-bh2017,news-25551.html
Goes back at least slightly before that, as I've had 2038epochalypse.com registered since March 2017, but I can't recall whether I thought I was being clever or whether I heard it somewhere else.
I've had my "2038 consulting" sites since Feb 2011, but someone got epochalypse dot com registered August 2007.
Heh, I've got five domains of a similar name too. Going to reach out to the owners of the website this post is referring to and see if they want them.
Telling home users to check that their existing smart devices will still work in 13 years seems like overkill. It seems unlikely that more than a tiny fraction of them will still be in use then, if any.
Businesses installing new smart infrastructure and devices will need to pay attention to this, and in 10-15 years they'll need to work out what to replace, of course.
Agreed. A serious approach to this problem would be: Identify critical computers which are currently 13+ years old (most likely embedded systems). Assume that the same sorts of systems will be 13+ years old in 2038. Focus on raising awareness with that particular target audience, e.g. give talks about the 2038 problem at embedded systems conferences. Try to get it included in university curricula. Etc.
15 years ago i was working at a startup in SV and a kid we hired was saying how he was sad he missed out on y2k because he was too young. I filled him with joy when I mentioned the 2038 bug. lol
Who will register a domain for the upcoming year 292271025015 problem?
Things to keep you busy until then:
https://en.wikipedia.org/wiki/Time_formatting_and_storage_bu...
Are they doing anything to fix it or just raising awareness?
Here's an example of measuring packages that report warnings for software that has suspicious conversions. Compile with `-Wconversion` with both 32-bit and 64-bit time_t, and see what the difference is. https://github.com/mkj/yocto-y2038
That is using yocto, but you could probably do something similar with other less-embedded distros too, if you can rebuild the world.
FWIW I didn't find much interesting with that apart from busybox dhcpd.
It looks mostly like a project for self promotion of the two authors. Maybe they offer some consulting services.
Funniest is that one of them wrote that they have "learned about it after Y2K bug". I thought one learns about this overflow in a "introduction to programming" class...
It also says nothing about a formal education, just that he has worked in IT since his teens. I didn't hear of the 2038 problem myself until the whole Y2K debacle, but then, I was in my teens at the time.
btw yocto has been using 64 bit time_t since last year: https://docs.yoctoproject.org/dev/migration-guides/migration...
When will milliseconds since the Unix epoch not be precisely representable in IEEE double anymore? (As it's used in JavaScript.)
Fwiw, it's much worse for 64-bit nanoseconds, running out in the year 2262 for signed, and the year 2554 for unsigned. As for who needs nanoseconds, trading systems, particle physics, and globally distributed systems do.
Which systems use a single variable for nanoseconds time? The APIs I know of all use a struct with two members, one for seconds and the other for the nanoseconds within the second.
It's a pretty common format for timestamps on modern file systems used by Unixy operating systems (e.g. ext4, XFS, APFS, ZFS, btrfs, etc.).
At least ext4 (and its predecessors ext2/ext3) uses 32-bit seconds for the timestamps, and has separate fields (available only when the inode size of the filesystem is greater than 128 bytes) for nanoseconds and for extending the seconds fields to 34 bits.
True enough [1] but the end result is actually a slightly shorter time range ("only" 544 years vs. 584 years for a "true" 64-bit ns-resolution timestamp). Though, admittedly, when we get anywhere near to the year 2514 and happen to still be using ext4 (or yet another extN+1), the format can always be extended again.
Also, I may have been wrong about XFS on Linux at least; though the data structures are not as easy to follow as for ext4, they seem to be using struct timespec64 for all times, which has both separate fields for seconds and nanoseconds as well as a 64-bit-wide seconds field [2], covering a range of 584 billion years.
[1]: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...
[2]: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...
MAX_SAFE_INTEGER = 2^53 - 1 = 9007199254740991. The next integer 2^53 is representable, while the following one 2^53 + 1 isn't.
2^53 + 1 milliseconds = 9007199254740993 milliseconds = 104249991 days 8 hours 59 minutes 0.993 seconds. Since the zero point is midnight 1970-01-01, we reach the first non-representable millisecond at 287396-10-12T8:59:00.993 (be careful to use the Gregorian leap rule if you want to calculate this manually).
In the year 287390 (285420.9 years after 1970) according to my calculations using DuckDuckGo’s calculator.
If you ask ChatGPT it’s:
= 285,616 years after 1970
= Year 285,616That seems a bit small seeing how doubles can represent 53 bit integers accurately.
>>> 2**53/(1000*3600*24*365)+1970
287586.41472415626
Epochalypse is clever. I'd been calling it Y2K38 so it made sense to people outside the industry.
I witnessed my first y38 bug a couple weeks ago - just as a matter of someone doing forecasting in a system with an incorrect casting.
At least, the good part is that people will get "hunches" about y38, just like you start getting "hunches" about bugs related to locales, time zones, character encodings, currency roundings, etc...
I don't know if there are courses, books, etc... about all those matters that are definitely non "computer sciency", but occupy so much of our engineering time ?
I've been planning for decades to have a party on that day.
Only 13 years left!
I would much prefer we put this off till the last minute so that I have one more job available to me before retirement. Thanks.
Patched in OpenBSD 5.5 [1] which was released in 2014. Patches were upstreamed into software in the ports tree, so hopefully that made life a little bit easier for other operating systems. Linux seems to have followed around 2020.
I worry that for a lot of devices (IoT, etc.), the fix will be "throw it away and buy a new one"
The fact that it's not common knowledge and not taught in schools and universities is concerning.
Is it? I was starting my career during Y2K. It was widely regarded as a bust, and I’d anticipate people expecting the same damned thing?
It was largely regarded as a success. Why do you say a bust?
I remember some people looking forward to the y2k chaos. There was a smug sense of “look at how badly those brainiac computer guys screwed up”
I remember a rumor that “if enough people take their phone off the hook before midnight it’ll take down the phone network”
Sorry, I meant from the perspective of people expecting chaos to happen. I agree it was wildly successful in that basically nothing happened!
"What am I paying my IT department for? My computers never have problems! It's a scam, I tells ya!"
I don't know about "largely", but I can point to a least one instance where the most week know comedy show at the time in France ("Les guignols") had a bit about how the Y2K bug was a "scam" orchestrated by "Big Tech" to make people buy new computers - given that nothing terrible happened on Y2K.
The sketch was a riff about the kindergarten joke :
- why do you [insert weird action] ?
- to scare the girafes away
- but there are no girafes here
- of course, I've been [insert weird action here]
And I was a software engineering student at the time, and I tried explaining they were missing the point, but I think it will be how we're approached this time.
So, I think nothing will be done, or not enough, and bugs will happen in 12 years.
But, thankfully, all our computers will have been replaced to allow running js.
my capitalist side remembers that many people made retirement money because of Y2K.
Maybe I should start thinking about an early retirement :)
Well, by 2038 the chatbots will have long since fixed all the time format problems, all our problems, and will be working away at problems of which we cannot concieve.
So, ahem, your “early retirement” is assured?
A year is a lot longer than it used to be. And quickly getting longer. Measured in change.
How exactly? asking for a friend
"You still know php4? We need someone to fix this 50000 loc file that was written in 2005. Willing to pay big money."
Wouldn't it just be a matter of recompiling as 64-bit? No need to mess with the 50000 LOC.
Is it too early to raise awareness for the Y10k bug?
Eh, I've got 13 years to deal with this. No big deal.
The best thing about waiting until the last minute to fix something is it will only take a minute.
The solution is simple: go back in time and get legislation passed to mandate the release of source code for devices that the original manufacturer no longer intends to support.
That might be imposssible right now, and not just because of the time travel thing. While it's a good idea, and while people are working to get legislation passed that would prevent the forced obsolescence of products by companies requiring, then no longer supporting, public servers, for instance, companies would fight to the death any requirement to make their source available.
But the problem isn't only theirs - open source projects that are steered by corporations often lack the willingness to expend some work now in order to save work decades from now.
For instance, NetBSD moved to 64 bit time on all platforms using clever versioning to maintain compatibility with 32 bit timestamps. They did this on 32 bit architectures, too, back in 2012. [0], [1]
Linux, on the other hand, has no shortage of people who either want to drop 32 bit support entirely or who think that the work to modernize 32 bit support isn't worthwhile (because, in part, they've forgotten that embedded Linux is a thing, and they also don't care about people who can't afford modern hardware). The 64 bit time transition for 32 bit platforms has taken ages, but is thankfully finally underway. [2]
There's practically no hope for corporations. If they aren't given stuff that's already fixed (that is, if the fix isn't already baked in to the Linux they choose to use), they're certainly not going to "invest" resources in fixing things from the past. Heck - it took Google more than a week to fix issues with Chromecasts that came from bad planning. [3]
What can we do? Most people aren't going to decide on a purchase based on whether their product might stop working in 2038 unless we make it very easy for them. We should test more things and point out when products will fail. Of course, nobody is going to care if their wifi device bought in 2025 won't run in 2038, but if someone told you that a $50,000 car's infotainment system will stop working, they might care.
[0]: https://wiki.netbsd.org/symbol_versions/
[1]: https://web.archive.org/web/20200122035908/https://twitter.c...
[2]: https://wiki.debian.org/ReleaseGoals/64bit-time
[3]: https://www.theregister.com/2025/03/13/google_chromecast_fix...
Born too early for Y2K, but just in time for the Epochalypse lol
Well we don’t know if it will actually be an e/apocalypse. Which sounds very scary and doomsy to me.
But it will be the end of time, as we know it!
yeah sure.
My my
(Not) Looking forward to 2038 onwards as people ignore the fact we coordinated to fix this and start saying it was all a big hoax.
Same thing happened with Y2K. You can’t let such people bother you, or you’ll be miserable.
Except "those people" are the ones who will run the USA for the foreseeable future.
Fixing Y38 will require some public spending. If "researching cancer" is not considered worthy of public spending any more, I'm curious about how the nerds will manage to justify replacing lots of embedded chips "because maths".
If it takes the usual 8 years to replace the current administration with one that accepts listening to experts, and unless big donors can make a profit by organizing the transition, we can expect serious efforts to only start around 2032 globally. No idea if that will be enough this time, we'll see...
(On the bright side, maybe they can charge customers ? That would work.)
Will people use planes falling out of the sky as the reason the fix must be priority 1 this time?
Even though it might seem unlikely, you absolutely can't eliminate the possibility of planes falling out of the air unless you have thoroughly audited and tested all their systems in advance. Not having planes in the air at the exact moment of rollover - just in case - might be a good idea too. Similarly financial markets and other key infrastructure.
13 years is plenty of time to make a start on all of this, and will allow the costs to be spread over that time period without undue hurry. It's also long enough that it will be possible for much software to be allowed to reach end-of-life without being audited, while applying rigorous testing to new software. Procrastination, though, is not a good option.
Planes will still be using floppy disks. Or they will have upgraded to the worst brand of Android, as usual.
Why should floppy disks be a problem? Even if they are fantastically out of date, if they can be maintained as a niche legacy technology, they're not a problem. They won't suddenly stop working, it will just slowly get more and more expensive to get the parts. Supply chains are magic - at the right price, someone will always be willing to make more floppies or drives. Of course that price might be $500 per floppy and $50,000 per drive...
[dead]
It’s Y2K all over again. Signed? RUFKM?
In 1980, it was probably more important to be able to represent 1960 than 2040, so to me it makes sense they were signed.