Going in circles without a real-time clock
rachelbythebay.com> By "everything else", I also mean WireGuard. Did you know that if your machine gets far enough out of sync, that'll stop working, too? I had no idea that it apparently includes time in its crypto stuff, but what other explanation is there?
WireGuard only requires a monotonic clock, because it periodically rotates keys to provide the forward secrecy. Peer clocks are otherwise not required to be synchronized [1]. I guess the clock had a higher rate than usual and it couldn't be corrected due to the lack of RTC?
[1] https://www.wireguard.com/papers/wireguard.pdf#page=7 "In fact, it does not even have to be an accurate timestamp; it simply must be a per-peer monotonically increasing 96-bit number."
Why does it use the time then? Why not just increment its own 96 bit number whenever you use it?
Because it is required to be monotonic per peer. WireGuard has no intrinsic states, so multiple machines with the same peer key will be seen as a single peer and that is actually a legitimate use of WireGuard. These machines would have to be synchronized to each other (but not necessarily to the external clock), and using a time is a straightforward and reasonable way to ensure this.
So could you share a single set of keys on multiple machines and Wireguard will work, as long as all machines use the same ntp server?
Technically yes, but I don't think (but haven't exactly confirmed) that you can initiate a new session to itself.
The Pi ecosystem is coming apart at the seams. Talk about falling off a pedestal.
By the time you buy everything you need to make a Pi 5 even remotely usable and reliable today, you'll have spent as much as a small form factor PC. What do you need?
1. Special power supply - Yes, it's USB-C but it requires a high amperage 5V supply instead of accepting a higher voltage like most USB-C stuff
2. Active cooler - Pi 5 throttles immediately without it; it's not an optional component
3. NVMe Expansion HAT - Seriously why the fiddly little PCIe header? Put an NVMe slot on the bottom and let people adapt *that* connector
4. RTC Battery - At least put a damn capacitor on the board
5. Enclosure - I'm halfway expecting the pi foundation to print some dashed cut lines on the cardboard box so they can say it comes with a case =]
Not like the Pi Foundation couldn't have contributed to uboot or the upstream kernel or anything before thier hardware was released. Pretty much nothing released for "Raspberry Pi" runs on a Pi 5 at this point in time.
They better try to salvage something before putting out the expected CM5, but I'm not expecting much.
They seem to be obsessed with desktops, and ultra high performance homelab NASes.
They could have just use any old less powerful chip, like, whatever has at least 3B performance is fine, and used the money saved for stuff we actually need.
Maybe ditch those awful micro HDMIs, it's kind of a nasty connector and USB C exists.
Onboard lithium charging is a few cents. Better yet, onboard LTO charging, but it seems nobody does that yet. Why just stop at an RTC battery when you could have a few minutes of real backup power to help stop SD corruption?
While you're at it, add an onboard I2C FRAM or battery backed SRAM.
Instead of NVMe why not hire a few devs to go around fixing stuff that can't run well from the SD card?
> They seem to be obsessed with desktops, and ultra high performance homelab NASes.
I don't know how you reach that conclusion; the network connectivity of pi's suck (especially pi5) and out of the box they do not support any traditional storage except for a couple of external usb drives.
The use as a low cost desktop is a primary mission statement of theirs, so I don't really read that part as a criticism though. I think it's probably the only thing it really does well.
Agree on your other points though. Other vendors have made lots of headway in the SBC markets in the years that Pi hasn't been able to ship product. I'm sorry in a way that they have not achieved their lofty goals, but in another way I'm not sorry because they really have been fucking it up bad for the last few years.
The raspberry pi is still a great and easy way to control GPIO lines. IMO that's it's strength, it was never a particularly capable desktop computer or server for any kind of load. The raspberry pi was underpowered compared to a "real computer" when it came out and that's still fine for many applications.
A hundred and fifty bucks to control some non 5v tolerant gpio pins; no it’s not actually great at that. The RP2040 is great at that.
The power supply is becoming less special over time. My first result for "usb c power supply" on amazon is a $25 anker 317 that supports PPS and will do 5 amps at any voltage. Not that that's an amazing price, but the feature is spreading.
But I agree that the price to get a happy Pi is pretty harsh compared to a tiny PC.
I don't have Pi 5 to check this, but from what I've read, it does not support PPS. And such support is required; otherwise, even with a source that supports PPS, the connection will behave in a non-PPS fashion.
Apparently, cable quality is also demonstrating itself to be a significant factor in the Pi 5 power supply environment.
Regarding both these points:
What??
Then how the hell does it negotiate the power and know if it's attached to 3A or 5A?
Well, as it turns out, although a lot of stuff out there implies that the Pi uses USB-PD, it actually doesn't negotiate anything; it's just a dumb 5V peripheral. It can measure its own power (current/voltage) and has a current limit setting that the PMC will use to adjust clocks, etc. and alert on overcurrent conditions. If you hook up a higher amperage supply you have to change your boot config to the higher current limit. Everything is on you to make it work its weird not-quite-standards-compliant way.
The "Official Pi 5 USB-C Power Supply" is USB-PD compliant and as it is advertised as such in conjunction with the Pi5, I think that is where the confusion originates
I thought the boot config was just an override.
The things I've been searching in the last few minutes suggest that the Pi can negotiate 5A but only if the power supply explicitly offers it as a PDO, which almost nothing does.
But even if that's right, it's almost as bad as not doing PD at all.
Edit: The documentation states "If the Raspberry Pi 5 firmware detects a supported 5A-capable supply, it increases the USB current limit for peripherals to 1.6A, providing 5W of extra power for downstream USB devices, and 5W of extra onboard power budget."
It also confirms no PPS, but it's not entirely dumb.
That may indeed be correct at least in some circumstances; maybe it at least works with some capability of the offical supply. In my experience I have had to use usb_max_current_enable=1 and PSU_MAX_CURRENT=5000 to get it to shut up and work.
Here's the thing though; the simple fact that we have to even have this discussion because the Rpi5 continues to be super damn weird is enough evidence for me that the platform has jumped the shark. I don't know whether to blame Broadcom or the Pi Foundation, but the Pi 5 suuuuuucks. Man, I wanted to love it though; it's the only board I have even been been able to buy from them in the last FOUR YEARS
Agreed, I'm very annoyed it doesn't support PPS. A whole host of issues made me buy a small pc instead recently and the price difference was negligible.
What is "high" amperage here ?
I find it hard to believe that it can be worse than RPi 3, which could try to draw up to 15W => 3A under the highest loads, while USB 2 could officially only go up to 2.5 W = 0.5A at 5V, resulting in countless corrupted SD cards. (15 W being the minimum maximum for USB C.)
5V 5A. Buck converter? Never heard of it apparently
I have been happily using my Raspberry Pi 400 as my main home computer for years.
The tooling for light software development and hobby embedded hardware development has been great so far.
I ran into a similar problem on a Raspberry Pi, where my program would ignore data because it was deemed too old. It turns out a Pi uses fake-hwclock. This tool stores a timestamp regularly in a file. On boot the Pi reads from that file to set the time. When the Pi has been turned off for longer then the initial time can be hours or days out of sync. The Pi then takes a couple of seconds to obtain a valid time via NTP, but my program ran quickly enough to work with the old timestamp. After a couple of minutes it would typically work again.
It's unfortunate that using DHCP to get rough time isn't more popular, especially for devices without RTC. Sure DHCP can be spoofed and should be considered more of a last resort, but it sure would be better than nothing and ending up in this kind of a situation where nothing works anymore.
Previous discussion on similar thing: "Encrypted DNS + NTP = Deadlock" https://news.ycombinator.com/item?id=34177331
Speaking of RTC battery, I've recently come to the realization that I have to make sure that BIOS battery is not absolutely dead in always-on PC boxes.
Background: I use an x86 box as home router. I've changed the configuration in the BIOS that it should automatically boot up on power. However if the BIOS battery is dead, the config will be lost and it will revert to default settings, which is not to boot on power.
But there is no way to know how much juice is left in the BIOS battery, thus there is no way to issue warnings that the battery should be replaced soon. If there is power loss AND the battery is dead, when the power comes back up again, the router will just stay off indefinitely until it's manually turned on.
That's when I realized why most consumer routers do not feature RTC nor do they require batteries.
> But there is no way to know how much juice is left in the BIOS battery,
True, but its voltage can be used as an estimate. Most PC MB still around monitor the supply voltages, some also of the battery feeding the RTC. If you have the (Debian) package 'lm-sensors' installed, look for
in /var/log/syslog.sensors[<pid>]: Vbat: +2.91 V (min = +2.70 V, max = +3.63 V) ALARMRan `sensors` and it only shows CPU core temps. I guess my MB does not report RTC battery voltage?
One thing are the sensors, 'nother thing is the software. Have you configured the lm-sensors package (by running `sensors-detect`) yet?
https://hwmon.wiki.kernel.org/lm_sensors also has a FAQ.
Yes I did run `sensors-detect` previously and just to be sure ran it again. No voltage sensors on this box unfortunately :(
Some unsolicited solutions, maybe not for you but somebody in a similar situation: maybe replace it with a supercapacitor (needs some wiring changes, otherwise it won't ever be charged, should last long enough for most power outages), or use a stack of coin cells in parallel (difficult due to physical dimensions, and they'll still go dead at some point). You can also short two wires on the ATX supply to automatically turn on the computer when power is supplied (which may not be good enough if you have to press a key during boot to dismiss the low battery warning). Or hack the BIOS :)
If using supercapacitors, use some known not for leaking else you're back to square one (do a search for capacitor recap).
That issue doesn't apply to supercapacitors (nor to any capacitor manufactured in the last 15-20 years, really). It affected some aluminum electrolytic capacitors manufactured with a defective electrolyte in the early 2000s.
Somehow I feel reliability might take a hit… :|
Most newer systems persist EFI options to NVRAM now days as well. Pulling the battery isn't generally enough to reset them anymore.
My desktop remembers all settings, but after power loss still goes into a warning where I have to press F1 to enter settings, then exit them without making changes. I couldn't find an option to not do that.
Yes, this is very annoying.
Voltage check on battery is part of regular PC maintenance. Like cleaning dust, checking all fans are spinning, capacitors are not getting bigger, checking for weird sounds in PSU, overnight memtest...
You should do it every year or two...
Funny enough, instead of doing that every year, my policy has been "replace the machine if it breaks" and in general, I've gotten a good five-year cycle out of all my PCs without having to do disassemble-maintenance with an air can and a static strip. This has been good enough because I do enough high-graphics-demand gaming that five years is about the cycle on which some new-shiny has come out that renders my machine too old to play modern games.
Maybe I'm just lucky.
Cannot do that if it acts as the always-on router…
Companies have regularly scheduled (weekly in some cases I know) maintenance windows affecting thousands of people. A home environment with an "always-on" router can surely find a 10 min window (say, 3 AM on a Sunday morning) where nobody cares.
I know I’ve brought this up before, but I can’t remember exactly where.
It would be nice if DHCP could provide the current time as one of the pieces of information it provides with a lease. If you’re already trusting it for your IP address, you might as well trust it with the current UTC time: The current 64-bit timestamp would be enough to get timestamp-sensitive things working.
> If you’re already trusting it for your IP address, you might as well trust it with the current UTC time
I don't follow! How does trusting DHCP with IP address automatically mean I should trust it with the current UTC time?
Time requires higher degree of trust than IP.
I may not care what my IP address looks like but I might care a lot about what the current UTC time is and I might want this to come from a more trustworthy device.
You don't need to trust the dhcp time. But it would be useful for bootstrapping, especially devices without rtc. This does not really apply to your average PC which probably already has good guess on current time, those can simply just ignore the dhcp provided value.
The DHCP server provided value can not be much worse than 1970-01-01T00:00:00Z default value if you don't have any other data. And if you have some other data, e.g. ext4 superblock timestamps, you can pretty trivially protect against DHCP providing time from the past (i.e. use the maximum of different sources).
Finally, you can restrict the use of the dhcp provided time to the initial bootstrapping process only; it's not necessary to use it for system-wide clock
DHCP servers can command your dns config and hostname too. It's not a total mitm story since the certs are still on the machine itself but it's definitely more than just a local IP address.
The don't really 'command' it, they 'suggest'. The client's free to ignore those suggestions :)
To offer an example, if an attacker can manipulate the time then they can make the target accept expired or revoked cryptographic certificates, potentially enabling impersonation or man-in-the-middle attacks.
In contrast, a different IP than expected isn't such a big deal... Although it might break the collaboration of two computers as crude denial of service
> It would be nice if DHCP could provide the current time as one of the pieces of information it provides with a lease. [0]
DHCP can provide the IP address of one or more NTP servers to consult for time queries. I use this feature so that all of the machines on my LAN (at least the ones that can be configured to ask for and/or use this information) have the same general idea of what time it is.
[0] See subsection 8.3 here: <https://www.rfc-editor.org/rfc/rfc2132#section-8>
I do that, with a local GSP-synced Pi. Very cheap and easy to set up with IPv4 at least.
That said, if you don't trust the DHCP server because some evil actor might give you the wrong time, why should you trust the NTP server it tells you about?
I agree that if you distrust the network operator (but must use the operated network for some reason) that you should accept as little configuration information provided by that operator as is reasonably possible.
That goes without saying.
I mentioned the fact that DHCP can provide NTP server information as a way to explain why DHCP wouldn't just straight-up provide "current time" information. Why do that when you can point people to a server running a far superior purpose-built timekeeping protocol? It'd just be silly to do the worse thing.
(Notice also that RFC2132 was published in 1997. In my mind, this moots any retorts that a good reason for providing time directly would be because the DHCP server's supplicant is too underpowered to run an NTP client. In 2024? On a consumer-grade device? No way.)
Oh yeah, certainly agree. Especially since devices can use SNTP[1] to talk to a NTP server if they just want to know the current time and don't need all the complexity of NTP.
I trust a free wifi hotspot to give me an IP (traffic will be protected by TLS), but not with the time. E. g. time from the past can be used to trick into accepting an expired TLS certificate. But for most wifi hotspots it will be unintentionally wrong because such devices often unmonitored and not maintained until they completely break.
If your DHCP server supports bulk queries (RFC6926: https://www.rfc-editor.org/rfc/rfc6926.html ) then responses to such a query from a dhcp client include a timestamp in the form of an 32-bit integer count of seconds since the epoch.
I don’t know how widely this DHCP RFC is supported though.
It would be nice if DHCP (or maybe wifi?) could provide the location as well, so that you could relay the airplane's geo position to the connected clients.
I had this happen before. I used it as an excuse to learn how to setup NTP with a GPS receiver. I made a little blog on it if anyone is interested in the results. Be sure to click the sandwich menu for some real-time data: https://www.developerdan.com/ntp/
Quick question, because I've followed a number of how tos before that don't, if you disconnected all network connectivity, would it sync to the PPS source? Second, have you tested this, I only ask because 1# I found a lot of time sync was really slow, and #2 some of them reported that they had sub-microsecond time sync but when tested they were several milliseconds out.
Anyway, I solved these problems with sbts-aru, but maybe it's interesting for you to test if you haven't already done this.
Good question, it absolutely syncs without any network connectivity. For better or for worse, I have it setup NOT to sync to anything else- being a stratum 1 source is enough. Now… I will admit I’m just a hobbyist with this stuff. Anyways, the way I got around it trying to sync to another source when using the PPS was to define the GPS as two separate sources. One for the PPS signal and another for the NMEA sentences:
To get around issues with sync being slow, I have NTPD configured with some extra flags to allow large time jumps at startup:refclock pps ppspath /dev/gpspps0 prefer time1 0.004 minpoll 3 maxpoll 4 iburst refclock nmea baud 57600 time2 0.068 minpoll 3 maxpoll 4 prefer iburst
I have two raspberry pies setup this way- which I don’t use for much else. My primary computers have these two servers configured as time sources- but also have external sources- so as long as network connections are available, they can determine if they are providing a sane time or are false tickers. As for accuracy, you can view the ntpviz output for each setup in the hamburger menu. For catwoman, its regularly within 4 microseconds.NTPD_OPTS="-ggg -N"Ah okay. Because I found that the default start order of gpsmon and chrony was incorrect meaning that the shared memory communications wasn't working but there were no error messages stating this. So sometimes, it would sync (slowly) and then chrony sources would say that it was having microsecond sync time but when I connected a switch (On/Off type) to the same gpio on two separate Pi's with linked earth connections and had a python program output the system time in nano seconds I found that despite telling you it was keeping time accurate to microseconds it was in fact many milliseconds out. The only thing I can conclude is that despite the setup indicating that it should use the PPS source that use the interrupt, this was in fact not happening and it was somehow getting it's time from the serial line only. Without indicating that this was happening.
When I switched the startup of gpsmon and the chrony daemon to the other way around time sync was achieved within 1/2 minute to a minute (Real fast) and the reported time on two separate Pi's with the joined up gpio's (So you can get very close to triggering a time check at the same time) was then sometimes as low as 30 ns's from each other, which is damn good considering that most of the difference would be jitter.
And none of the how-to's that I read talked about changing the start order of chronyd and gpsmon so I can only assume they all might be suffering from the same problem. It's important to note that the diagnostic commands from chrony will not indicate this and the only way you will notice if this is happening if you setup a rig like I mentioned above. I noticed because my sound localization calculations were showing errors that didn't make sense.
I must admit I've never used chrony, so I'm unfamiliar with how to configure it. I've read a lot say saying gpsmon's shared-memory-segment is a really great interface and to go with it instead of NTP's own GPS drivers, but I cannot say that lines up with my experience. I do have one computer setup this way, and it works, but I found on the PPS configurations, it seemed to me to have better accuracy and less jitter using NTP's own GPS drivers and taking gpsmon out of the equation. Its quite possible I just don't know the right configuration combination, but ntpsec's own GPS drivers work great for me.
Ah okay. Conversely I've never tried that setup :) Nice to hear about it though.
This is off-topic, but I was looking at your sbts-aru project and remembered having read the hackaday post on it about fireworks. I'm curious if you've ever seen the RaspberryShake BOOM sensor? If so, any thoughts on how your project and it differ?
https://shop.raspberryshake.org/product/turnkey-iot-atmosphe...
I hadn’t seen that. It uses a pi so in principle it could sync the time the same way. How well related to localizable recordings would also depend on how it establishes the actual time arrival with the clock and how it deals with jitter.
Currently I only support USB mics at 16 bits that can talk to jackd. This was a conscious decision because it was all that was needed but also because jacks makes it essentially have multiple source sinks. Such as as well as recording a real time audio processing pipeline that potentially leads to real time gunshot and other sound event localization.
But it would be cool if the infrasound sensor had a USB sound interface. As then it could be useful in localizing elephants.
If it syncs time accurately by any method maybe you are able to localize some very interesting events! The localization code I provide with my project should fine with times obtained from this project. I seriously doubt whether that project uses a memory overlayFS but it’s to add as an improvement.
Interesting! What is the accuracy like when lacking "PPS precision"/Clayface? I'm wondering about the magnitude by which it is off (seconds, minutes, or other).
PPS turned up quickly on Wikipedia, so that one is readily answered
> PPS signals have an accuracy ranging from 12 picoseconds to a few microseconds per second, or 2.0 nanoseconds to a few milliseconds per day based on the resolution and accuracy of the device generating the signal.
You can see for yourself the level of accuracy in the ntpviz output. Notice the units of measurement on the graphs:
* Clayface: https://www.developerdan.com/ntp/#./clayface/7-days/
* Catwoman: https://www.developerdan.com/ntp/#./catwoman/7-days/
That Wikipedia quote should mention temperature! Temperature variations have a big impact at this level of accuracy. These really cheap GPS receivers do not have temperature adjusted clocks. Unfortunately my server closet (this is just a hobby) does not have well regulated temperature, so you can see the impact of temperature on the clock accuracy. Also, I found if I start running a bunch of stuff on these computers - that makes the CPU heat up, which also affects the jitter. If you really want high-precision, you'll have to shell out some extra cash then I did: https://www.sparkfun.com/products/18774
rtk gnsses really rock. I would to save up for one for next year :)
As for the GT-U7, this software can be used to avoid Windows, it seems:
> While not intended to be a direct replacement, the application supports most of the UBX configuration functionality in u-blox's Windows-only u-center © tool (only public-domain features are supported).
I'm going to have to check this out! Thank you for sharing!
I've personally had so many issues with SD card corruption on raspberry pi's that I've decided it was a mistake to even allow SD card disks
I only ever use USB storage on my pi's now and not had a single issue since
Most SD cards for sale are not suitable for write-intensive random-access workloads. However, they do make SD cards that are intended for this, I've had very good luck with cards marked "high-endurance." They cost more, but they work.
I've always been amazed that the Foundation never mandated the use of validated brands or types of SD cards.
Not that it matters much to me, I've had to retire all of my Pis earlier than the 4. It seems that all of my Raspberry Pis prior to the 4 all had some sort of design issue with their power regulation. After a couple years of constant operation, they just start locking up randomly around once every day or so. I've had everything from a Pi 1 through 3 exhibit this behavior. I spent WEEKS troubleshooting this and conclusively ruled out wonky SD cards, power supplies, connected devices, proximity to other equipment, and every other factor I can think of. I found many threads with other people reporting the same issues but the Foundation refuses to acknowledge it as a problem. I only run Pi 4's now, and am gradually switching over to Intel N100-based systems for low-power operation.
My money is on the use of a single USB 2 port for power for both of these issues :
Same with "real" servers. For a while there, it was trendy to boot ESXi from an SD card stuck to the system board and use all of the 2.5" disks for just data storage, no OS. It worked great... until it didn't. It turns out that writing logs and swap to an SD card roasts it after 2-4 years pretty consistently.
I wouldn't rule out running the OS off of that kind of storage if you had a read only root file system, but what madness would inspire you to write logs to it? And swap!?
The smart ESXi admins I knew used Compact Flash cards as the system boot disk. (It has been too long since I've been in touch with those guys to remember if they offloaded logs to local disks (or maybe a datastore on a SAN) or if they also wrote them to the CF device. I bet they did the former, but I no longer clearly remember.)
Using an SD card is just nuts.
Indeed, except if you run an in-memory overlay filesystem this might change dramatically for you. I have several Pi's that run for years and years without SD card corruption this way. Normally, however, as in my sbts-aru project, I don't just do that but also create separate data partitions that are read-write. It appears that even though there are some partitions on the card that are read-write, if they are not the system one it is also much more resilient to fatal corruption.
I heard that this is caused by not supplying enough power to the Pi, did you use the official powerbrick? I have had corruptions happen a couple of times but I have never tried with a good powersource if it actually does a difference.
SD cards are not made to run an OS from. They are quite limited in read/write cycles .
Using SD cards for an OS is a really bad idea in general. You may make it work with a read only filesystem, but if you don't sooner or later you will have issues.
I think the last paragraph was my favorite part of this article.
For those of you who didn't read the fine article, I've quoted it here:
> As usual, this post is not a request for THE ONE to show up. If you are THE ONE, you don't make mistakes. We know. Shut up and go away.
I dunno, I've always found that to be a... less impressive part of the blog. When the author hits problems in her stack, anyone who would have caught it should shut up and stop being THE ONE. Fine, we've all run into people with big skills and bigger egos. But... the other half of her blog posts are her making snide remarks about how stupidly other people have set stuff up. To her (ex-)coworkers, she is THE ONE. So it leaves a sour taste in my mouth.
> I figured they must be running DNSSEC on that zone (or some part of it), and it must have a "not-before" constraint
Since clients will attempt to resolve ntp.org in order to actually sync their clock, there is a good probability that some clients will be way off.
Enabling dnssec on that zone was probably not without important drawbacks? I wonder if the operators thought about that potential pitfall. Seems like they might be doing a disservice to their core mission of allowing devices to sync their clock.
It feels like enabling DNSSEC on any zone is inviting a bunch of fun service risks and unexpected failures.
Is my clock more likely to be accurate after NTP.org got signed? It looks like it's less likely, on average.
If you _don't_ enable dnssec on ntp.org, then a mitm can intercept the dns request to redirect to an attacker-owned timeserver with a time in the past. Then the host can have old and expired (without loss of generality) keys/certificates replayed against it.
If memory serves though, Raspbian used to not even have `fake-hwclock` by default and even more Pis would end up with this "wildly wrong time near the epoch and can't bootstrap DNS/NTP" failure mode. You'd sometimes also see it with VMs too (esp if they were doing a pure-software clock that ran slower than realtime, instead of patching clock calls to the real hardware clock on the host).
> If you _don't_ enable dnssec on ntp.org, then a mitm can intercept the dns request to redirect to an attacker-owned timeserver with a time in the past. Then the host can have old and expired (without loss of generality) keys/certificates replayed against it.
Preventing DNS mitm only matters if you prevent NTP mitm too.
What percent of NTP clients talking to these servers are doing it in a secure way? And is that share growing?
If you don't have an RTC I'd recommend having a tlsdate with some bounding heuristics to prevent extreme clock fixation from a mitm. You can relatively cheaply hit a large number of public servers that are likely to have good times available over TLS and trust the common result. You validate the certs without considering the notbefore stamps and then if you're feeling aggressive validate them after you've managed to approximate a date from the cohort. I know there are commercial packages that do this, I'm not sure about OSS ones.
Roughtime would be far better, but essentially there's no broad deployment of it yet.
Ideally something good would be picked by Raspbian and delivered in the distro as standard.
It's surprising that it was left out. Ie, the workaround makes sense, but this is a peripheral that is sometimes built into MCUs themselves. For example, most (all?) STM32s include an RTC... with the caveat that if you are using them for canonical use cases, you will need external hardware in the form of a dedicated 32kHz oscillator, and possibly a battery.
For a lot of micro-controller timing uses, they aren't required; for example, hardware timers based on the primary clock source, or the Cortex-M systick, but for maintaining accurate dates and times over long periods, the RTC is the right tool. It can also output the dates and times in a convenient format as well, as long as you find named register fields convenient!
An RTC really needs a battery, and even a smaller battery like a 1220 takes up a fair bit of board space -- space that could be used for another connector or more functionality. And even if you have an RTC, you still have to handle the edge case where the battery dies or is removed and the clock resets. Critical low-level internet services probably shouldn't require clients to already have the correct time.
I use an external RTC with a Raspberry Pi for some sensor systems I work with. Its purpose is to give a reasonable chance of having the correct time for recording data in remote locations where no internet connection is available at startup. For a Pi that only provides network services (and thus always needs a working internet connection), I probably wouldn't think to bother.
I had to support an IoT device once that lacked an RTC.
That was actually the easier hardware platform, because it either had good NTP sync or it didn't reach our C&C servers.
The difficult one was the platform that had an RTC. Those would slooowly drift out of sync if NTP was broken. Slowly and silently.
RTC is just an electronically readable Casio watch, it's not even responsible for keeping real time while OS is running. All it does is always incrementing regardless of motherboard power state so to provide time as OS loads.
Install sbts-aru as a base system and it will keep your Pi accuracy to sub-microsecond error.
https://github.com/hcfman/sbts-aru
Problem solved.
Or... Install one of these on a Pi somewhere in your network and use that as a time source for all of your devices. Problem solved for all your machines.
So you know, sbts-aru is an audio recording program. However, in one command it sets an an overlayFS protected SD card setup and chrony GPS based time synchronization. You can turn off all the recording related stuff and then you have a stratum #1 time server for your network. Real easy, real cheap. No more time problems.
There's no reason now for all geeks not to have at least one stratum #1 time server in their networks. Really none.
IF you get a DCF77 (stratum 0) signal, there are receivers for Pi below 10€.
Getting time from DCF77 seems like the best solution for timekeeping on a lot of computers, but there don't seem to be nicely packaged ones over USB etc. I had a quick look for Pi receivers and none of them looked to be nicely packaged, but with an external antenna.
If you are running Bind or SSL/x.509 certs, than an RTC is pretty much a requirement on the raspberry pi. Network time leaps over NTP even when working (first big leap was allowed etc.) can still bork a lot of services.
Local stratum 1 GPS time (even with PPS pin) servers based on Pi's can suffer similar issues when they lose satellite lock (weather etc.)
Have a great day =)
Having said that, I've never personally seen my stratum #1 Pi loose sync due to weather and I run about 4 of em here. I also use a cheap 7 euro ublox 7m. So long as you position it or an antennae with a reasonable view of the sky that shouldn't happen I reckon.
Why all the complex solutions when it's so easy and cheap now to install a stratum #1 time server, even a Pi 3 will suffice. It works on a Pi Zero, but then you only have a wifi link.
The sbts-au project make's it incredibly easy to setup. git lone and then install. One command, it's that simple.
Any advice for a cheap USB GPS for a stratum #1 time server?
EDIT: Any any advice on how essential PPS is?
The PPS can be good (pi kernel support is stable), but one needs to be careful as some modules PPS signal has a great deal of instability.
Some U-Blox modules support the additional ground based navigation signal and cellular beacons. This is recommended in addition to the 3 satellite networks classes, but note some modules will not work with some antennas (get the unified hardware with built in antenna/lna if unsure about RF notes).
Best regards, =)
If you have good sky visibility, then I don't need an antenna. Though this often tends to mean putting a Pi outside. I'm also using just the first Antenna I find actually.
I have good results with cheap ublox 6m's and 7m's from aliexpress. For some reason the 8m's I got were poor in sensitivity.
Adafruit ones also work great, but are a lot mor expensive, I do have several though.
PPS is criticial to proper timing. However... I was using the time synched systems for sound localization. But as they only cost around 7 euros on aliexpress, why not.
I’m surprised tptacek hasn’t shown up to defend DNSSEC from this libel
DNSSEC is doing a fine job of representing itself here.
The wireguard problem is a pain in the neck. Even if you're happy for your system to not have a realtime clock, when it does come online you'd want wireguard to not start until after the clock has synced to a timeserver. The below is what I'm doing at the moment, but I can't say I'm sure it's working - haven't seen wireguard start before the time is synced since doing it, but it could be probabilistic:
systemctl enable systemd-time-wait-sync.service
mkdir -p /etc/systemd/system/wg-quick@wg0.service.d/
echo "[Unit]
After=time-sync.target
Wants=time-sync.target
" > /etc/systemd/system/wg-quick@wg0.service.d/override.conf
Well... I see where you are going with this but is it necessary? What happens if the clock gets it's correct time sync back, will not wireguard come back online by itself ?
Then the real solution might involve ensuring good clock sync. Using a GPS synched time source is good for this, such as this one :) https://github.com/hcfman/sbts-aru
No, I can't say I understand it fully - but if the clock goes backwards, peers will not talk to a client again until I restart the other peers' wireguard services. No matter if the client's clock is subsequently correct.
For some people, all their network access goes over wireguard and that includes time syncing. That's not the case for me, but for those people if wireguard isn't working their clock can't sync.
That GPS receiver looks cool! However an RTC and a battery will do just fine for me, which is the plan.
That was my thought as well - if we know certain things need time to be in sync, can we use systemd dependencies to enforce that? I'm not sure it would help with OP's problem because that was more or less a circular dependency (time sync needed name lookup which needed time to be in sync), but it could mitigate some of the effects or at worst make the problem more explicit/obvious.
What is she referring to by “THE ONE”?
It's pre-empting the sibling comment saying "RTC module is $1"
Amazing. So when you read about some weird (and mostly self-inflicted) problem you've never had in your life, you are not allowed to express your surprise because by doing that you're implying you're better than other people, and that would be horrible (the implying part is what is horrible, I think? Or maybe actually being better is also wrong? Can you even be better―yeah, I guess it's possible to be better than someone else at something).
You're totally allowed to express your surprise, you're totally allowed to be a condescending jerk while doing it, you're totally allowed to call it "self-inflicted".
The author is allowed to ask you to shut up and go away if you do.
You're allowed to. We are then allowed to think you are an asshat for doing it.
Sharing cautionary stories is good. Learning from others' mistakes is good. Jumping on these as an opportunity to put someone down for the purpose of self-promotion ("mostly self-inflicted", express your surprise) is unkind, probably bullshit (I have seen soooo many people pretend to be above problems and then run smack into them), but most importantly: it shits up the dynamic where people are willing to share their mistakes, allowing group-learning. It would have been the easiest thing in the world for rachelbythebay to not make this blogpost and thereby avoid providing an opening for asshats like yourself, but I'm glad she made it anyway, even knowing that you would take your shot, so that everyone else could learn.
Found the one!
Sorry about wasting your time. This is not for you.
You can be better and nice.
The know it all. This “wouldn’t have happened to them because they are” THE ONE. Also known as, “that guy” as in, “don’t be THAT GUY.”
As sibling comment somewhat points out, she is literally referencing The One (2001)
Unfortunately a Raspberry Pi is a bit ill suited for production environments. Id recommend an RTC module. Otherwise this might be helpful: https://github.com/hcfman/sbts-aru
Tangential: I have a pi 4 running openwrt and a wireguard interface that is routed over the WiFi access point. If you connect to this network it's as if you are connecting in another country. I have outbound traffic only going over wireguard. If wireguard is down then no internet (to avoid any leakage). However if pi loses power the clock resets, wireguard can't handshake with that much clock skew, NTP can't connect without wireguard. So I need to manually update clock after a power failure.
I guess I need requests to the NTP server to go over WAN directly. Always seemed like a hassle to get this to work with openwrt zones and stuff. My eternal gratitude to anyone who shoves me in the right direction...
If you “always” have Internet access, you should really just place ntpd onto the physical network where it belongs. Otherwise, the options are all mentioned in the article (and might be used in your distribution).
— Restoring more sane shutdown timestamp from file instead of just using 1980.
— Making blind single use ntpdate request to some public server to set local time after the network is available, but before any other application uses it. There might be appropriate hooks in your init system if you don't want to make it an independent step.
— The same, but using some local time source. It is not widely known that even plain old Windows desktops, while not having a complete NTP server, can be trivially configured to announce time over some form of SNTP. If you have any computer with a real clock running nearby, you can probably rely on it. Next step router is also a good potential candidate.
As for compartments and access control, you can try to use basic VLANs (one for ntp traffic, and only it, and one for VPN) or maybe some two-step network configuration process (get time with one configuration, reset and proceed with other services).
All of that is not a problem, the problem is what to do when something fails. Time server becomes unavailable, intermittent internet connectivity, server you use for pings is available, but others are not, etc. You can choose to stop completely if there is no time source, and wait, but that might render the system unavailable for remote configuration. You can try to restart the device or restart the network router automatically (though only a fixed number of times, restart loop will surely break something), but that won't help if the problem gets fixed some time later. You can also use a fallback configuration that doesn't depend on correct time (e. g. only start an ssh server so you can connect to it).
> I guess I need requests to the NTP server to go over WAN directly. ... My eternal gratitude to anyone who shoves me in the right direction...
Do you have something on the LAN side that could run an NTP server? I'm thinking you could add your laptop to your Pi's list of NTP servers, then just start that service up temporarily as-needed to give the Pi a time that's good enough, then it can get actual correct time sync from somewhere else once Wireguard is working.
I also use wireguard a lot. I didn't know that clock sensitivity could be an issue here. I've never seen an issue before. Now that I know it could be one I'll make a point of deploying all cellular based systems with a GPS synched time source as well and avoid that sort of problem.
Yeah you got it. Set the listening interface to eth0 or whatever the wan interface is. Might need to get full ntpd instead of the busy box version. The Interface option here: https://www.ntp.org/documentation/4.2.8-series/miscopt/
I’m actually setting up a router with a rpi4 right now, only using Fedora IoT instead of OpenWrt. It’s a bit more assembly required lol
I need to have a play with it, but the problem is i deliberately have a 'kill switch' style setup to stop any requests accidentally going not over wireguard (for reasons) so I assume I'll need to do some other firewall type thing. Maybe something something tagging.
I’m not sure this is related or not but if there’s one thing I’ve learned about complex objects (engines, computers, software, sewing machines, etc), it’s that every _custom_ thing you do to it has some kind of order of magnitude impact on the overall complexity of the machine, and thus, you should avoid customizing it. At least if your primary concern is usability vs solving specific problems.
It’s definitely served me well in a wide variety of disciplines.
There is not much of an allowance any more for devices that are not on the internet 24/7 without fail.
Multi-booting PCs with Windows and Linux over the years, I have seen the time sync problem go from nonexistent to show-stopper.
For devices that you only need to connect to the internet occasionally or sporadically (so that's what you do), that's where I noticed it most.
Linux sets the RTC to UTC, then when you reboot to Windows, Windows uses the RTC as local time. Which for me is 5 or 6 hours different from UTC, depending on Daylight Savings Time.
Plus when Daylight Savings comes around, each OS wants to make a 1 hour correction the first time you boot it after that date. With multi-booting this can add up to more than one hour difference too.
Didn't used to be so bad, if you were a few hours (days or longer for many PCs) away from the actual time, things did not fail for this reason.
Eventually only one hour off was OK for a while, now nope.
There are just so many more obstacles to smooth reliable operation, and weak links in a more extensive chain that must remain perfectly strong. Otherwise the chain is broken, you are disconnected, and the weak link is too many sections away from you now to be within reach.
Complete perfection is required more so than ever, while at the same time, your efforts to approach perfection are being made more difficult.
Set Linux to interpret RTC as local time, or set Windows to interpret RTC as UTC (possibly better, as it won't adjust to DST). https://askubuntu.com/a/1377011
In OpenWrt (Linux distro for routers that often don't have RTC) dnsmasq starts with `--dnssec-no-timecheck` until the ntp client gets a first sync (https://github.com/openwrt/openwrt/commit/5acfe55d7139a52941...)
Two nights ago I rebooted my router and it wasn't able to update its clock. The logs were filled with "ntp: start NTP update" every 5 seconds.
Well, it was something related to using AdGuard's (94.140.14.14) DNS and pool.ntp.org. I added Google's (8.8.8.8) to resolv.conf temporarily to get the clock synced. But I'm still not sure what the root cause was.
Its weird that people work OK without knowing exactly what time it is. Sure it helps us but if a person lost track of the date for a few days in a city they would still be able to perform basic functions. We only have had timekeeping devices for a tiny part of our history but we invented alot of stuff without precise synchrony.
Isn’t that just because individual human activities aren’t very precise, and for the most part, the position of the sun in the sky is enough for an individual to broadly synchronise themselves with the rest of society.
Unless someone is actually aiming to do a time coordinated task with others, you really need to know the time, or even the day-of-week to do stuff, because most human things in world aren’t actually occurring in small time sensitive windows.
How did they do sound localization then in those times ???
My raspberry pi is also currently dead, something it didn't like about updating the OS caused it to be bricked. All updates were done within the OS, so didn't expect anything bad to happen.
I could reflash the OS, but I also "don't have the time". I'll just use my laptop until I do have the time someday to resurrect it.
Quite likely installing the sbts-aru as a base project would have prevented this as well. Loads of SD cards get corrupted historically without apparent good reason. Using a memory-based overlayFS file system solves so many of these problems. sbts-aru, as well as setting up a stratum #1 time server sets up a resilient memory overlayFS as a base.
I ran into this before! https://medium.com/@cyounkins/encrypted-dns-ntp-deadlock-9e3...
Note that the Pi 5 has a real time clock - you just need to add a small battery and it will keep the time when powered off: https://picockpit.com/raspberry-pi/raspberry-pi-5-has-a-real...
I guess that's what the article meant here: "This particular Pi doesn't have a real-time clock. The very newest ones (5B) do, but you have to actually buy a battery and connect it. "
tl;dr: machine is unable synchronize its clock because it won't trust the NTP server's DNSSEC certificate because its clock isn't synchronized. Oof.
This is the perennial problem with trying to secure NTP. I've also seen people suggest that we just use HTTPS for time sync, not realizing that if your clocks are too far off you can't make a HTTPS connection.
It takes more creativity, but you can start to make a narrowing range of times if you're careful.
You'll have to ignore DNSSEC time based validation while you start up, of course... Start with the last time you were properly synced: it's most likely less than a month behind then.
Then take a sampling of certificates offered from well known sites. If the certificates validate, other than time, no reasonable CA that you trust issues not-before dates in the future, so it's not before the most recent not-before date in a certificate. It's probably not much after the not-after dates either, well know sites tend to replace their certificates (but maybe you're getting MITMed with a compromised, old certificate).
Maybe ask for OSCP stapling, which should get you a more narrow range of times. I think OSCP responses are valid for a week? But if you get several from different https servers, chances are good you'll narrow the range.
In the case as suggested that the time was only off by about a day, IMHO, something is seriously wrong if it can't get to sync from there, but I guess I'm not really surprised either.
You can use plain HTTP for time sync. Almost all HTTP servers respond with a time header.
The whole point of the exercise was to make it secure though. If you don't care about MITM attackers then NTP works great.
I don't get the supposed security aspect of getting false time. Hacker gives you 1980-01-01 or 4096-13-32 and mess up CRL and ruin your day...how.
Years ago I've tried privilege escalation exploit to play with a phone and it involved rolling back date to unexpire signature, so I know there is exploit potential, but it... it just feels like RTC bootstrap problem should be something solvable.
Isn't that a provably pointless exercise though?
Security protocols (at least the ones in common use) require certificates or keys that eventually expire, because of the risk of a permanent key being compromised. If they expire, the protocol needs time. QED.
https://github.com/systemd/systemd/commit/abf4e5c1d3ad767bc0...
While systemd-timesyncd contains a workaround for this, chrony does not seem to.
Yeah in hindsight it might not be the best idea to enable this for ntp.org, especially when the ntp protocol itself isn't secure.
RTC module for Raspberry is ~ $1.
Right, it's only $1.
Plus the time to learn a few modprobe commands. Perhaps run a few commands echoing numbers to random sysfs files. Perhaps learn WTF a 'Device Tree Overlay' is - a concept that's completely absent on x86 systems. You see, this is an 'I2C' device, so it doesn't plug and play like USB and PCIe devices do. It is much simpler, and hence for some reason more difficult. Perhaps your device tree overlay needs to be complied? Or maybe not. Perhaps you'll edit your modules file and your kernel command line. Of course that won't involve update-grub, why would you think that?
Don't forget to periodically check on your settings, just in case an OS update or something has disabled one of those settings.
Oh and of course you won't be able to fully test this over SSH, if the network is up NTP will have set your clock whether the RTC is set up right or not. As we've learned from the article, if you run into clock problems you might not be able to connect remotely.
Then simply invent a time machine so you can do all of this in the past, before the pi SD card fails and it drops off the network.
If the user has issues with all of that, then RPi is not the rigth tool for them.
Ah yes the Raspberry Pi, a charitable project intended to get young kids into coding, which comes with kid-friendly projects, brightly coloured beginners' guide books, and pre-installed with Scratch, is truly only a tool for experienced professionals.
/s
Yes, if he/she was using it for that (kid projects, Scratch, etc.) it would have been the right tool. Maybe it's because she skipped all those steps in his/her IT education is the reason he/she is having troubles now.
I learned about the devicetree on a RPi too, but I didn't complain about it in a blog. When I bought my first RPi I expected to encounter problems and learn a lot from solving them. In fact, I expected it to be much much worse - so much worse that I even bought the RTC module even before the RPi arrived in the mail. Why didn't he/she think of that? It's because he/she is lacking the basics, the kid project with gpio leds and i2c sensors.
and you certaintly get what you pay for, DS3231s fry almost as easily as MAX3232s,
(And you can do the same thing were you write the time to the RTC before you have the correct time)
I have used these in product designs before with zero failures. Where exactly did you buy your samples?
We found they drift less than 28s a year in standalone mode. Also, were repeatable within +-18ms when using a NTP client.
Microchip RTCs are harder to setup, but also seem to work just as well (ignoring the goofy epoch).
Chip shortage caused people to do unspeakable things.... ;-)
You could buy an rtc and put up with that kind of clock drift, or.... for the same money or less you could be a GPS and have none of that drift.
(a bit more money for the GPS to be fair) not even mentioning the fact that you should have clear-ish access to the sky, the time taken for you to receive the time data, and the possibility of jamming. I could go on but the two methods have differing modes of operation and failure despite ultimately being capable of fulfilling the need for a reasonably synchronized clock
I have a DS1307 installed ever since I bought a RPi3B+. It didn't fail and it's accurate enough until NTP updates the time.
Are you sure you connected it correctly?