Navigating the timezone nightmare in product development
getlago.comDate and time handling in software is one of the most dramatic examples of software being a means of communication between humans and machines. What appears to be a seemingly basic physical process turns out to demand reams of code to manage leap years, leap seconds, time zones, the birth of Christ and that of Unix, Gregorian and Julian calendars, and worst of all: daylight saving time (DST).
DST was once supposed to save energy, but my own personal frustration with it alone could warm the Earth's atmosphere by a whole degree Celsius.
Most of confusion can be resolved by introducing 3 separate entities (and naming them properly):
- Instant - a point in monotonously increasing time (Unix timestamp is usually a good enough approximation for most cases)
- Calendar
- Wallclock - non-monotonous human-readable clock
Being precise about which entity is used is enough to resolve most of the confusions.
For example, scheduling a doctor appointment is always Calender + Wallclock in a location of doctor's office. No Instant should be used to record this, and changing of DST rules for this location shouldn't affect the appointment. DST rules only matter if we want to convert from one Calendar and Wallclock to the other, then Instant may be used internally in calculations.
Back to the article: if we are talking about subscriptions, then we should specify, if subscription is valid for time interval of 365*24*60*60 seconds or the expiry is tied to Calendar+Wallclock, but in this case Calendar+Wallclock should have a specific location. The first option is much easier to implement, and to avoid any timezone/DST frustration adding an extra day of subscription (366 instead of 365) will be enough.
I think in most systems, calendar is called LocalDate and Wallclock is either LocalTime or combined with a LocalDate into a LocalDateTime.
I remember having a bit of fun computing the number of work hours between two dates.
It involved:
- the day of the week (office closed on weekends)
- day of the year, with leap years (office closed during holidays)
- Easter day (some holidays are based on Easter)
- DST and timezones (office hours are in local time, timestamps are UTC)
Thankfully, no Julian calendar or leap seconds to deal with.
Many problems would be solved by using TAI for storage instead of timezones/UTC. At least for past events.
Save the planet, remove daylight saving
This is a topic I often see overlooked from the user perspective and I'm happy to see more discussion about this.
My anecdote: when I was working for a B2C startup we had to ensure we billed customers on the correct date, like OP. Timezones are hard; dates are easy (or so we thought). When we billed a customer on their billing date we had to attempt to take payment at a time, which meant that our naive date handling converted that date into midnight UTC on that date, causing many western European customers to be billed at 23:00 or earlier on the previous day from their perspective.
Furthermore, from the operations side we were often dealing things that happened "on a date". I was pushing for at least our internal customer service systems to present two timestamps to agents: the date and time at which the thing occurred in the place that event occurred and also the date and time at which the thing occurred in the customer's location. For example, something being changed about a customer's account at 23:55 in London on a Monday actually happened at 00:55 on Tuesday for the customer in France. However, the timezone information was either not stored or not presented, interfaces were not consistent, and the result was pot luck whether the customer or a member of the customer service team would see Monday or Tuesday.
Timezones are hard. Presenting that information in a contextually appropriate way to your own employees and customers can be just as hard.
I like that the authors of the article have reached similar conclusions, especially around "dates have timezones". I think datetime capture and presentation is a fantastic UX topic.
Another detail about dates people often forget or don't know (though it's not usually important), is that a "date" can be 50 hours long. Let's say you want to have some sale to occur all day on Christmas, midnight to midnight in your customer's local time.
For easier math let's say your business is in GMT tz. People in the Line Islands (Pacific/Kiritimati) will have access to the sale 14 hours before yourself. Then the sale-time occurs in GMT, 24 hours pass, and the sale ends in GMT. But people in American Somoa (Pacific/Midway) will still have the sale active for 11 hours. (Technically someone using satellite internet in the open ocean east of Samoa has 12 hours, but the UTC+12 timezone is completely uninhabited).
Such a sale is kind of a farfetched example, but I've encountered this issue when trying to do analytics reports looking at user data that happens on specific global holidays.
And when you interact with people 8 time zones away, concepts like "today" or "tomorrow" break down completely. It's a very uncanny feeling.
I remember having a call with two colleagues once, I was in Sydney, one colleague was in london and one in LA
It was quite surreal, although time wise I think the LA person was up early on Tuesday and I was up late.
One member of my D&D group moved to Denmark, one moved to Tokyo. The rest of us are on the US east coast. We have the hang of it now, but for our first few online sessions there was always someone who missed it because they showed up a full day late or early.
It's also funny how some of the group is having coffee and breakfast while other members are getting drunk and having midnight snacks.
On the project I'm on now, we have regular weekly meetings like that. One guy in the UK, one guy in the DC area, me in Texas, another guy in Seattle, and another guy in Sydney. Plus possibly some other people that might also join from some of those same timezones.
I now have an app that runs continuously in my menu bar to track all the different time zones that I care about. Tel Aviv and Tokyo also figure into the equation sometimes.
> the UTC+12 timezone is completely uninhabited
New Zealand would like a word.
If it was a burn, it was pretty hot.
The thing with dates depends on the context though. A birthday for example isn't associated with a timezone.
That's true, but wishing a birthday at the right moment depends on the timezone. It's not just about the date itself, it's about the interactions other people have with this date
Completely true. I think the main point of your comment is about "presenting that information contextually to customers". You often have to make choices when you decide to bill your customers, and those choices have to be crystal-clear to your end users. When I was working for a neo-bank, 50% of the complains was about billing not being clear, and timezones was one of the top reasons.
When I worked at an online travel agency building our iOS app, dates were a giant pain in the ass. What is today? What is tomorrow? I covered it at https://thecodist.com/what_time_is_tomorrow_tales_from_the_t...
Common mistake that I don't think the author got right either, is that if you want something to happen at a future local time time, say 2PM in São Paulo at some date next year, you don't use UTC + a timezone, you use local time + timezone.
Timezones are not immutable, and timezone database updates happen every year. Only if you track dates that happen in the past UTC makes sense.
In some cases/jurisdictions they change DST quite late so you'll only know with certainty what UTC time corresponds with what local time a few weeks in advance.
Yeah, I often like to emphasize that many "scheduled events" are not simple numbers, but pattern-matching conditions, a contract of triggering-rules for something an unknowable number of elapsed seconds from "now."
To be specific, it's "whenever the desk-calendars and wall-clocks in that jurisdiction will say X because the local government says so."
The government of Bizarro São Paulo could decide to pause the clocks at 1:00PM, wait for one sunset and one sunrise, then set their clocks to 12:00PM, and then advance it by one "minute" every time a rooster crows until it reaches 12:10pm at which point they skip straight to 3:00pm.
When is your schedule your 2PM wall-clock meeting? Probably around the tenth rooster-crow. If you had reason to distrust the clock-culture of Bizarro São Paulo, you shoulda made it a different timezone. :P
To elaborate further on your point, the key takeaway is that you can not, in general, predict what second a future date will occur because dates are societal and civil constructs meant for coordinating human activity.
Counterintuitively, this is not true for the past and present. A specific second is unique. As long as you know the second you can derive the exact date (in the past) it occurred in whatever civil time standard you are using. The past and present should always be stored in TAI. The future should usually be stored as “pattern-matching conditions” (though you can use TAI if you are recording a elapsed time/seconds until the event).
Can off-the-shelf libraries translate from any Bizarro time to UTC?
Not in advance. Or better, they can translate it in advance, but they will translate to an incorrect value.
So what do you do, record both the local time and UTC?
For future events, you need to store at minimum the local time; of course, that's not super useful to use, so you may also want to store the UTC as determined by the best current information when it was saved.
Whenever you get new rules, you need to look at at least all future events and see if their UTC offset changed and then decide what to do about it. Often the right thing to do is just adjust the UTC offset based on the new rules; but sometimes you might want to ask the user, or maybe just notify the user: hey, we got new DST rules, please confirm these modified events: [list]. Users don't always input times accurately[1], so a heads up lets them review the modified dates and update the events that were modified in a way that doesn't reflect their actual intent.
You might also want to look at past events and maybe also notify users that the rules changed, and you're sorry about any inconvenience because you didn't get the rules or didn't process them until after the event.
[1] If I'm planning to watch a sporting event on TV, I'm most likely to input it into my calendar in my local time, but the event is most likely scheduled in local time at the event. But not necessarily. IMHO, the closer to the time change the event is, the more important it is to get user feedback.
IMO the way to avoid insanity is explicit modeling, the pain comes from where different code was written with different beliefs about what the value "really means."
For example, if there's a lunchtime event in Bizarro São Paulo you record "Noon in Bizarro Brazil Time" or even ("Noon in whatever timezone that office building will exist in") because it really does depend on the rhythm of the city. In contrast, if it's an international video-conference, you might record it as a time in UTC or the timezone/city of the most-critical participants.
In both cases, users of the system should be able to see the "contract" along with the predicted moment in a probably-useful local representation. (I might not be in Bizarro São Paulo while I'm looking at the event, but I might know I'll be flying there later...)
If you're afraid a Bizarro-govt is going to do something weird at the last second and cause all sorts of customer complaints for missed events, you might code a system that keeps track of when predicted-times (in something safer like UTC or TAI) suddenly change for events in a way that might catch users by surprise, and then send out notices.
But also "whatever timezone that office building will exist in" may not actually be the same as the timezone that is actually used by the occupants of that office building. There are numerous communities which unofficially observe a different timezone to the one officially assigned to their locality.
You record a context sensitive date/condition based on the user intent (i.e. birthday travels with the user, in person meeting based on meeting location, international call based on local time or UTC, explicit timezone keeps the timezone, etc.). How you map user intent to recorded date is problem specific.
When the event happens you record the TAI. You may also wish to record the context sensitive date for auditing purposes, but it is not strictly necessary. You can rederive the occurred date if the event occurred in the past.
https://mm.icann.org/pipermail/tz/2020-October/029376.html
For example - 5 days notice for a DST change.
You use local time + location, because tzdata timezones can and do get split.
Local + Timezone is still going to be wrong if the people are viewing/planning this scheduled event are in different timezones and the timezone DB makes changes to either the viewers' or creator's timezone.
In my opinion, it is better to just store UTC when you can't possibly make the date right in advance for all users in all timezones anyway.
And if everyone does this, hopefully it discourages government from changing the timezone DB frivolously and without spending a lot of money broadcasting the changes to anyone that may be affected.
I think this case is sufficiently complicated that we can't simply declare that the author got this wrong and that you should always use local time + timezone. What if that local time zone changes its DST rules to completely skip over that 2pm? Can we really confidently say that the user's intention when scheduling that event was that the event should be skipped if such a time zone change occurs between now and then?
> Time Zones often have a "friendly name" in the format Continent/City|Island
Continent/City are not friendly names, they are political boundaries that gice more precision. For example, CET covers many countries that might decide to stop applying DST at different points in time. If you store CET in your database you won't know if you need to apply CEST or not. This is why you always need the Continent/City form if you want to be future proof (and past proof)
If you want to be future proof, you need to store locations, not timezone identifiers. The timezone database does not claim that its currently defined timezones are controlled by a single entity (i.e. one timezone can span multiple governments with timekeeping authority), nor could it possibly guarantee that in the face of changing borders and laws.
Time in the U.K. is europe/london. It’s entirely possible in the future that outlying Scottish islands may decide to keep DST while the rest of the country goes to year round daylight saving.
You can’t always be future proof.
good to know, thanks
Political friendliness (n.)
> And another fun fact (We had a lot of fun!): Time Zones often have a "friendly name" in the format Continent/City|Island except for UTC and... GMT+12, which only covers two uninhabited American islands :D
There's quite a few other non-slash-having names out there:
ls -pL /usr/share/zoneinfo | grep -v '/'
https://github.com/eggert/tz/blob/c3e966c59b02b1f47f0b7b0e4a...
The only other timezone that currently has a non-1h offset for DST -- Ireland's is -1 hours -- is Australia/Lord_Howe, which has a 30-min positive leap.
Edit: there are also tzdb entries of the form "a/b/c" -- America/Kentucky/Louisville --- though they are not commonly used in practice.
> GMT+12, which only covers two uninhabited American islands
New Zealand users are now mad at you :D
There is a lot of interesting history in that file.
I recently joined a small online forum from my country. They had some people write the whole thing from scratch more or less.
I happen to be traveling currently.
When I log into the forum from my location, which is one hour away time zone wise from the country the forum is from, and I make a post, I then see the timestamp for my own post presented as “in 59 minutes”.
I reported the bug, but they didn’t fix it yet. I still find it hilarious.
I happen to know that the owner of the forum is planning to go abroad sometime in January. If he does he will surely see this weirdness himself firsthand when he posts to his forum. Maybe then, once he gets to see it himself, a fix will be prioritised :p
What would you expect as a fix? How do they know about the timezone you are currently on? Not an easy fix in my opinion as they have to be aware of your timezone (or ask for it) anytime you post
They are rendering the timestamp with JS.
JS has the ability to see system time zone offset.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
It’s not difficult to send the time zone in UTC, and convert it client side to local time zone.
And since they do “x minutes ago” kind of things they are certainly using a library. They are probably passing in “naive” timestamps without TZ info into a library that is perfectly able to handle it correctly for them, had they just read the docs for the library they used and taken a moment to make sure that they include TZ info.
And if everything was done server side it would likewise be similarly easy to get it right because then my own time zone does not matter.
In our applicant tracking product, we had a close date by which candidates must apply for the job.
Early on there was a bug due to timezone handling where the job would close an hour too early.
No big deal one would think - the job is open for weeks, maybe months, so who cares if it closes at 11pm on the last day instead of midnight?
It turns out that a lot of people live their lives by the "last minute" principle. They want to do things at the very last minute, and they get furious when the last minute comes too soon.
It's silly to demean the people who did that. You were the one that defined the existence of a "last minute we will accept applications," not those people. If you had wanted everyone to submit applications an hour before that time, you should have defined that to be the last minute (and then not demeaned people who submitted in that last minute). There's either a last minute you will accept applications, or there isn't!
Biggest thing we did with our hospitality related app (so check-in and check-out biggest time related fields): store time and date separately. Which I know isn’t possible in a lot of applications but whenever we only needed the date - we didn’t have to worry about timezones at all.
Also working with dates, times and timezones inspired me to get an ISO8601 vanity plate for my car.
Indeed, this cannot be a solution for most of the business out there, as date and time are tied together to ingest a certain event. But this could be an easy fix for some businesses
1) Outsource the nightmare to a 3rd party / open source library, whenever & wherever you can.
2) All events will trigger at UTC+0 time. Notify users on expiring events 24 hours before the event is triggered. Specify in contracts & EULAs that all events will occur on UTC+0 time.
3) "Soft expiry / leniency periods", wherein X amount of time is given after the event trigger in the "rare" cases where customers are late to respond to events. Do be warned though that customers will eventually treat this leniency period as if the expiry didn't happen, and some will inevitably complain when they can't do Y because they delayed well past the leniency period.
I built a logistics SaaS for ecommerce and the issue of timezones almost drove me insane. Finally, I discovered the 3-step solution to preserve my sanity.
1. Store all dates/times in UTC. This way, you know at least know exactly what point in time the data represents.
2. Display in the user timezone as needed
3. Delegate handling of time manipulation to an external library (like Lumen). This way you don't have to deal with all the intricacies of understanding international law and geopolitics required to turn timestamps into actual real life event
It's solved, in that I'm never going to do better than zoneinfo and I'm not going to go crazy trying.
https://en.wikipedia.org/wiki/Tz_database
https://www.iana.org/time-zones
I think of this as "difficulty snap back": Things can only get so difficult before everyone punts and uses some external library.
The article isn't about implementing your own time zone handling directly. It's about incorrect assumptions you can make when using such a timezone database or library.
For example, carefully consider when events need to happen at a fixed time relative to server time (utc) vs user local time. Consider how you handle changes to the timezone database. A timezone change can cause the scheduled time of an event (such as billing) to not exist in that time zone, or for a scheduled time to occur twice. A user changing their timezone setting can cause a similar effect.
Timezones are not hard to build. It's indeed the interpretation of your app and the display to your end users that are hard to manage. Think of your birthday: it's not hard for you to set a datetime at your current location to define when your birthday starts. But the birthday wished from your friends might changed based on the interpretation they have and where they are located
Seems like something people are always re-inventing solutions for when these problems have already been solved by so many others.
Timezones are easy. All you need is a complete understanding of Special Relativity and inertial frames of reference.
Bada bing, bada boom.
The relativity and frames of reference are trivial compared to the whims of international politics
God doesn't play dice, but random governing bodies sure do.
Now if AWS would learn that lesson as well... for example, AWS Backup Plans or CloudWatch Events always run in UTC, and there's nothing I hate more than doing timezone math - and on top of that, at least for CloudWatch Events (that trigger starting up/stopping a number of very expensive instances based on business hours) I have to manually update the schedule at each DST shift.
AWS, do. fucking. better. I know you have the resources for it.