How setting the TZ environment variable avoids thousands of system calls

Those who quote George Santayana are condemned to repeat him.

With the layering of clusters, containers, micro services, bet you probably have 10x worse than that. There is always a cost to abstraction. On the surface it might make things simpler but if you were to peel it apart, you would reveal a hidden layer of complexity. Hopefully, it's done well right enough that there will never be a need to peel it apart.

Xe just did mplayer as well. It calls non-blocking select(), then non-blocking poll(), then nanosleep(), in a loop.

* http://www.tedunangst.com/flak/post/mplayer-ktracing

> System calls in Linux are really fast. So saving "thousands" of system calls when /etc/localtime is in cache doesn't actually save that much actual CPU time.

"fast" is a relative term, and is somewhat orthogonal to "efficient".

There's a reason why certain functions use a vDSO. If you're just going to use a syscall anyway, there's kind of no point.

On your base system, yes. Lots of things can hook random syscalls, or environments might have syscall monitoring.

One example is the folks over at slack record every syscall for security auditing. https://slack.engineering/syscall-auditing-at-scale-e6a3ca8a...

This might be true for your system and libc, where the system calls make use of things like vDSO for gettimeofday go fast, but in general this isn't guaranteed at all. Even on x64, for certain libc implementations, like musl, if I recall correctly, syscalls are made the old fashioned way by trapping 0x80, which would mean you would see a much bigger effect by reducing the number of syscalls.

System calls in x86 are fast. Other archs behave differently. And the syscall time is not the only thing that matters, but potentially yielding execution

I did the same experiment on a Raspberry Pi 2. The net saving was 5.803 seconds, so 5.803 microseconds per call.

Obviously if you care about performance then you wouldn't be running your program on a Raspberry Pi in the first place. But for everything else there's this free speed up.

System calls in Linux are not faster than not doing them.

Yeah, this is a perfect example of micro-optimization being unnecessary. Not only will you not see performance issues from this in the real world, it might cause problems down the road, because since it isn't set by default this way, some apps may not expect it and behave erroneously.

But it's neat information to have in the back of your head.

I did the same, but with 10M iterations:

    $ time ./tz     
    ./tz  2,24s user 6,28s system 98% cpu 8,612 total
    $ export TZ=:/etc/localtime
    $ time ./tz                
    ./tz  1,35s user 0,00s system 98% cpu 1,364 total

> TZ=:/etc/localhost

Hope this is just a typo in your comment, not the actual test ;)

Oh, and another page that recommends strace without warning about overheads. Dangerous.

It feels like any code that needs to know the timezone of the server is inherently wrong. If timezone ever comes up in any context, it's either the timezone of the client from whom the request originates - in which case it should come as part of the request - or else the timezone somehow associated with the business process (e.g. "warehouse open 8-5 Eastern time"), in which case it should be part of the configuration for that one service.

:<whatever> means "read it from the <whatever>" file. See the last part of the relevant glibc documentation: https://www.gnu.org/savannah-checkouts/gnu/libc/manual/html_...

However the reason it works without : is that the implementation is being lazy and just ignores the : delimiter and falls back to parsing out a filename either way:

https://sourceware.org/git/?p=glibc.git;a=blob;f=time/tzset....

Here is the answer:

https://www.gnu.org/software/libc/manual/html_node/TZ-Variab...

  The third format looks like this:

  :characters

  Each operating system interprets this format differently; in the GNU C
  Library, characters is the name of a file which describes the time zone.

See https://news.ycombinator.com/item?id=13704054. The colon forces the file to be loaded, without it the other formats are tried first.

For the second question: There doesn't seem to be an explicit reason for the difference of treatment. The code that does it has been there since 1996, and hasn't changed since. The only reason given is "Caching happens based on the contents of the environment variable TZ.".

https://sourceware.org/git/?p=glibc.git;a=commit;h=68dbb3a69...

I'd argue it should cache the same when both old_tz and tz are NULL (but start with an old_tz that is not NULL).

I was about to file an upstream bug, but found https://sourceware.org/bugzilla/show_bug.cgi?id=5184 and https://sourceware.org/bugzilla/show_bug.cgi?id=5186

The latter actually implies the opposite should be happening: files given in TZ should be stat()ed just as much as /etc/localtime.

This is due to the multiuser nature of Unix-like systems.

/etc/localtime is set by the administrator. It may change without notice to the user.

TZ is part of the user's environment and the user sets it. All applications run by the user should honor the user's wishes if the user's not falling back to system defaults.

If you're setting TZ for yourself, your libc can update things when you update the variable and restart any applications you're running under the old value. It can therefore save cycles. If you're falling back to the system default that's not under the same user's control, then it must be ready to deal with unexpected changes.

Hi, both are answered in the article:

First:

> What’s going on here is that the first call to localtime in glibc opens and reads the contents of /etc/localtime. All subsequent calls to localtime internally call stat, but they do this to ensure that the timezone file has not changed.

and second: read the section titled "Preventing extraneous system calls" for the answer to your second question.

Portability, compatibility. The system should not set environment variables if there is a reasonable default action. Env is intended to be set by the user.

In general, the timezone is set during OS setup, and the system is left in a state where it's up to the applications to figure out what to do. For example, you might configure Apache (yes, I am old, leave me alone) to use a particular timezone. But if Apache senses an env var it may choose to override the configured value with what's in the env var. Or SSH might be configured to pass along all env vars, including TZ, which in all honesty it probably won't even if you tried, but it could, and then the destination server's application has the wrong timezone.

Point is, it's safer not to mess with env vars unless you need to.

The default behavior is done so programs don't need to be restarted if the timezone is changed, which also has real world impact.

Probably because while there may be tens of thousands of additional syscalls, the total amount of added latency and resources consumed are more likely to be on a scale of micro/nano/milli seconds.

...which fixes all the code you wrote, but of course, you may have legacy binaries that you don't have access to source to change... hence a simple setting of an environment variable, hardcoded though it may be, fixes the situation for all.

Though PeterWillis makes a good point akin to yours, and your (plural) point does make sense.

(Edit: added mention of comment with additional background on why to avoid hardcoding the variable)

You should be able to get maintenance for something like this pretty low. Add a line with a nice comment to the config template for running your apps that adds an extra env var. Any machine that runs any app now has the line...

I would definitely like to see a before/after real-world metric on impact here though.

The embedded Linux system I'm working on right now takes about 17 us per stat() call due to this but time will always be kept internally as UTC, so taking advantage of this is worth considering for me.

Since most embedded systems can directly translate the amount of processing needed to achieve the product goal into a real dollar cost of the hardware, any savings, even a small one, is worth investigating. Since the hardware and system are generally well understood, implementing something like this is much more reasonable.

But I agree, on a general purpose OS doing general purpose thing, an optimization like what's proposed by the article may not be worth the other tradeoffs.

Its right there in middle of article (under "Preventing extraneous system calls" section):

    $ TZ=:/etc/localtime strace -ttT ./test

There's also a surprising difference in behavior between tm = localtime() and localtime_r(..., &tm).

The former is the "traditional" function which returns a pointer to a statically allocated, global "struct_tm". The latter is the thread-safe version receiving a pointer to a use-supplied "struct tm" as it's second argument.

    :   do {
    :           t = time(NULL);
    :           localtime_r(&t, &tm);
    :           printf("The time is now %02d:%02d:%02d.\n",
    :                  tm.tm_hour, tm.tm_min, tm.tm_sec);
    :           sleep(1);
    :   } while(--N);

    write(1, "The time is now 07:23:33.\n", 26The time is now 07:23:33. ) = 26
    nanosleep({tv_sec=1, tv_nsec=0}, 0x7ffd9e798470) = 0
    write(1, "The time is now 07:23:34.\n", 26The time is now 07:23:34. ) = 26
    nanosleep({tv_sec=1, tv_nsec=0}, 0x7ffd9e798470) = 0
    write(1, "The time is now 07:23:35.\n", 26The time is now 07:23:35. ) = 26
    nanosleep({tv_sec=1, tv_nsec=0}, 0x7ffd9e798470) = 0

    stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2335, ...}) = 0
    write(1, "The time is now 07:30:56.\n", 26The time is now 07:30:56.) = 26
    nanosleep({tv_sec=1, tv_nsec=0}, 0x7ffc868c3010) = 0

Note, this is using glibc 2.24 under Arch.

    $ /lib/libc.so.6
    GNU C Library (GNU libc) stable release version 2.24, by Roland McGrath et al.
    (...)
    Compiled by GNU CC version 6.1.1 20160802.

...while the call count was high, the time taken by the calls in total was not significant(<1% if I remember correctly).

Yes, on ordinary filesystems if you run stat() over and over again on the same file then it's just copying from the in-memory inode into your struct stat, there's no IO.

It's hard to do that without any system calls, though.

The way you'd do this is to open an inotify (or platform equivalent) file descriptor on the first call to localtime(), and on future calls, only bother statting /etc/localtime again if that file descriptor reports something has changed. But checking if an FD has data requires a system call; you'd do a non-blocking read on the FD (or a non-blocking select or poll, or an ioctl(FIONREAD)). It's possible that system call is faster than stat, but it's still a system call.

You could do it with a thread that does a blocking read, but that's a mistake under the current UNIX architecture: lots of stuff (signal delivery and masks, for instance) gets weird as soon as you have threads, so unconditionally sticking a thread into every process using libc is a bad plan.

You could do it with fcntl(F_SETSIG), which would send you a signal (SIGIO by default) when the inotify file descriptor is readable, but you couldn't actually use SIGIO, since the user's process might have a handler. You'd need to steal one of the real-time signals and set SIGRTMIN = old SIGRTMIN + 1. (See https://github.com/geofft/enviable for a totally-non-production-suitable example of this approach.) This would probably mostly work, except changing SIGRTMIN is technically an ABI break (since programs communicate with each other with numerical signal values). You could maybe use one of the real-time signals that pthread already steals. Also, using signals is sort of a questionable idea in general; user programs now risk getting EINTR when /etc/localtime changes, which they're likely not to be prepared for. A single-threaded glibc program that sets no signal handlers never gets any EINTRs, which is nice.

In an ideal world, every program would have a standard event loop / mechanism for waiting on FDs, and libc could just register the file descriptor with that event loop. Then you'll get notified of /etc/localtime changing after the next run of the event loop, which is good enough, and it would take zero additional system calls. But unfortunately, UNIX wasn't designed that way, so something at libc's level can't assume the existence of any message loop. A higher-level library like GLib or Qt or libuv could probably do this, though.

There's no usable mechanisms glibc can use for watching /etc/localtime for changes that does not mess up the program if it also decides to use any file watching features.

And how does it watch it? With the stat() syscall.

It's not in any way hidden, just run a whois query:

    $ whois packagecloud.io|grep Owner
    Owner Name    : JOSEPH DAMATO
    Owner OrgName : COMPUTOLOGY, LLC
    Owner Addr    : 359 FILLMORE ST!12
    Owner Addr    : SAN FRANCISCO
    Owner Addr    : CA
    Owner Addr    : US

Maybe it's just that I use packagecloud and follow people in their circle on Twitter, but Joe Damato is the CEO and founder: https://twitter.com/joedamato

I'm under the impression that he may also write a lot these himself? Not entirely certain, though.

All the links are in the footer of the page.

If TZ is set properly then you don't need to change it twice a year.

(man tzset for more info and examples of different values TZ can be set to. Mine is set to "Europe/London" which handles the DST switches automatically.)

Are you talking about DST? You don't change your timezone at DST switch, the timezone data knows how to calculate the correct time based on your location (roughly) and the time of year. Mountain standard time and mountain daylight time are both part of mountain time, for example.

Do all your work in UTC. Seriously, you'll be glad later.

Convert to local time only on the edges, and only for end users.

Starting or stopping Daylight Saving Time is not a timezone change. A timezone is — roughly — a set of timekeeping rules that some set of people use. DST is just part of those timekeeping rules. That is, your timezone is not the same thing as your UTC offset.

For example, the timezone America/New_York contains information not only about the UTC offset, but the offset during and not during DST, and when DST starts and ends. (And historical starts and ends to, so that UTC → local conversions (and vice versa) use the rules that were present at that time, not the rules that are present now, which may be different.)

E.g., my home desktop runs in the timezone America/Los_Angeles all year around. Most of my servers run in the timezone UTC all year. Both always have the appropriately correct time.

TZs should be defined in the form "Europe/London", "Asia/Beirut", "Pacific/Auckland", etc.

Although the hour offset can change at extremely short notice (e.g. discontinuing Daylight Savings during Ramadan [1]), the timezone declaration (e.g. "Africa/Casablanca"), shouldn't need to change, just the underlying timezone database.

[1] https://en.wikipedia.org/wiki/Daylight_saving_time_in_Morocc...

For Debian and derivatives:

    echo 'TZ=:/etc/localtime' >> /etc/environment

Experimentally, no. The example program calls localtime(3) 10 times but only accesses the file once, per truss:

    write(1,"Greetings!n",11)			 = 11 (0xb)
    access("/etc/localtime",R_OK)			 = 0 (0x0)
    open("/etc/localtime",O_RDONLY,037777777600)	 = 3 (0x3)
    fstat(3,{ mode=-r--r--r-- ,inode=11316113,size=2819,blksize=32768 }) = 0 (0x0)
    read(3,"TZif20000000000000"...,41448) = 2819 (0xb03)
    close(3)					 = 0 (0x0)
    issetugid()					 = 0 (0x0)
    open("/usr/share/zoneinfo/posixrules",O_RDONLY,00) = 3 (0x3)
    fstat(3,{ mode=-r--r--r-- ,inode=327579,size=3519,blksize=32768 }) = 0 (0x0)
    read(3,"TZif20000000000000"...,41448) = 3519 (0xdbf)
    close(3)					 = 0 (0x0)
    write(1,"Godspeed, dear friend!n",23)		 = 23 (0x17)

It is unlikely, they usually don't use GNU C library.

Yes, at least the Docker images that use glibc as their libc. (eg, most Debian/Ubuntu images)

It looks like musl, which is used on Alpine Linux images for example, will only read it once, and then cache it:

https://github.com/esmil/musl/blob/master/src/time/__tz.c#L1...

It has a mutex/lock around the use of the TZ info, but avoids re-stat'ing the localtime file.

If you havn't set the TZ variable in the image, then yes.

I would be surprised if this doesn't. There's nothing specific about docker when running this C code from any other processes running on a Linux host.