Everything you never wanted to know about file locking (2010)
apenwarr.caOne sure way to get a lock is to make a directory.
#!/bin/sh
if mkdir /your/lockdir
then trap "rmdir /your/lockdir" EXIT INT ABRT TERM
...code goes here...
else echo somebody else has the lock
fi
Is this guaranteed to be atomic on all filesystems?
For POSIX, yes.
https://rcrowley.org/2010/01/06/things-unix-can-do-atomicall...
Windows has a deep well of POSIX in the kernel (plus hard file locks), and it appears to hold there.
It’s even atomic on NFS. In fact, it’s probably the only reliable locking mechanism on NFS.
this is great thanks,
was just wondering, could something else remove the dir in between the if and then, before trap?
Just wondering about the atomicity.
The permissions on the parent and lock directory could restrict the access to a specific user and group, but yes, other processes could interfere with this locking if directed to do so.
One condition where this interference is helpful is a crash, where a @reboot entry in the crontab could:
You would also not want to place the lock directory in /tmp or otherwise where other users could manipulate (or see) it. In Red Hat, there is a /var/run/lock directory that might be appropriate.[ -d /your/lockdir ] && rmdir /your/lockdirMy biggest use case for directory locking in scripts is handling inotify events.
The problem with lock files and lock directories is that if the lock holder dies without cleaning up you now need to do something to clean up.
On Linux, this is why I always turn to using abstract sockets when I only need local locking. Only one process can bind and the kernel cleans up automatically on process exit.
You could do the same thing with TCP/UDP, but abstract sockets give you more flexibility in naming with 108 characters vs. being forced to use a 16-bit integer. Also it means you aren't using up a port that could otherwise be used for actual network communication.
Abstract sockets also make for a nice process existence monitoring mechanism since any processes connected to the bound socket are guaranteed to be immediately notified when the process dies.
Yes, but that is not a weakness in the locking.
Usually when I read these writeups, I walk away thinking "Wow, $foo was a more complicated problem than I thought".
With this one, it was "Wow, $foo was a simpler problem than I thought and Unix (and thus Linux and OSX) just totally screwed it up for no reason"
As TFA mentions, Unix/POSIX locking is insane.
Note that this page is slightly outdated wrt. flock(). From the manpage (online at https://man7.org/linux/man-pages/man2/flock.2.html):
>
Since Linux 2.0, flock() is implemented as a system call in its
own right rather than being emulated in the GNU C library as a
call to fcntl(2). With this implementation, there is no
interaction between the types of lock placed by flock() and
fcntl(2), and flock() does not detect deadlock. (Note, however,
that on some systems, such as the modern BSDs, flock() and
fcntl(2) locks do interact with one another.)
CIFS details
Up to Linux 5.4, flock() is not propagated over SMB. A file with
such locks will not appear locked for remote clients.
Since Linux 5.5, flock() locks are emulated with SMB byte-range
locks on the entire file. Similarly to NFS, this means that
fcntl(2) and flock() locks interact with one another. Another
important side-effect is that the locks are not advisory anymore:
any IO on a locked file will always fail with EACCES when done
from a separate file descriptor. This difference originates from
the design of locks in the SMB protocol, which provides mandatory
locking semantics.
Remote and mandatory locking semantics may vary with SMB protocol,
mount options and server type. See mount.cifs(8) for additional
information.
NFS details
Up to Linux 2.6.11, flock() does not lock files over NFS (i.e.,
the scope of locks was limited to the local system). Instead, one
could use fcntl(2) byte-range locking, which does work over NFS,
given a sufficiently recent version of Linux and a server which
supports locking.
Since Linux 2.6.12, NFS clients support flock() locks by emulating
them as fcntl(2) byte-range locks on the entire file. This means
that fcntl(2) and flock() locks do interact with one another over
NFS. It also means that in order to place an exclusive lock, the
file must be opened for writing.
Since Linux 2.6.37, the kernel supports a compatibility mode that
allows flock() locks (and also fcntl(2) byte region locks) to be
treated as local; see the discussion of the local_lock option in
nfs(5).It would be nice to have a unix for the new millenium--one that discards everything the greybeards deem a mistake. The window of opportunity is closing. We won't have them much longer.
Another good read is the SQLite locking module, https://www.sqlite.org/src/artifact/0240c5b547b4cf585c8cac35..., since these guys have to deal with the insanity of locking across different systems in real life.
You know things are bad when the least awful implementation of OS-level locking is the one from Microsoft.
POSIX locks are insane enough that when I reimplemented the SQLite file system API, I gave up on them: https://github.com/ncruces/go-sqlite3/tree/main/vfs#file-loc...
This is my go to article every time I think about using file locking to solve a problem.
Also, OFD locks are great if you target Linux only these days.
Hmm I just ran into an issue with uv where it deadlocks because of something to do with file locking on NFS. This looks informative!
So good in depth post. THANK YOU.