When you deleted /lib on Linux while still connected via SSH (2022)

tinyhack.com

191 points by todsacerdoti a month ago


gleenn - a month ago

One time I was flipping back and forth between directories, compiling some code, then checking it, then rm -rf'ing it. I accidentally hit up an cd ..'d one too many times. Suddenly the rm command hung and I was confused because it should be nearly instant as it was a few files. I stared in horror as I was accidentally deleting everything in mybhome directory. Luckily back then I had a .pr0n directory with a significant amount of content. A few things were lost but that .pr0n folder was luckily early enough in the list and big enough to slow down the deletion of my photos and documents. That's why I always recommend having a big "buffer" of video content for such situations, ya know, for data integrity ;)

eitland - a month ago

Something similar happened a place I worked somewhere between 10 and 20 years ago.

Service technician did not see the . in the command

  rm -rf ./bin
so he proceeded to run

  rm -rf /bin 
When that didn't work he did what everyone who knows a little bit Linux does and added sudo in front.

I was on a terminal from the other side of the globe when the server suddenly started acting weird.

We were able to use scp or rsync (one of then was in sbin or something) to get back the bits from an identical server, which saved me from three days of tedious work :-)

In hindsight of course we should have written the docs in a way that would prevent this exact situation but in the beginning it was just the output of the history command after I had done it, dumped into a document with some explanations.

inejge - a month ago

The heroic version of the story is now almost 40 years old[1]. (One HN mention with the link to a HTMLized version is here[2].) In both cases, the upshot is that as long as you have a running shell with root privileges, at least one existing executable file on the filesystem, and the means to overwrite that file with arbitrary binary content, you can write a small program which can recreate a skeleton system structure and dig yourself out of the hole.

The reason why this keeps happening is that in regular UNIX root is omnipotent and the filesystem is ultimately unprotected. Immutable systems and restricted execution environments may make this a thing of the past.

[1] https://www.wolczko.com/rm.txt [2] https://news.ycombinator.com/item?id=7892471

ryao - a month ago

The technical term is unlinked. The files in use by running processes are still anonymous files in the filesystem. They will not be garbage collected until the last program that mmap’ed them is gone. If you could find the inode number from /proc (possibly from /proc/$PID/maps), you should be able to use a filesystem specific tool to retrieve them, such as debugfs or zdb.

lloeki - a month ago

A long long time ago the team I was part of managed old unix systems.

A coworker telnet'd (or rsh, can't recall) into such a machine to do some maintenance and after a while fat fingered:

    umount /
Would you believe it, back then being root meant this was absolutely unprotected and the minicomputer OS (some ancient AIX) dutifully complied.

The chaos that ensued is but a blur.

vessenes - a month ago

I “rm -rf /“ ed my Linux box in 1996 and realized what I had done about 30 seconds in. It was through like a-d in /bin.

‘Recovery’ wasn’t a strong concept in Linux os installers then, and I wasn’t sure my home directory would survive a reinstall, (not on a separate partition) and I didn’t think I’d be able to reboot successfully in any event.

My brother ran the same distribution but at a school across the country; I was able to recover by carefully pulling what I needed down with ftp to his dorm computers IP. I’m not sure scp even existed then on Linux; I guess maybe we could have used netcat in a pinch. Well, this was already a pinch. In a real pickle.

This was one of the first moments where high bandwidth connections to endpoints really saved me / impacted me, and I haven’t forgotten. For many years after university this kind of direct high bandwidth connection got much harder to achieve, first because we were back in a low bandwidth residential world, then because ipv4 was mostly denied to consumers, then because of nat.

Today this would again be achievable, but with vastly more complexity. For a home computer rescue you’d want Tailscale in both sides. And it’s extremely unlikely you’d be using the same distribution as your sib, much less have the same libc linking. And god help you if you had to restore your systemd directory by hand.

nasretdinov - a month ago

I think I now know why on macOS "Applications" (with capital A) comes first in the list of root-level directories. I've been showing to my colleague that on modern systems "sudo rm -rf /" does nothing, however that turned out to be GNU-specific thing, and macOS of course uses BSD tools. So I've started noticing that something is off very quickly and stopped the command. On HFS+ (probably on APFS too) the file list is always sorted, so it only deleted some of my Adobe apps and stuff like Addressbook, so I didn't even notice anything at first :)

nullorempty - a month ago

At the start of my career I removed the `x` attribute from all files :)

zavec - a month ago

I actually started a short blog series about a similar problem where a friend had blown away /bin and a bunch of other stuff, but/lib was still there. Unfortunately it didn't end up getting anywhere because even though I was able to drop executables on the machine with echo and make them executable with a .so from lib I wasn't able to get back to root permissions as sudo and everything had been blown away and I didn't think I'd have great luck trying to find a zero-day in the kernel. It was still a lot of fun though.

Dwedit - a month ago

If you deleted /lib, you'd probably be better off reinstalling packages while booting off of USB or something. You're gonna have downtime because programs won't work correctly.

nurple - a month ago

My workstation seems fine:

  $ ls -R /{lib,usr,bin,sbin}
  ls: cannot access '/sbin': No such file or directory
  /bin:
  sh

  /lib:
  ld-linux.so.2

  /usr:
  bin

  /usr/bin:
  env
Oh right...

  $ ls -l /usr/bin/env
  lrwxrwxrwx 1 root root 65 Mar 21 23:39 /usr/bin/env -> 
  /nix/store/9m68vvhnsq5cpkskphgw84ikl9m6wjwp-coreutils-9.5/bin/env

  $ ldd /usr/bin/env 
        linux-vdso.so.1 (0x00007ffff7fc4000)
        libacl.so.1 => /nix/store/dyizbk50iglbibrbwbgw2mhgskwb6ham-acl-2.3.2/lib/libacl.so.1 (0x00007ffff7fb3000)
        libattr.so.1 => /nix/store/vlgwyb076hkz7yv96sjnj9msb1jn1ggz-attr-2.5.2/lib/libattr.so.1 (0x00007ffff7fab000)
        libgmp.so.10 => /nix/store/dsxb6qvi21bzy21c98kb71wfbdj4lmz7-gmp-with-cxx-6.3.0/lib/libgmp.so.10 (0x00007ffff7f06000)
        libc.so.6 => /nix/store/maxa3xhmxggrc5v2vc0c3pjb79hjlkp9-glibc-2.40-66/lib/libc.so.6 (0x00007ffff7d0e000)
        /nix/store/maxa3xhmxggrc5v2vc0c3pjb79hjlkp9-glibc-2.40-66/lib/ld-linux-x86-64.so.2 => 
        /nix/store/maxa3xhmxggrc5v2vc0c3pjb79hjlkp9-glibc-2.40-66/lib64/ld-linux-x86-64.so.2 (0x00007ffff7fc6000)
ofalkaed - a month ago

Many years ago I wrote a backup script where I did "rm -rf /etc/" instead "cp -r /etc/ /mnt/whatever," not sure how I managed that. Took ages to figure out what was causing /etc/ to disappear since /etc/ going missing often went unnoticed for awhile and I was running Arch back in the days when running pacman -syu always caused exciting things to happen. I even did a complete backup and reinstall trying to figure that one out, and I was extra cautious about making a backup of that backup script which I had spent so much time on and was fairly proud of since it was my first non-trivial bash script.

I also once did "rm -rf /", was deleting a dir which started with a "[" and accidentally hit "enter" instead of "\." That one taught me the dangers of absolute paths.

Edit: That last one is not quite right, would not have been an absolute path issue, that dir must have ended up in root somehow, can't quite remember the details, been too long.

throwanem - a month ago

"When you discover yourself to be in a hole, the best first thing to do is stop digging."

Some of the best early professional advice I ever received was, in moments like these, to keep my hands off the keyboard for at least a timed minute.

mastax - a month ago

    chattr +i /lib/ld-linux.so.2
Sounds tempting.
scripturial - a month ago

‘rm —rf’ should have default behavior to prompt “are you sure” when deleting top level folders. People are literally talking about secret video files saving their files from being accidentally destroyed.

How in 2025 do we not have these types of innovations? Are people afraid of breaking old scripts?

(macOS saves us from a lot of this stupidity, Linux should have something similar. I would love a way to mark folders as “permanent” or as “restore on reboot”)

conceptme - a month ago

A long time ago I started working at this company and I fat fingered 'rm -rf /' instead with a dot.

When realizing what I had done and it was taking too long I powered off the machine. When I told the sysops he looked horrified and asked me how far it got.

The fun thing was all websites and data where mounted from the server to each workstation to make it easy to update source code.

chatmasta - a month ago

I’ve seen this same story in exploit writeups - it’s funny to see “living off the land” techniques used non-maliciously.

smw - a month ago

I had someone do this on an important production Solaris machine once, many years ago. Luckily they just moved /lib instead of deleting it -- and on Solaris (some of?) the binaries in /sbin were statically linked, including ln. Hard linking /lib back to the correct path was enough to recover.

asah - a month ago

This happened to me once. I caught my breath, found an identical server and copied the files over using uuencode and cat, until I had enough working to create a second terminal window and could use better tools to verify everything.

5 minutes that felt a lot longer.

udev4096 - a month ago

Reminds me of this: https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issue...

jmclnx - a month ago

Interesting, at one time ages ago I thought Linux came with a static linked bash in /bin or maybe /sbin for this type of problem.

Just checked Slackware and no more, I wonder if that is a casualty of the /bin /usr/bin merge ?

Fizzadar - a month ago

I once nuked the entire OS partition on an openvz host. Vz data was still good so we ended up copying the root fs from another similar box, manually updated the network config and it ran for another 4 years until retired.

userbinator - a month ago

This could be considered a variant of the "bootstrapping" problem.

nunez - a month ago

Super interesting, though it feels like a response to an interview question!

o11c - a month ago

> Please note that busybox can’t function with a name that is not a busybox applet name.

This is somewhat wrong. From ksh/bash/zsh, you can run:

  (exec -a someappname /arbitraryexecutablepath args...)
This won't work on most ash derivatives (including /bin/sh on Debian, FreeBSD, or NetBSD), but does work on busybox ash.

The parentheses prevent the `exec` from actually replacing your current shell, which might be less important for emergency rescues, but which otherwise is often what you want with `exec -a`.

- a month ago
[deleted]