Increasing Attacker Cost Using Immutable Infrastructure
diogomonica.comIt is not a good idea to restore attacker-owned applications to a "known good" state before you have done at least a cursory post mortem. Not only do are the security holes intact but since the attacker now knows they been found out, you can invite more serious damage.
The article tries to pitch read only Docker images some kind of solution, but running your applications read only (and what other permissions you grant your application) has nothing to do with Docker images. Using file system and process namespaces for application isolation is a good idea.
But lately it seems to be getting more popular to drop untrusted applications in containers (possibly even under outside control and full of who-knows-what) as if that somehow solves "security". The thing is, you still need to be able to reason about what permissions your respective application requires, there's no getting away from that.
The invention of containers (FreeBSD jails) was to automate restoring vulnerable web apps that couldn't be patched / updated (edit: as well as allowing shared hosting with different perl requirements to co-exist on the same server) Every time tampering was detected they automated restoration and watched the attacker squirm as their changes kept reverting and files were disappearing.
Don't dismiss this concept. It's a perfectly valid approach in some scenarios.
> It is not a good idea to restore attacker-owned applications to a "known good" state before you have done at least a cursory post mortem. Not only do are the security holes intact but since the attacker now knows they been found out, you can invite more serious damage.
I would also add that attackers are actually after the data. Exploiting application vulnerabilities is just a mean to that end so bringing back exploitable application from the previous image is a BAD idea.
Having said that previous image can be a good starting point to patching up the vulnerabilities and bringing the application online.
> I would also add that attackers are actually after the data.
Not strictly true - depending on the parties it's still a very desirable goal to be able to snoop on a site's continued operations, inject some code into visitors' pages, or just have a platform for further attacks.
I agree that rolling back an image to a "known good" state is not the appropriate solution; if you can do it efficiently, what's stopping you from doing it very regularly?
Attacking a system that disappears and restarts on a regular basis is a nightmare for current attackers; it's not something that they have the tooling to deal with yet.
I believe some of these ideas were discussed by Dino Dai Zovi in a talk[1] he did which combined a whole bunch of rather out of the box ideas on defense.
I'm not sure I understand the argument against reverting state. If you are cleaning up any sort of mess--including getting hacked--it's vastly easier to start with a good before-things-got-messed up state. This is one of the reasons people buy expensive storage arrays that support fast snapshot reversion.
Also, the author is arguing fix now but preserve all information for fault diagnosis later. For a lot of problems that seems like exactly the right choice.
Should probably just tiger-tree-hash your containers. If they deviate you give them a copy of your database, but it's a "virtual" shadow copy. No writes to it reflect on the real database.
If attackers get your database, you lose. Containers don't fix application vulnerability mitigation and leaking data.
I know the author is pitching the immutable infrastructure side, which definitely has its merits. But the ability to so easily diff the image and running state of a container also opens some intriguing opportunities for honeypot automation.
Are there any open source solutions that already take advantage of features like this? Or are those mostly kept secret for security and business reasons at this time?
Intriguing indeed. Something like when a write happens, instead of just plain blocking it redirect the write to a honeypot environment and send out an alert. Sure, that won't fool a good attacker for a very long time, but would be interesting to capture various drive-by and skriptkiddie attacks.
> Until we fix this RCE vulnerability, the attacker will
> still be able to execute code on our host [...]
I'm wondering how far we can take this. The ability to execute code on the host is there because that's what Linux does, but what if we removed this interface, and replaced it with an interface that just accepts one or more ELF binaries at compile-time? Then these would become the only Linux executables that this kernel can execute.
As far as I can see, we could do the same to system calls: if an executable can enumerate all the system calls it needs, we can compile a kernel that will accept only these system calls, which should be a small subset of all available Linux syscalls.
This is called a Unikernel: https://en.wikipedia.org/wiki/Unikernel
> Unikernels are specialised, single address space machine images constructed by using library operating systems. A developer selects, from a modular stack, the minimal set of libraries which correspond to the OS constructs required for their application to run. These libraries are then compiled with the application and configuration code to build sealed, fixed-purpose images (unikernels) which run directly on a hypervisor or hardware without an intervening OS such as Linux or Windows.
That is correct. I guess what I wanted to point out is that it would be cool if Linux could become this - a library that you can compile into your program, rather than a program in which you run your programs (an OS).
There is a library port of Linux, lkl https://github.com/lkl
Linux is purposefully and by design monolithic.
However, you could base your unikernal capabilities on the same interface (like POSIX).
Current unikernals: http://unikernel.org/projects/
The GPL would likely give any legal department pause for thought in that regard.
Or as i like to call them, DOS-in-a-can...
That's not fair given DOS was 100% unsafe in architecture and implementation language. The Ocaml or Rust ones are more like the Pascal OS's (eg Solo) with a certain amount of safety built-in. I agree on the minimal part but the overall metaphor doesn't fit.
DOS like Denial of Service? Or DOS like MS-DOS?
Either one doesn't make sense to me.
the latter.
DOS was a downright minimal layer of code between the software and the hardware.
Little to no memory protection etc.
If it walks like a duck and quacks like a duck...
> As far as I can see, we could do the same to system calls: if an executable can enumerate all the system calls it needs, we can compile a kernel that will accept only these system calls, which should be a small subset of all available Linux syscalls.
That is what pledge essentially does at runtime
This industry is one of the least aware of its own history, everyone is focussed on tomorrow. We should really include History of Computing as a discipline. Round and Round and Round we go.
---
You want to make your way in the CS field? Simple. Calculate rough time of amnesia (hell, 10 years is plenty, probably 10 months is plenty), go to the dusty archives, dig out something fun, and go for it. It’s worked for many people, and it can work for you.
— Ron MinnichI wonder how much of that has to do with the mainframe-minicomp era having been largely insular relative to the wider population.
Keep in mind that todays server hardware traces its lineage back to IBMs response to microcomputers like the AppleII and C64.
Docker is not (designed to be) a security technology. Yes, rolling back servers/VMs/containers to their previous state is a good capability to have (although most of us just use backups for that!), but assuming that an attacker cannot break out of a container is, at least, optimistic.
> assuming that an attacker cannot break out of a container is, at least, optimistic.
Agreed. However as long as you don't look at it as your primary line of defense, it increases the cost to an attacker. And that's currently the best we can ever do.
There are arguments - which are at least plausible - that the same effort is better spent on VM / Solaris Zones / FreeBSD jails / traditional chroot / SELinux / just using dedicated hardware for everything / ...; several of these have the advantage of not encouraging people to throw up a compartment (of some kind) and never update it again, which is certainly something that happens with Docker.
Of note is that an immutable/noexec filesystem doesn't prevent code being downloaded to an environment var/typed out and run - tools like https://github.com/SafeBreach-Labs/pwndsh just pipe source to an interpreter (in that case BASH, which generally isn't installed in smaller base images).
Reducing the attack surface is important, but if a running container is compromised it's imperative a post-mortem is performed immediately - and the issue remediated - to prevent re-exploitation.
potentially you do not need any interpreters available at all, which certainly increases attack difficulty.
This is definitely one of the main reason we decided to this road for VMs as well at Boxfuse (https://boxfuse.com): minimal images (5 MB for a full VM with a Go application) and immutable infrastructure all the way.
So I am confused, how does any of this increase attacker cost? Cant you do the same thing at the OS level already? Wouldn't making the dir read only do the same thing?
If your data gets stolen and your website defaced what's the use of immutability? I mean you can always run a diff tool against the current code on the server with the code you have on your repo right?
> how does any of this increase attacker cost?
Because it forces the attacker to write a specific payload for your service. Standard, reused "drop shell.php and register IP" will not work anymore. And realistically if the target of the attack was a WordPress installation, it will likely be a trivial, automated script.
> Cant you do the same thing at the OS level already?
Yes, you can. Even better, split execution privileges from file privileges, then make it read only, then put a grsec/apparmor/selinux profile on the service. It's not docker specific, but docker does make read only service a little bit easier.
> Wouldn't making the dir read only do the same thing?
Yeah, but who would do that old school thing. Docker security! :-(
it's pretty hard to diff filesystems that aren't designed for it. Either you need to lock the whole filesystem somehow - e.g. by taking it offline - or you have to deal with the fact that other processes are reading/writing as you scan the filesystem, which is rather difficult to reason about.
And it's not just about diffing your code with your repo - that only works if the attacker tried to attack your code. What about other running processes? New files on the system containing malicious code, outside of the paths you usually deploy code to? what about new, unexpected cron jobs?
Overall, it could become a pretty complex job. A filesystem with some intrinsic snapshotting makes this a lot easier.
COW by default file systems which provide a "snapshot at time x" can help with this. NTFS can do it with shadowcopy.
You can also use lvm snapshots, which work with any filesystem.
Real immutability would be good in more applications, as when you load a program into the memory of an embedded processor and then blow the programming fuse so it can never be changed. Hardware where parts of memory can be made read-only after boot and can't be written again without a full processor reboot would be useful.
I remember when mounting /tmp noexec and chmod g-w on htdocs was enough.