Crowdstrike: What % of sys admins can remotely boot to safe mode, given BSOD?
What kinds of setups would make this possible?
For those who can't do it, what is their least bad option? Seemingly pretty low. I work in a large org and getting to the point of a quick streamlined solution hinged on just a few people. Without those few, it would have been a 100% manual effort. Most didn’t even seem to know a good way to identify the issue at scale. The solution didn’t bother with safe mode, it used a boot image that would search for the offending file on all the drives and deleted it, then rebooted back onto the normal boot drive. With specialization, so few people know all the various pieces to be able to quickly solution something. Good generalists are often overlooked, but they are the saviors in these moments. Oh, below 20%, I'd wager. The kinds of sysadmin skills people used to have that would allow that to happen (PXE, serial over LAN, etc) have atrophied quite a bit in the last ~decade. Maybe around 20% if you limit it to people whose titles actually are "System Administrator" (which implies an older, steady-as-she-goes IT dept) instead of e.g. also lumping in DevOps or SRE or cloud people. But probably above 1%, given that it's a serious enough tail risk that you might keep an old geezer around who remembers how to do it just in case something mission critical happens. I would have to disagree, PXE boot is default for most enterprise shops, they are not usb booting or burning dvd images w/ OOBE and such. This is also Windows world, where everything to do PXE booting is literally click and click. Devops/SRE and "cloud" are also different, I would say Devops/SRE's would have no experience w/ general windows deployment. Cloud can be 50/50 if they are on Azure, doing Windows servers and mass deployment/runbooks there. AS for the old geezer, those are the ones I'd be worried about. While in the XP days pxe boot was a bit new, and USB booting was finally getting implemented in bios's - they are the ones that'd probably suggest a windows recovery via DVD. And not to mention, the skills really for this are really low - the barrier is bitlocker, whether the key was backedup on the AD server and/or if the ad server was essentially bricked as well. There'd be a few, and if they go down, then disaster recovery would be the other half and hopefully they wouldn't restore backups - but thats another side of the coin here. tl;dr the clients are easy enough to fix, any proper org can reimage a computer probably in an hour or 2 per client - if needbe can re-AD join them and be almost up and ready, if non-encrypted (rare but sure) then a quick repair would probably work if org was not aware of how to boot and delete says file in system32. You need out of band remote management, basically a separate computer on the side that can do things to the main computer, or a read only OS with networking and management abilities that things fall back to. It’s definitely a thing in the datacenter world, and there are examples of cloud stuff. MDM solutions come to mind for mobile, and MacOS can do that too in a limited fashion. For any remote workers - send laptop to HQ and get it back a few days/weeks later. IT Support depts are going to be overwhelmed. My ex company just moved all IT to India. Good job. So what you are saying ... Karma struck ... could be weeks... I was thinking I could go around to companies doing their updates, for a fee of course. Waste a weekend, but make a bunch o money. But then it struck, how will I find the customers as their computers are down ... I suspect that deploying a new image via PXE will be the fastest resolution. Ideally one you just built without crowdstrike. If you have important data trapped in userspace, print out some bitlocker keys and get in the car lmao. I'm not a Windows user anymore but have seen references to bitlocker. For any others curious its disk encryption, so users with it cannot apply the fix because booting into safe mode to delete a file requires unlocking the disk encryption. I'd imagine safe mode doesn't have networking? That doesn't sound too safe so yeah this is an IT nightmare logistics issue for remote workers with encrypted drives. Microsoft added 'safe mode with networking' a long time ago. Wait, does this mean if all your machines had Windows and Crowdstike installed, and you had no backups off site, all your data would essentially be wiped out?