ProcMon for Linux (Preview)
github.comSomething that is really cool about Sysinternals tools are that they have a pretty usable GUI (granted, once you need to understand what does each UI icon do). Mimicking this experience in the TUI/terminal would certainly be a challenge. I would love to have a Procmon and ProcessExplorer equivalent with the GUI on Linux and macOS.
i always thought of the sysinternals tools as like the missing tools that just ship with unix systems.
without them, there was no real visibility into open file descriptors, network ports, command lines for running processes, thread trees for processes or any of the rest.
the gui was always just the windows way of presenting it. bringing a gui when the rich tools already exist on linux seems ... redundant?
(seeing this brings back nightmares from windows admins who refused to install sysinternals tools on production machines)
This was my reaction as well.
I haven't used Process Monitor/Process Explorer for a while, are they bringing anything to the table that's not covered by htop? With htop, I can even start tracing a process (using strace) or list the file opened by a process (using lsof) right inside htop. As might be obvious by now, I think htop has a fantastic TUI that also fits its use case very well.
Of course there are other tools that allow deep introspection into what's really going on aside from htop. Another favorite of mine would be perf top, which uses the sampling profiling mechanism to profile the entire system (including the kernel) on a symbol/instruction level. In other words, you don't just see that 30% of CPU time is spent in /usr/bin/cpu_hog, you also see that the time is spent in some_symbol and you can jump right in and see disassemble some_symbol (as long as cpu_hog is not stripped).
Maybe I'm wrong, but I don't think htop can do things like: list every process which opened a certain file for read, how much bytes have been read at which offset in each case and how the threads stack frames locked like at the time. Then revert that filter and follow some hint to some other events, e.g. what network operations did a certain process do in a certain period of time.
> With htop, I can even start tracing a process (using strace) or list the file opened by a process (using lsof) right inside htop.
Hmmm... didn't know it could do that.
Not sure about the Sysinternals, but I've found that a GUI can be quite useful for tools like these, especially when you want to do things like select multiple rows (generally fiddly with TUI interfaces) or draw more complex graphs. And, of course, a GUI usually brings all the disparate tools together into one package.
It's not quite inspired by the Sysinternals tools, but I'm working on something like this for macOS. An early beta should be released soon™ (read: when I stop procrastinating on polishing some of the last important bits I want to finish before then).
I'm not familiar with the Windows version of ProcMon, but judging by the Linux version, that looks like it's primarily replicating the function of `strace` in a CLI GUI.
Worth reading Joe Damato's excellent article on strace: https://blog.packagecloud.io/eng/2016/02/29/how-does-strace-...
It's not quite strace, ProcMon also does network traffic stuff too. I used it quite a bit when my job was trying to shove proprietary (usually academic, usually engineering-related) software installers into a shape that we could make silently install. ProcMon was one of our first resorts whenever an installer would inevitably mysteriously crash when it ran inside an SCCM-managed installation of windows. Which. Was. OFTEN.
Sorry, it's been over a decade now and I still have nightmares sometimes. But yeah procmon is cool.
I do other sysadmin now but the job of getting whatever random vendor's installer into an SCCM deployable state required tools to see just WTF they were doing.
I recall one .exe installer, unzipping to an MSI, which installed the program files but then ran .bat file to run three separate driver installers for the associated hardware. And I had to get it all working in AppV.
I don't use Procmon much now as my work has changed, but I still consider it a good friend.
I am glad linux has other ways to do it but as a linux noob, I'd try procmon first if I had to.
I was in a similar role about that time, I bet you had fun with msvc and manifests :)
Yes, ProcMon is the Windows equivalent of strace, with some filtering capability built-in. It also shows you the stack involved with a particular event, which can be useful for diagnosing the otherwise black box that is Windows. Eg [1]
[1]: https://github.com/msys2/MSYS2-packages/issues/138#issuecomm...
The "filtering capability" actually makes up a good half of its worth. Being able to very quickly drill down to an annotated call stack for a specific IRQ on a specific file is really handy.
> call stack
strace -k
> specific IRQ
IRQs are very low level and can't be traced by strace or procmon. both can easily log specific syscalls, e.g. strace -e. strace can't quite filter on ioctl, but apparently procmon can't trace ioctl at all: https://stackoverflow.com/questions/9947933/how-to-log-the-d..., so strace is ahead there.
> specific file
strace -P path
>> call stack
>strace -k
>> specific file
>strace -P path
That strace can do it too is fine and all, but the Windows ProcMon collects a trace that you can then filter, like Wireshark. So it enables a workflow where you have no idea what's going on, collect a trace, and discover what entry to focus on and what paths to filter for. (This is what I demonstrated in the msys issue link I posted.)
An equivalent workflow is possible with strace too of course; pipe `strace` to a file and then discover what paths you want to grep for. It does become a bit noisy when the stack of every syscall is included, and requires more complicated grepping because it spans multiple lines, so you might have to first pre-process it into something structured like JSON.
Original ProcMon used ETW, Event Tracing for Windows; the analogous technology (although very different in style) on Linux is eBPF so that’s what this tool uses.
I think you’re mistaken. ProcMon doesn’t use ETW on Windows and I don’t believe it ever did?
Sorry about that; I guess I misremembered?
This file says it does, though only for network events: https://documentation.help/Process-Monitor/documentation.pdf
Indeed I don't think so. ProcMon uses a kernel driver for the event tracing.
This seems to be using eBPF, so presumably your kernel will have to support that (only stating Ubuntu 18.04 is a bit unfortunate).
18 is already 3-4 years ago :)
The version itself is fine, but specifically listing Ubuntu as a system requirement is strange. If Ubuntu truly is a requirement, the name of the project should be "ProcMon for Ubuntu".
Presumably it isn't, and the README should list the actual requirements (kernel versions, etc.), with a note to the effect of "packages are available for Ubuntu LTS releases since 18.04".
It could also just mean it hasn’t been tested on anything else. But yes, the README should mention these things.
Sorta related: If y’all haven’t used netdata - wow - it blew my socks off last night. I have it installed on all my hosts now and it’s already helping me diagnose a redis bgsave issue that I didn’t realize existed. Really well put together software.
Hey,
I am odysseas, DevRel at Netdata. I would love to hear your story and what Netdata helped you diagnose in your systems.
Hit me up (odyslam) in our forums: https://community.netdata.cloud
Why do you think most people don't know about netdata and stick with the "status quo" of gathering data -> timeseries db -> manually creating dashboards & configure alerts?
What are the differences between this and htop, by example?
I know htop can strace processes with the "s" key.
This was my first thought, too.
Which useful feature does this tool have what htop does not have?
Let's say there is such a feature. Why did they not implement it in htop? Made a patch and offered it to upstream? This is why we have FOSS. To benefit from each other's work.
> Why did they not implement it in htop? Made a patch and offered it to upstream? This is why we have FOSS. To benefit from each other's work.
This is why so much open source software is crap and without any structure.
Htop is very different from proc on. Integrating proc on into Htop would make zero sense. Why would you suggest this would be a good idea without knowing what the software does?
Finally Vi become emacs-evil.
Because Ubuntu 18.04 is used for the development environment, it relies on pretty old dependencies.
Bug report: https://github.com/Sysinternals/ProcMon-for-Linux/issues/44
Yeah, if you have a modern distribution, look elsewhere. For a "preview" release it's interesting that they're only targeting older distros - maybe they're focused on enterprise use.
While I see a lot of comparisons to htop, I wonder how this compares to sysdig? I’ve found sysdig to be an invaluable swiss army knife of debugging performance issues; the only difficult part is finding the right documentation among the cloud offering of the same name.
I've been looking at a few eBPF tracing implementations around GitHub recently to try and get my head round it, and this codebase has to be one of the best documented (if not best written) examples out there. Kudos.
Even though I don't use procmon on windows basically never. Sysinternals on linux could be useful especially with same gui as on windows. It would be great to see autoruns and tcpview on linux.
Licensed under MIT.
This looks like an amazing tool, can't wait to begin using it.
I remember the first time I discovered Sysinternals tools for Windows. It was a game-changer for me. Especially Autoruns and Process Explorer
This will come in handy during DFIR, although we have different workflows for doing live analysis on machines for now.
I never thought I'd say this, but: Thank you, Microsoft, for making useful, user-friendly software.
Sysinternals is under Microsoft since 2006. Is this the ‘extend’ stage?
Why are libllvm6.0 llvm-6.0-dev and libclang-6.0-dev needed for this?
Seemingly because they download and build bcc as part of the build process. As to why they do that... probably because the three year old distro they're targeting has a three year old version of bcc.
Just out of curiosity, How is this tool different when compared to htop