Htop explained
peteris.rocksI switched to macOS from Linux two weeks ago and I am starting to regret that decision. Most — if not all — the commands that I thought I had mastered after years working on Linux environments stopped working because their BSD counterparts have slightly differences that completely break everything I try to do. It is even more frustrating when I realize that I have to deactivate security features built-in the new Apple computers in order to make things like "dtrace" (the strace alternative) work.
You should give Macports a try if you haven't already! I run it on my macbook, and it's made a world of difference. 99% of everything I use in Linux can be found in Macports, and it "just works" for the most part.
You can't get kernel-specific things like strace, but they do have just about everything else. The big sell for me was being able to install the GNU versions of everything.
Macports keeps everything in /opt, so you also never have to worry about it clobbering anything that MacOS expects to work in a certain way.
edit: some background on Macports vs Homebrew - it's basically a vi/emacs kind of a thing. Homebrew utilizes the MacOS system stuff a fair amount, and installs everything to /usr/local. Macports tries to keep everything fully separate, and installs into /opt.
My personal experiences with Homebrew weren't great, but others will probably have the opposite opinion.
One of the things that I really dislike about homebrew is its refusal to run via sudo plus the requirement that you own /usr/local.
This means:
1) It is a security hole; any application can now modify system-wide applications at any time.
2) You can't have multiple users able to use homebrew.
I was expecting homebrew to be something similar to apt/yum/etc, but this makes it pretty fundamentally different.
How does Macports handle this? Does $USER still need write access to /opt?
< plus the requirement that you own /usr/local
no, this is not required anymore... https://github.com/Homebrew/brew/commit/a2f19f0a675f7cb384b4...
> You can't have multiple users able to use homebrew
what i did is i run Homebrew as a user i called Admin and when logged in as another user i simply do a 'su - admin' to run the brew commands..
There's no requirement that Homebrew be in /usr/local. Some individual (poorly-written) packages may assume that, but I've been running Homebrew in $HOME/.brew for multiple years on multiple machines with no issues at all.
That's actually fucking brilliant, and aligns much better with what homebrew is. I'm setting up my new macbook from work tomorrow and this is absolutely the way I'm going to do it, and how I'll suggest any of my colleagues do as well.
It's also very difficult to roll back to a previous version of a package unless the maintainers specifically opt in to tagging them. Homebrew also updates itself when you install something now, so say if you install python, it might update readline, so your postgres client will break.
It's not the worst approximation of ports, but it doesn't share the quality and care that goes into package maintenance and dependency management.
Re 2) I'm running a multi user homebrew setup for years. I have my users (personal and various work accounts) live under the "Local" Group and do
``` sudo chmod -R 770 /usr/local/ sudo chmod -R +a "group:Local allow delete,readattr,writeattr,readextattr,writeextattr,list,search,add_file,add_subdirectory,delete_child,file_inherit,directory_inherit" /usr/local/ ```
I remember sometimes running into permissions issues but nothing re-running that command didn't fix.
It's a bit of a smell that Homebrew doesn't do this itself.
Really? If you're going to do that, it seems like you might as well just merge all those users into a single account anyway, considering how trivial it would be for any one of them to execute arbitrary code as any other.
Why is 'sudo' required? Just curious.
I moved away from homebrew, to compiling my own software on macOS which I just put in ~/build. I don't use sudo make install since I own the ~/build folder.
What am I doing wrong in this case?
This is no longer the case, as of Homebrew 1.0.
See this change: https://github.com/Homebrew/brew/pull/896
This is only about /usr/local itself, not the directories under it, the one with executables - that's still owned by you, not root. Some difference in ease of initial installation, zero difference in terms of (in)security.
Another recommendation for macports. You'll find htop and all your other linux friends in there.
eg for htop: `sudo port install htop`
(also in brew; `brew install htop`, vim, tmux, wget, lynx, etc.)
"You should give Macports a try if you haven't already! I run it on my macbook, and it's made a world of difference. 99% of everything I use in Linux can be found in Macports, and it "just works" for the most part."
First thing I installed in the mac store. Just like on linux, it just works and I get all my favourite tools.
For general dev use in macOS I find that installing the GNU userland (through MacPorts) keeps me sane, "sudo port install coreutils moreutils htop gnutar" and then put /opt/local/libexec/gnubin/:/opt/local/bin at the start of your PATH.
Doesn't help much with dtrace though!
yep! Macports is awesome.
A little nitpick, dtrace is not an alternative to strace. DTrace is far more powerful.
Alternative to strace would be dtruss, which is implemented through dtrace.
Until I can, as an unprivileged user, watch my own process's syscalls with dtruss, it's not an alternative to strace. Requiring root is a major hindrance.
`brew install coreurils && brew install htop`
You probably also want to chmod +s htop, or it won't actually show you CPU/memory stats.
and if you want to use `sed` not `gsed`, do `brew install --default-names coreutils` to supercede the OS provided utilities.
Or you can add it to your PATH:
"gnubin" contains the all utilities with the original names.PATH=/usr/local/opt/coreutils/libexec/gnubin:$PATH
You can install the GNU coreutils: http://apple.stackexchange.com/questions/69223/how-to-replac...
Homebrew has packages prefixed with `g`, standing for GNU, that you can install to avoid some of the BSDisms on macOS.
Example: `gsed` actually parses whitespace the way I think it will; the builtin `sed` does not.
> to avoid some of the BSDisms on macOS
to be fair, you should probably call the GNU behaviour "GNUisms". The BSD behaviour is the historical POSIX behaviour. The GNU behaviour is, in a twisted sense, the result of embrace, extend and extinguish by the GNU people :-)
With a couple of small exceptions[1], the GNU behavior is also the historical POSIX behavior; the differences that will be jarring are where both BSD and GNU implement things that aren't in POSIX. Yes, the GNU behavior is GNUisms, but the BSD behavior is also BSDisms.
[1]: The most notable thing is that GNU generally allows flags to come before or after non-flag arguments, where POSIX requires them to come before.
Isn't dtrace a completely different (and broader) tool than strace ?
Yes, dtruss would be a closer equivalent but it is using dtrace under the covers.
What made you switch to MacOS?
Not the OP, but the reason I switched to the Mac 12 years ago (and never looked back) was to bring together my nerdy, creative, and "end user" sides under a single OS.
I'd been using mostly Linux on the desktop since 1996, I manage servers for a living and couldn't live without a terminal. That's the nerdy side.
On the other hand, I was serious enough with photography and electronic music production to need best of breed tools, which are still largely Windows and Mac-only to this day.
There's also a time for everything, and there are days when regardless of your computing skills, you need your creative person/end user hat on and apps that "just work" to keep your right brain fully focused on the end result (e.g. "No thanks, I'm not going to get distracted compiling some Gimp plugin to get red-eye removal", which may or may not have changed since I last tried)
MacOS, in spite of its recent slight drop in quality, is still second to none in satisfying all these requirements at once.
You normally do not have to disable security features to run programming/debugging tools, contrary to most of the advice you'll see in forums (and sometimes even Stack Exchange). I haven't used dtrace but non-bundled debuggers were a bit of a pain. You can self-sign these using the procedure described by a quick Google.
Normally, no, but if you want to dtruss uptime, or any other bog standard system utility, then, yes you do.
Very much this. Sometimes the differences are very subtle which makes them more annoying IMO (can you omit the target directory when using `find` if you want to use `pwd`?). For extra points, since you probably aren't deploying on a non-Linux *nix, the PITA will continue in perpetuity.
I posted a link to a instructions on getting the GNU equivalent of commonly used BSD programs on OSX yesterday: https://news.ycombinator.com/item?id=13084969
> Most — if not all — the commands that I thought I had mastered after years working on Linux environments stopped working because their BSD counterparts have slightly differences that completely break everything I try to do.
This is the true 'unix philosophy.'
"[explanation of acpid] But I'm on a virtual server that I don't intend to suspend/resume. I am going to remove it for fun and see what happens. ... I was able to successfully reboot the droplet but after halt Digital Ocean thought it was still on so I had to Power Off using the web interface. So I should probably keep this."
Science! :D
I love the approach of systematically going through and prodding everything to see what things do and whether they're actually required. I remember doing this on my Windows 95 machine, stripping down running tasks using Task Manager. Of course, in Windows 95, if you killed the wrong task the system would just reboot... :P
I used to do a lot of that when I was first learning Linux (back in the day when each Linux distro came with a full "HOWTO" collection.) I have become much lazier since, usually going for defaults and not touching anything unless I need the behavior to change.
I don't think acpid would help much for the halt command. However, it would enable to halt the machine from the web console.
htop is one of the most missed tools when I am on a non-Linux OS.
On servers I find atop to actually be a better tool for finding issues that happen incrementally and not an obvious issue. I think that atop and htop both are very complimentary.
"Atop is an ASCII full-screen performance monitor for Linux that is capable of reporting the activity of all processes (even if processes have finished during the interval), daily logging of system and process activity for long-term analysis, highlighting overloaded system resources by using colors, etc. At regular intervals, it shows system-level activity related to the CPU, memory, swap, disks (including LVM) and network layers, and for every process (and thread) it shows e.g. the CPU utilization, memory growth, disk utilization, priority, username, state, and exit code. In combination with the optional kernel module netatop, it even shows network activity per process/thread."
unless by non-Linux you meant non-unix, htop recently got updated to better support BSDs and OSX/MacOS.
That's excellent news.
https://archive.fosdem.org/2016/schedule/event/htop/
Found a presentation from Hisham about the effort.
It can also me installed and run on the new "Ubuntu bash for Windows 10" from Microsoft, which uses the new "Windows subsystem for Linux"
How much of /proc there is filled with dummy data though?
> The second number is how much of that time the machine has spent idle, in seconds
Actually, the second number is the sum of the number of seconds that all cores on the machine have spent idle, so that number may be higher than the first one. Source: https://access.redhat.com/documentation/en-US/Red_Hat_Enterp...
Thanks for catching that. You are right. Here is the source code: https://github.com/torvalds/linux/blob/master/fs/proc/uptime...
My source (first result in Google) was this link https://www.centos.org/docs/5/html/5.1/Deployment_Guide/s2-p... which looks like an older version of the documentation that does not have that last sentence.
Slightly related utility I discovered to my great delight today: ncdu - "htop for disk space usage"
ncdu has an option to store the scanned fs and display it again without walking the FS; useful if you're dealing with deep mess and slow, remote or else data.
oh yes, i mostly don't bother with 'du -sch' anymore since ive come to know ncdu
Have you tried
egrep filter for files at least 10m ends up being a nice speed up, otherwise you end up spending most of the time sorting all the tiny files to the top while up writing hundreds of megabytes of /tmp/sortXXXX files in the process.du -macx ~ | egrep '^[0-9]{2}' | sort -n
I LOVE this style of documentation/teaching.
What is it? How does it know that? How did I figure that out?
The second and third question are far more useful but usually not included in documentation or commentary.
Thank you! Great writeup!
I applaud the way you approached it - figuring out every single thing about a tool.
Also, I didn't know about 'od' command. Thanks again!
please do explain in 2 words the od command. I was looking on the page and there is only one od command and it is not explained. thank you
> please do explain in 2 words the od command
man od
You asked for two words ;).
I really didn't know about this command ... is it that bad to give me -1? ... I will never ask again here because of this and in the same time thank you you to the guys that actually lost some time to explain.
I'd like to strongly encourage you in the opposite direction, to always ask questions. Always do this, regardless of how others react.
Many younger people have security issues explaining their knowledge with others, and sometimes exhibit this as criticism and acidity. It's possible that's what happened here.
I'm reminded of http://www.huffingtonpost.com/catherine-clennan/being-a-dumb... - it highlights the importance of always asking questions regardless of the context.
It's also possible that someone disliked your use of the phrase "two words", maybe because they interpreted it literally. Some things resist explanations that are both short and accurate.
Regarding od, it's a flexible format conversion tool that lets you convert arbitrary data to hexadecimal, octal and C escaped printable text. I use it a lot!
The nicest article in a wile now .. :) thank you for it and for the extra explanation on od :)
No problem :)
(PS. gia-web looks interesting)
:) no it doesn't. I've send you an email
Similar to "cat". But it shows the contents of a file in octal. try...
echo hello | od
echo hello | od -c
Useful when you want to figure out the real contents of a file.
"octal dump"
(can also dump characters and hexadecimal)
You swapped the child and parent in the printf statements here https://peteris.rocks/blog/htop/#z-defunct-zombie-process-te....
printf("The parent process is exiting now\n"); printf("The child process is sleeping now\n");Overall, I really enjoyed the article.
Thanks for catching that.
"You'll probably want to keep cron. But if you don't, then you should stop and disable the service. Because otherwise when trying to remove it with apt remove cron it will try to install postfix!"
...wat?
Is that actually a thing?
Dependencies in debian systems which rely on "virtual" functionality can often have surprising results. At a guess removing cron removes things which depend on it - perhaps one of those is Exim. Something else "requires" an MTA, so it installs postfix instead to fill the gap.
This would be apt. Apt requires an MTA because maintainer messages / change-lists are sent as mail to the root user.
Close. It is cron that requires a package which provides `mail-transport-agent`, because output from cronjobs is traditionally emailed to users.
You can see this via:
Note that the `apt` package itself does not require a MTA.apt-cache show cron|grep -i mail
Wow so much unix and shell goodness in this post.
Useful bits/tricks to me:
# see what files a given program opens
strace PROGRAM_HERE 2>&1 | grep open
# get last pid
echo $!
# all sorts of useful commands and information on a given PID
ls -alh /proc/PID_HERE
# get a tree listing of all processes
pstree -a
# echo into a file using sudo
echo "some text here" | sudo tee -a FILE_PATH_HEREIs there a resource out there on how to write these beautiful terminal HUDs?
I'd really like to step up my game for GUI monitoring apps without having to choose between either terminal log spam, PyQT, or a web server. Ideally I'd have something like htop/wavemon/power top that I could stream compactly via ssh.
Use https://github.com/yaronn/blessed-contrib or something similar
I think you're just looking for a book on (n)curses.
For another fun ncurses tool (on a similar wavelength) I have become fond of glances[1] - it breaks out docker containers nicely, along with the good top-like overview of all procs.
For general use I'd like to add multitail (logging done wonderfully right), canto-rss (RSS reader/server), ncTelegram (Telegram client), turses (Twitter client), centerim (multi protocol IM client), weechat (IRC) and iftop (traffic monitoring for me), alpine (mail client) and mc (file manager). At least they're the tools termux starts with on my VPS and I pretty much live on there.
check out http://urwid.org/ for python
Can every technical article be written this well and using this way? It's a far more interesting read than the title suggests...
I love htop and use it daily on desktops and servers, but I admit I didn't know half these things. Great practical writeup!
Well, his 'Extreme edition' is actually not so extreme. This is how Debian base installation looks like.
A server can perfectly run without dbus, accountsservice, logind, policykit and acpid. And, unlike he wrote, timesyncd NTP synchronization works well without dbus.
(Debian base has also cron and rsyslog running by default, for legacy reasons.)
> It turns out that id gets this information from the /etc/passwd and /etc/group files.
$ strace -e open id 1000
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
open("/etc/group", O_RDONLY|O_CLOEXEC) = 3
A more generic tool is 'getent', which will query all the underlying databases for you. For example: "getent passwd" or "getent group".
Nevertheless, using strace to get a first approximation is an excellent exercise :)
The problem isn't really that they used strace+id, it's that they abbreviated the output a bit too much. A better clipping would have been:
Where nsswitch.conf informs it how to look up the information; the default on most systems being$ strace -e open id 1000 ... open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3 ... open("/usr/lib/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 3 ... open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3 open("/etc/group", O_RDONLY|O_CLOEXEC) = 3 ...
which tells it to the functions in /usr/lib/libnss_files.so to access the "passwd" (user) and "group" databases. libnss_files.so uses /etc/passwd and /etc/group respectively for these.passwd: files group: files
> If I had two cores, my CPU usage would be 50% since my computer can run two processes at the same time. The load average of a computer with 2 cores that has a 100% CPU usage would be 2.00.
If I recall correctly the CPU usage would still be 100% but could go up to 200% if both cores are fully used.
I am getting feedback that I should not be using the terms CPU usage and utilization when talking about load averages. I will update the post later.
My understanding from my Solaris days was that CPU load is the number of processes queued for CPU time over the given period. It doesn't related to usage directly at all. (Other than higher cpu speed and io through out leads to a reduction in load because tasks finish quicker)
That's correct.
Nice explanation, but too much to remember in practice.
If only the UI had something like hover-tips to show what each field actually means...
kill as a shell built-in also allows you to pass it job numbers rather than PIDs. For example, kill -9 %1 would kill the first process you sent into the background, either with & or ^Z
I like the investigation into various cruft that Ubuntu installs (and runs) that can safely be turned off.
Goodbye polkit!
Uh, wait. Can't really get rid of policykit-1 on Ubuntu desktop, I think. Maybe server.
Pretty OT, but does anyone know if that's true (or worthwhile?)?
To be a bit snarky, polkit is freedesktop dbus centric (and thus over-engineered) reimplementation of sudo for the GUI.
It uses a session tracker (used to be Consolekit, these days it is systemd-logind) to tell if something is running under an active user session and, and checks that against a bunch of rules (stored in XML, natch) to see if what is being attempted is allows (and if not, spawn a password prompt so the user can elevate their privileges).
If you want to do anything "complicated" like powering down your PC by clicking a button in your DE of choice you need polkit (and entourage) to be present these days.
Its one of those CADT things that seems to lead to "year of the (linux) desktop" never happening.
This is fantastic write up! I've bookmarked this one for future reference. Kudos to the author!
I tend to use "atop" because it gives me a better overall picture of my system. But, there is something really cool about "htoo" that many users don't realise and that I hadn't realised until somebody pointed it to me recently:
You can use your mouse/TrackPoint in htop to click on menu options, processes etc...
atop is way more superior than htop, specially with the log history. I just use htop when I need to check cpu usage or a particular process. But atop gives a better system status overview.
Beware -- not only about htop :)
Love this post, first function explanation for 'uptime' module gives a concise 'strace' demo showing how to determine the files the standard 'uptime' binary opens.
Really excellent example of showing your work!
If you have python available, I find `glances` to be a great tool (`pip install glances`)
Has disk, network, proc, mem, cpu, etc. all wrapped up in a a curses interface. very slick and useful.
love it, good overview over all functionalities of htop and easy to follow
Good job.
Wow, that was very interesting and much more information about how Linux works in a general sense than I would have expected a discussion on htop to contain!
lsof (LiSt Open Files), which used to be a third-party tool and needed to be install on Unixes, may be built-in by now in some, and fuser (built-in, usually) are also useful tools for investigating processes and files that they open (including sockets), and so on. lsof was written by someone at HP, IIRC. Has a big list of command-line options. Possibly some of the other tools mentioned in this thread call lsof internally.
this has nothing to do with htop, it might as well have been top explained, man ps for crying out loud.