Red Hat Enterprise Linux 8 released

Some of the backports aren't nearly as fast or performant as using a newer kernel though.

For example eBPF was back ported (and also in CentOS), but running a syscall heavy work-load in a docker container on the older kernel about 50% of the CPU time was spent in the kernel filter.

I ended up moving our entire CI/CD platform to Ubuntu 18.04 and the performance issues went away and my workloads now run at full speed without slowdowns.

RHEL 8 comes with the 4.18 series of the Linux kernel that is already EOL. That's a shame and once again it will fall behind quickly :/

Colorful dmesg output was already available on Cent7, just not the default.

The main source of stability for RHEL isn't that any one arbitrary version of a package they ship is better than another one, or that their patches on top don't suck. It's that they ship a long-term "stable" (as in "doesn't change much", not "sucks less") set of software for production use.

Thus, if you install some random vendor's shitty software you can rest assured that the version of libcurl and 50 other libraries they depend on is something they themselves have tested on RHEL.

The same goes for hardware that you buy. When you buy e.g. Dell rack-mounted servers you can safely assume that the open source driver version maintained by the vendor shipped as part of the RHEL kernel is something that's seen extensive production use, unlike the latest upstream kernel, or whatever "in-between" Debian et al are shipping.

Am I recommending you use RHEL? No, it's not the right answer for everything, and I certainly have my share of RHEL scars, including a couple of times where a mundane bug in my program turned out to be a kernel bug (one in RHEL's own shitty patches, another "known" bug with their ancient kernel).

But this is the reason to use it, and why some major commercial vendors say "we support Linux, as any distro you want as long as it's on this list of RHEL versions". They just want to deal with those kernel/library versions, not any arbitrary combination out there in the wild.

>layering years of patches over top actually generates more stable, more secure code

Well, I think your definition of 'stable' is different than what RHEL/Debian customers think. Stable isn't seen as "doesn't have bugs", its "works predictably". Which is a subtle but meaningful difference.

> But seriously, has anyone ever empirically verified that the Debian Stable/RHEL model of shipping a bunch of really old packages and then layering years of patches over top actually generates more stable, more secure code?

Debian has released a new stable version every 2 years for the last 14 years. RHEL/CentOS are the only ones on a 3-5 year cycle.

Stable doesn't necessarily mean 'doesn't crash' what it means is that the API/ABI interface is stable.

EG: let's say libfoo.so.1 implements DO_FOO; libfoo.so.2 implements DO_FOO2, but not DO_FOO. In this case, anything you need that links to libfoo.so.1 and needs DO_FOO would need to be patched, recompiled, and shipped out to all your customers. For the distribution provider, this is not really a huge deal. But RHEL is merely the platform. The value-add is that 3rd parties can write software and compile against libs and know they're not going to break arbitrarily.

Similarly, if you've ever written a kernel driver, you'd know that kernel function names and signatures can change from release to release. The same example above applies to kernel code as well. So, compiled binary drivers would have to be patched and recompiled, and shipped out. If you're writing a driver for a network card, would you prefer having to ship your (non bug) driver updates every few months, or every few years?

It doesn't generate more secure code—as you say, patches themselves may have bugs, and way fewer people are looking at the patched branches. Active development happens on HEAD, and dodgy code is often rewritten before anyone goes actually looking for bugs (security or otherwise). Many years ago I helped with a paper on how the practice of applying only "important" security bug fixes doesn't work: https://arxiv.org/abs/0904.4058

But the goal of the long-term-stable approach isn't security or stability per se: it's striking a tradeoff between operator work and risks to security and stability. You could, of course, snapshot Fedora (or Debian testing, or Arch, or whatever) from 2013 and keep running it. Nobody is stopping you, and it'll still run on new machines. And then you have to do zero work to keep your system up-to-date, but you'll likely have tons of security and stability bugs. On the other extreme, you could run Fedora rawhide (or Debian unstable, or current Arch, or whatever) and update nightly, which would mean you get security fixes as fast as possible (they're almost always developed on HEAD and backported to release branches), and you get performance and stability fixes that people haven't deemed worth backporting, but you also risk API-incompatible changes that break the actual applications you care about. You'll need to set up really good CI to make sure you have coverage of everything in your application, and it's not just a matter of automation: you'll need a well-staffed team to respond quickly every time that CI goes red, figure out what changed, and update your applications to match. (And, of course, you have the risk of security issues in new code that hasn't been subject to public scrutiny yet—the inverse problem of security issues in old code that's no longer subject to public scrutiny.)

The goal of a long-term stable distro is to be in the middle of those two, to give you something that changes rarely (stability in the sense of "no surprises," not "doesn't crash in prod") but often enough that you get major, identified security fixes and particularly safe performance (and stability-as-in-"no longer crashes in prod") fixes.

And yes, part of the goal of a long-term stable distro is that it provides you measurable security and stability over unmeasurable but potentially greater security and stability. They don't fix every CVE, but they do fix the flashy ones. You can look at it cynically and say, this is the distro for people who want to tell their boss "Yes, we patched Heartbleed and Shellshock" but don't inherently care about security. But on the other hand, flashy vulnerabilities are more likely to be exploited, so it's not a particularly bad tradeoff.

I guess it's just what I'm used to. I'm also running my own NTP stratum 1 server from a GPS source using ntpd, but that's on Raspbian (https://rwmj.wordpress.com/2015/07/04/stratum-1-ntp-server-p...)

There's been a huge effort to get the base RHEL image size right down, so obviously getting rid of Python would help there. As for why we need to reduce the size of the base image, the answer is - as always - because containers.

Ansible the client, or the target? Ansible doesn't need anything installed on the target (except sshd). For the client which presumably would be installed explicitly on far fewer machines it would bring in whatever Python it needs. I don't know if it uses System Python or a module however since I don't have it installed on RHEL 8 right now.

Installing a python interpreter is one package. If you just want to write a quick script using the standard library, it is cheap as all hell.

Is python being promoted as a wholesale shell replacement? There certainly are plenty of overlapping usecases, but shell is a better fit for many of these tasks. Until something like ipython can replace the interactive shell, I don't see python replacing shell scripts entirely.

It just wasn't the practice back in the day. Does redhat ship a "platform-perl"?

The difference with system shell scripts is that you can't accidentally clobber libraries that the system tools require to work - shell scripts don't really have any notion of installable libraries.

There isn't a history of issues with people wanting to update to a newer bash causing issues though.

Generally sticking with whatever bash your distro comes with is fine, whereas the services you deploy often depend on a particular version of python.

There absolutely is a platform sh interpreter. That is why when ksh came out, it was called ksh instead of sh, so that /bin/sh would still function as expected (same with csh, bash, zsh, etc).

And many of these will emulate /bin/sh behavior when called as such.

Ubuntu had dash - a minimal sh implementation.

For the system I think it is a good idea to limit library interactions with things related to keeping the system running or booting.

Thanks for the detailed response.

openntpd can set the clock on startup, but it requires a non-default `-s` option [0]. In chrony it's optional and controlled by an `initstepslew` parameter [1] which also considers a threshold to determine if the clock needs a large adjustment or if it's fine to just skew it as normal.

0: https://man.openbsd.org/ntpd

1: https://chrony.tuxfamily.org/doc/3.4/chrony.conf.html

My terminal isn't configured to use any colors. It's just xterm. The only colors configured are background and foreground, and those colors various tools insist on as default are always clashing with them (and with my vision too). There are no color environment variables enabled, or anything else indicating that colors should be used, but even so colors are coming out of various command line tools lately.

> If you're a Red Hat customer, you could file a support ticket to get it pulled back in.

I have never spent $1 of my own money on Red Hat. After seeing this, I never will.

That's not fair. If you want to blame something for that, blame software patents. Some stuff used to be a huge minefield because of that.

No. https://twitter.com/fkooman/status/1125767993764536321

Mint has been doing that for a while.

And I believe Ubuntu has finally started doing it as well in the most recent release?

RHEL comes with a price tag (although I think they now have free developer licenses.)

Also RHEL development moves sloooowly. This is a feature and one of the main reasons to go with RHEL instead of not only unsupported distros but also supported-but-faster-moving distros, kind of like on Windows LTSB (I know to little about both to compare them, but enough to know that in certain organzations the promise that it will stay the way it is and by default only receive security updates is a huge feature.)

Oracle has some pretty beefy hardware. They can afford the cycles. ;)

I know there's quite a bit of testing involved to verify their unbreakable kernel stuff and ksplice stuff is compatible. I would suspect there is also the spacewalk integration stuff that is fairly different than redhat's satellite stuff.

DTrace esp.

Yeah, I know it's cockpit. I'm not sure what it brings to the table. It's already possible to lock down webmin pretty heavily if I want to trust a windows admin to do linux.

But does it at least do that 5% well?

If they are in the ubi namespace, they are freely available. The Container Catalog also will tell you if you can pull without a login when you look at the details of a container.

The article is clear on it being allowed.

Try the developer download: https://developers.redhat.com/products/rhel/download

There is a telling. (Except for CentOS 6. Wonder why it delayed so much.)

https://en.wikipedia.org/wiki/CentOS#Older_version_informati...

(Expand the table for older releases.)

The sale is not completed, yet.