Asahi Linux Progress Report: January/February 2021
asahilinux.orgFeel free to sponsor Asahi Linux here - https://github.com/sponsors/marcan
> On the other hand, the rest of the 64-bit ARM world has largely converged on two competing standards: UEFI + ACPI (largely used by servers running Windows or Linux),
It's not supposed to be just for servers, the Arm Base Boot Requirements (BBR) require it for any non-embedded use https://developer.arm.com/documentation/den0044/latest
The new ARM64 laptops follow BBR so they have UEFI+ACPI, which means they can run standard Windows or Linux distributions from Red Hat / Canonical. By using device tree instead, it means other operating systems won't be able to run out-of-the-box without some modification. I can totally understand and respect the decision not to touch ACPI, I just think it might not be a great long term decision.
I spent about five minutes wandering through the doc you linked trying to figure out how they define "embedded" and gave up.
In any case I've done plenty of hairy "embedded" development stuff (kernels, uboot, weird hardware drivers, etc) but I draw the line at device trees. I may have some kind of PTSD or something but I just refuse to deal with them in any way. Of course passing a path to a compiled binary device tree is fine but I refuse to do anything else.
I've met plenty of people that have no problem with them, make a living working with them, etc but other than the niche that is device tree expertise (and income) they may be experiencing some kind of Stockholm Syndrome.
I truly hate them and while they'll likely always have a place in the (hopefully) deep deep deep "embedded" world the sooner UEFI + ACPI generally takes over in the ARM ecosystem the better.
So you think that ACPI tables are better than fdts and are fine with writing them instead of fdts, when it comes down to it?
Like when you need to write support for a new board or if you need to patch your ACPI tables, because of bugs in there: https://www.kernel.org/doc/html/latest/admin-guide/acpi/init...
`PTSD` sound more like an ACPI table name ;)
It's not really a question of firmware support; the standards need amending to support all of this, and then OSes need to implement core support. Basically, there will never be a way to run Windows natively on Apple Silicon without someone pushing to have a pile of new Apple-specific stuff added to ACPI, and then getting Microsoft to implement support for it. The only way to run Windows on Apple Silicon without all of that happening will be inside a VM that emulates a more traditional platform.
The latter might happen at some point; m1n1 is already going to become a minimal hypervisor for experimentation, and that could easily grow a GIC-to-AIC mapper and an HVC-based PSCI implementation some day. At that point, sure, throw some ACPI tables on top and Windows will boot, with decently near-native performance (assuming someone writes all the actual peripheral drivers, of course). Just not really natively. Incidentally, we're making all of our bespoke driver code dual-licensed (MIT/GPL), so people are free to take it and port it to BSD, Windows, or what have you.
Of course, Windows already works in QEMU today under M1 macOS (with emulation of the rest of the hardware as variants that Windows supports), and it should just work on Linux/KVM modulo a silly patch that's still missing related to a smaller than typical physical address space on M1. So full-fat VMs are not a problem, but obviously that has nothing to do with the bare metal boot chain.
As a community project, we can't exactly throw stuff at the UEFI Forum to get them to go down that road and specify representations for all this Apple-specific hardware in ACPI. However, Apple is a member of the Forum, so perhaps they should be the ones to do this :-)
Mainstream Linux distributions should work fine once support trickles upstream, as we will provide UEFI via U-Boot, and any reasonably generic ARM64 kernel should support devicetree and our devices (if they don't, someone should file a bug with RH/Canonical, as it would be completely silly if they don't turn on those config options in their kernels). OpenBSD also supports DT, and is already booting on m1n1.
Edit: I've just been reminded that RH/CentOS deliberately build their kernels without devicetree support, to force vendors to implement ACPI. This is, unfortunately, also mooted by the fact that they build their kernels configured for 64K page sizes, which the M1 does not support. Those kernels will never work on M1 machines, not even in a VM; effectively they target a stricter platform subset / standard than general ARMv8-A. Presumably they do this for performance on large systems, at the cost of less efficient memory usage. Maybe if Linux ever gains multi-page-size support on ARM64...
Incidentally, there is one other platform in the same boat, that deviates from the ACPI standards (no GIC): the Raspberry Pi 2 and 3. Microsoft are using proprietary ACPI extensions and patches to core Windows code to make that work. So we'd need at least another nonstandard ACPI extension and explicit Windows kernel support added by MS to make that happen on M1.
If Broadcom of all companies switched to the GIC, I wonder if there's any chance Apple would do the same in a later chip – in a less rushed dev cycle that's not just "quickly desktopify the iDevice SoC". (Also SMMU along with the GIC…)
Of course the unfortunate reality is that there is just no motivation for them to do this. Goddamn vertical integration! But.. maybe Boot Camp could motivate this? Though so far it seems they're just pushing virtualization instead.
Thanks for the great response. I didn't realise that the bindings deviate so much from what is already specified in the standard.
> However, Apple is a member of the Forum, so perhaps they should be the ones to do this :-)
Agreed.
> build their kernels configured for 64K page sizes [...] Presumably they do this for performance on large systems, at the cost of less efficient memory usage.
My guess is that they do that to enable a larger amount of memory (very relevant on servers, which are RedHat's main target market). AFAIK, both the LPA (large physical address) and LVA (large virtual address) extensions, which allow using 52 bits instead of 48 bits for the memory address, require using the 64K page size.
Doesn't the MacOS kernel use some bastardized version of device tree? I'm not sure of their commitment to UEFI/ACPI..
On ARM it does, as I mentioned in the article (though it isn't directly useful to us, as it's too different from Linux ones).
But yeah, assuming there will be no Intel machines in Apple's line-up at some point in the future, I wouldn't be surprised if they drop out of the Forum.
They even use device tree on x86 too (in addition to ACPI/EFI). The plaintext password and user guid is passed in chosen fields for instance when boot.efi unlocks an FDE disk.
It pre-dates Device Tree support on Linux for Arm. Both have a common ancestor (OpenFirmware, notably used for PowerPC Macs)
I remember getting to the midpoint and thinking "Man, all of these PowerPC-isms really takes me back to the Wii modding days!".
Imagine my face when I read this line: "m1n1 traces its past to mini, which is a minimal environment that I wrote for the Nintendo Wii’s security CPU".
As if there was any doubt: Hardware reverse engineers are a very special breed of dedicated people. There aren't many of them, but those that are around tend to stay around!
Yeah marcan helped hack the Nintendo Wii. See his original talk at 25C3 here [1].
The other guy in the talk [1] is the late Ben 'bushing' Byer. I only knew him from his online work, but it's still very sad that he passed away in 2016 [2]. I'm sure if he was still alive, he'd also continue to be doing great reverse engineering work!
How did they figure all of this out from a non-documented platform, especially the specifics of Stage 1 (LLB)? [1] It blows my mind.
That's mostly just imported from existing iPhone research, where people have been reverse engineering iBoot (from exploitable phones, since it's encrypted) for a long time now. Combining that with what is known about how the M1 works from observation / the boot config data structures yields a decent picture, without actually seeing the code.
Here's a fun one though: how I found and documented the Apple-proprietary memory compression/uncompression instructions.
https://twitter.com/marcan42/status/1362450439845781505
A lot of the hardware research ends up looking like this; twiddle random bits and see what happens. For more complex drivers (e.g. the GPU), my plan is to run macOS under a thin hypervisor built on m1n1 that can log hardware accesses.
>my plan is to run macOS under a thin hypervisor built on m1n1 that can log hardware accesses
The fact that people like you exist in this world really makes me happy.
If you read the post, the guy says that is the same approach used to make the nouveau nvidia drivers.
They do it in-kernel (with a Linux patch and the Linux drivers) instead of a hypervisor, but it's the same overall idea.
I could do it that way too (XNU is vaguely open source, and the most recent source release is buildable for M1), but I honestly find the idea of writing a little hypervisor a lot more appealing than learning to hack on XNU, and it's probably a similar amount of effort all things considered.
Technically, m1n1 is already a "hypervisor" for very small values of hypervisor; as of last week you can `chainload.py --el1` another m1n1 into VM guest mode and run a kernel from there, and will get exception report dumps if it crashes in a way a hypervisor would notice, but there is no virtual memory in use. So it's mostly just taking that, making some page tables, adding exception handlers to handle page faults and and log MMIO as I want to, and somewhat virtualizing the CPU startup hardware (which is the only thing we can't just pass through because we need to control the CPU boot process). And making chainload.py able to load macOS kernels (needs some extending to support missing Mach-O features and handling some memory structures properly).
I've always enjoyed the Dolphin Progress Reports and as intended this had a similar feel. I was never that interested in Wii emulation, but I have an M1 Mac that I would love to run Linux on.
I signed up as a GitHub sponsor based on this post. Thanks for the great work and write up marcan!
I got a real kick out of "our ARM Mac bootloader is actually the thing you used to backup and restore your Wii NAND a decade ago".
One of the things I did when porting it to the M1 was "rm nand.c"... :-)
This thing actually goes back further than the name "mini" and BootMii; in the beginning it was a never-released thing called "ios_stub" and actually served the very same purpose on the Wii, to experiment with the hardware over a USB Gecko (which was a USB serial-like interface that plugged into the memory card slot) using the very same Python approach. That code hasn't changed much in checks Git history 13 years... though it obviously flipped endianness on its way from the Wii to other ARM32 platform experiments, before making it to ARM64 and Apple Silicon [1].
The Python side is still mostly the same too, other than getting ported from Python 2 to Python 3 and growing a bunch of utility functions. The Python-side malloc implementation actually got written when I was using all this to experiment with a Chinese MP4 player. That version ended up being called "minimp" [2]. Another thing that happened on the way to the M1 was deleting the "P_RENDER_BUNNY" command [3].
So yeah, what I'm doing now on the M1 is literally, down to the code, shell.py, and proxy command names, the same damn thing I was doing 13 years ago on the Wii. AES engine [4] back then, IRQ controller [5] now... though evidently I put a bit more effort into the MediaWiki register documentation templates back then; the GitHub wiki is a bit more limited... :-)
[1] https://mrcn.st/t/ios_stub_vs_m1n1.png
[2] https://marcan.st/2009/06/sunplus-spmp305x-media-player-hack...
[3] https://www.youtube.com/watch?v=3tg7KSSUl8Q
This read is amazing. This feels as much an exploration as visiting another planet or going to the extremes of the earth.
I wonder if it is possible for any Linux laptop with USB-C to deliver USB-PD commands to M1 USB-C devices in order to get the serial stuff working?
It depends on whether the PD controller chip supports sending SOP'DEBUG/SOP"DEBUG packets, and, for those that do, whether it is exposed to the host OS or the firmware running on it does so.
However, even once you have that working, you still need to connect the other pins to a serial port adapter, and I don't know if any other laptops implement this in a compatible way, so you would probably still need some kind of cut-up cable that breaks out the serial wires.
So what is different about an M1 that lets you do this without the serial port adaptor?
M1 machines expose serial over Type C, so of course if you cross-connect two M1s they can talk to each other over serial. Conveniently, one of the pin pair options you can select is SBU1/2, which are cross-connected on standard cables, like a null modem serial cable, so you end up with TX connected to RX in both directions and it just works.
This is not a standard Type C feature, so unless another machine happens to also implement a 1.2V serial port compatible with Apple's version and pinout completely by chance, it won't work.
It only works on a specific port on Apple devices (the same one used for DFU, which has the fun debug features like this one), not on all the others.
I see. So will Linux on the M1 have enough driver/etc support to do the serial stuff?
Sure; Linux already has a driver for a variant of the USB-C chips in these Macs, so normal functionality should work with minimal changes. I don't know if the driver exposes the low-level command protocol to userspace off the top of my head (since we need that to send vendor-specific commands), but if it doesn't, we can probably get away with just using raw I²C access to the chip directly, via i2c-dev. Alternatively, we could add an interface, e.g. a debugfs entry for the Type-C driver that exposes raw commands.
Not sure what approach we'll end up with, but there are certainly multiple ways of doing this under Linux.
I noticed the author is active on this thread, so
> I recommend Will Deacon’s talks, such as this one and this one.
They're both the same link! Is the talk that amazing, or is there a second one? I've been looking into ARM lately, and I'll take anything that'll help me understand memory ordering/synchronization.
Argh, thanks for catching that! It's fixed now :)
Is there an RSS feed available for the blog? I cannot seem to find it mentioned anywhere on the website.
Looks like their website is built with Hugo[1], and Hugo automatically generates RSS. So you can grab it manually if they don't provide a link for it by appending index.xml in the section URL.
Here's the one for the blog: https://asahilinux.org/blog/index.xml
Might be nice if we have an actual link to the RSS somewhere in the blog though.
I'll add a link later; this was on my TODO list but I didn't get a chance to do it yet.
Link (and header <link>s) added!
I now see it on the blog root but it would be great if the actual article was also marked up with the RSS feed. This is pretty standard and I only bothered looking at the blog homepage because the writing was really good :)
Doesn't seem so but you could add an enhancement on the repo.
Does that Trap SMC bit in SYS_APL_HID1_EL1 let you setup a poor man's PSCI even though there's no real EL3? Or is that something different?
And do you think that all those HIDn registers are more PASemi legacy? The PowerPCs are known for HID SPRs being a grab bag of random one off features and chicken bits in a very similar way.
It still needs to trap to somewhere - e.g. a hypervisor at EL2. But then PSCI can run over HVC anyway, so it doesn't make much of a difference, though being able to pretend EL3 exists from EL2 is nifty. We still lose VM functionality if we run all this at EL2 and the OS at EL1.
I do think the HID register naming scheme comes straight from PowerPC. So does their "DART" IOMMU (again just a name thing, unrelated to the old PowerPC DART IOMMU). A much more interesting question is how much the M1 design directly derives from those older PASemi PPC cores (beyond names and such which could just be a little nod); that's much harder to know, but I'm interested in any hints that might point in that direction :)
WRT SMC, I guess I figured it'd be used to put some firmware code in EL2, but not be the exclusive owner of EL2 if that's possible. Cut out a reserved region of mem in m1n1 or u-boot, and an EL2 SMC call would simply be a jump to that code without a mode switch. You'd see that kinda thing in some older ARM RTOSes which would run everything at the same privilege level, but use SWI as a core executive invoke. Of course now that I've typed this all out I realize that I was kinda assuming that there'd be some SMC-hack specific vector table that it could own, but it probably traps to the normal EL2 one which I think is what you're hinting at...
Anyways, is there anything that a systems engineer with RE experience and isn't looking for any of y'all's donations can do to help out? I'm between jobs and it seems like fun.
On a side note. Is there any benefit to a micro kernel operating system anymore?
Linux is a monolithic kernel.
One drive of course, is to have a micro kernel that will allow you to update your operating system as needed, without the need to reboot.
Can this still be accomplished with a micro kernel system, or is this now an obsolete method for operating systems design?
Most of the modularity and security advantages still apply today. Performance is also good, unlike early implementations. Google is developing a microkernel operating system called Fuchsia, possibly to replace Android https://en.wikipedia.org/wiki/Google_Fuchsia
Is it because of runtime speeds (the slowness) that prevents microkernel operating systems from being more prevalent? Or is it just because Linux is good enough, and it became popular and free?
There is a speed impact but it's mostly negligible, especially on modern hardware. The main reason is that building an entire operating system requires enormous investment beyond what most companies are willing to spend. But Google seems to be willing. And even if you succeed, people will inevitably chase you on Linux app compatibility.
Some well known open-source examples are seL4 and RedoxOS.
The speed is not really “mostly negligible”, it’s just something that you accept when you design a microkernel.
Agreed. More services does mean more context switching. The most pure microkernels will have many, and the more practical ones may consolidate some services to reduce context switches. Also, some types of IPC can have a very lightweight context switch; they typically transfer only a few words of data, but this is usually mitigated by shared memory.
Ah, the author is here, and leaving feedback here seems easiest -
> Our current Linux bring-up series is in its third version
There's a #fragment in the link, and it goes to a less interesting part of the page. I only circled back after looking at the first and second versions.
The links should go to the same style page now, thanks!
Confirmed!
I see the booting process is quite complicated. Isn't ARM and PowerPC chip similar enough that many PowerPC linux can be ported over to use the M1 chip?
The M1 boot process isn't really much more complicated than the boot process of many other embedded ARM systems. Linux already works on ARM; everything I talk about here is about dealing with Apple's particular flavor of hardware and their non-standard boot process. That doesn't have anything in common with PowerPC systems either, so there isn't anything to be gained from PowerPC Linux (other than the I²C driver which I do mention).
Good work and thanks. Looking forward for the 2nd report !
An update from the Corellium folks working on Linux on the Apple M1, looks like they already got everything working:
Everything except the GPU (and display controller)... which is a bigger challenge than every other driver combined :) (their kernel runs only on the boot-time framebuffer so far)
Unfortunately, we've yet to hear any feedback from Corellium (they've been CCed on my upstream submissions), nor have they interacted with the Linux kernel community in any other way, so I have no idea what their plans are.
So far, after working with upstream on solving the core problems I mentioned in this article, the M1 support series we are submitting supersedes Corellium's patches for FIQ, the nGnRnE issue, SMP, AIC, UART; we settled on different solutions for all of those from how Corellium did it. I²C is also another one that will be superseded most likely (Corellium wrote their own driver instead of improving the PASemi one; it doesn't seem like there are any show-stoppers that would warrant that approach). I don't know what they're going to do moving forward; perhaps they will re-base on top of mainline and drop their conflicting patches, or perhaps they will attempt to maintain their kernel as a Linux fork...
Hmm, I see them participating on linux-arm-kernel list, nothing since January though:
Mohamed is not a Corellium employee, but rather a third party who early on volunteered to attempt to upstream some of their patches.
The only e-mail from a Corellium employee to Linux mailing lists, to date, is this one:
https://lore.kernel.org/linux-arm-kernel/ce74bb29-1237-b0e7-...
This is the most puzzling bit to me: nobody from Corellium participated in the upstreaming discussion about their own code either.
They probably just don't have the time to upstream it themselves. I think it's just a publicity stunt for them.
This is an incredible and high quality post. Kudos
> Welcome to the first Asahi Linux Progress Report! In this series we’ll be taking a page from the Dolphin playbook and giving you monthly updates on the progress of the project.
Nice work! let's see the progress...
~60,000+ words later...
> We could keep talking in depth for another 10000 words, but alas, this post is already too long.
Please no. A TL;DR is just enough for the busy. The Dolphin report even shows more screenshots and diagrams at least.
We have already seen how complex the reverse engineering, booting, discovery and bring up process of this M1 chip running on Linux is, which the first step is already a complicated hellhole in itself, because its Apple. For explaining all of this, you need diagrams of the whole process which would be much better than us deciphering all of these CPU internals / peripheral technical soup.
Just put a TL;DR at the top next time or some diagrams for those interested in helping out.
Other than that, great progress.
"Those interested in helping out", at this point, are generally only within the demographic of people interested in consuming 60,000 words of necessary braindumping/orientation/synchronization.
Device bringup is, as you say, complex. This complex.
Well, Dolphin is an emulator, so screenshots and videos are quite an obvious tool to demonstrate game emulation issues... and it's also easier to boil down most changes to "fixes X issue on Y game" for a mature project; that model doesn't map quite as well to the early stages of an OS port project, especially since most of the stuff in this first report is literally required to get anything to work at all. The one screenshot of the framebuffer with 8 penguins at the end effectively represents the result of the entire prior saga, and everything before it would be various flavours of "black screen", "kernel panic", "the serial port stops working", etc... not very interesting to show!
What kind of diagrams are you looking for? There's lots of things that could be diagrammed here, but comprehensively explaining every concept involved would turn this into an embedded systems course, diagrams or not. What I tried to do was give a brief introduction to concepts that are relevant to the issues we ran into, and have links for those who want to go deeper. If you have specific suggestions of bits that are hard to grok without diagrams though, please do let me know. It's tricky knowing what is most confusing to other folks when you've been neck-deep in this stuff for weeks.
The alternative to this long-form post is to just have a laundry list of things that work today, but I don't really know how I would get across what the challenges were without going into at least some level of details like I did here. I figure that if I'm going to do that, I might as well make it a more educational endeavour. Of course, if all you want to know is what works and what doesn't, it may not be for you... I'm open to suggestions though!
Keep in mind that a lot of the early work ends up being "how to find the right solution to problems" (and the post goes into more detail about this); the current feature support status of Linux on M1 almost hasn't changed for the past 30 days, because instead I've been re-visiting and re-working the code into a form that is upstreamable, as well as building tools and chipping away at little details. It's a lot of yak shaving, but it's all things that need to happen sooner or later. Unfortunately, it doesn't really tick boxes in a TL;DR bullet list of working hardware.
For my part, I loved the long form, in-depth post, and I learned a lot. I'll admit breaking it up visually with some diagrams or photos is tempting (maybe a photo of your serial setup?) but I felt the explanations were all clear without it.
Ah, that's a good point. I actually need to port over a Hugo shortcode to handle little image boxes for this kind of thing; once I have that it probably makes sense to add a little photo of my setup as an aside, not so much as an explanation but rather for visual interest.
> instead I've been re-visiting and re-working the code into a form that is upstreamable, as well as building tools and chipping away at little details. It's a lot of yak shaving
This is exactly what needs to be done to make this a viable project, and leaving this stuff out is, for me at least, what categorizes Correlium's project as a mere publicity stunt rather than a serious porting effort.
I liked the long form too because there is a lot to learn there. I really appreciate what you do. Actually one of the decisive factors for buying M1 was the fact that you work on making it gnu/linux friendly. Perhaps some shorter form in the beginning could also help for better understanding before one can comprehend all the details but if we ask for that it's better be done in a respectful way. Thank you for your hard work.