Achieving an open-source implementation of Apple Code Signing and notarization
gregoryszorc.comI think Apple's official code signing tool is also open source? https://github.com/Apple-FOSS-Mirror/security_systemkeychain...
xar is another tool for codesigning Mac installer packages, which runs on Linux. I've used it successfully in the past.
https://users.wfu.edu/cottrell/productsign/productsign_linux...
Some of Apple's code signing is open source (mostly in SecurityFramework). But not enough is open source to be able to build a modern `codesign`. The source you linked is ~10 years old and woefully out of date, for example!
I don't believe there are any Apple open source references for how notarization works (at least none before it was a public App Store Connect API).
There are even times when Apple's open source releases trail functionality they are shipping in macOS. For example, Apple recently added an alternative DER encoding of entitlements, which are expressed as a plist. I don't believe Apple ever published code for how the DER encoding works. Instead, we needed to use Apple's tooling as an oracle to incrementally derive the encoding.
Good pointers indeed -
Apple's code signing tool is open source, but heavily dependent on Cocoa/OSX libraries like CoreFoundation. I'm not sure if it works with any of the open-source reimplementations of these libraries, although I kind of want to try now, just for fun - has anyone done this already? Plus, it's not Written in Rust (TM)!
`xar` only signs packages, not Mach-O binaries, at least as far as I know?
XAR signing is effectively just an RFC 5652 CMS signature plus some minimal data structure manipulation. Code at https://github.com/indygreg/PyOxidizer/blob/faa7dfcea5d66bf5....
Mach-O and bundles, by contrast, require a myriad of additional data structures requiring thousands of lines of code to support. To my knowledge, nobody else has implemented signing of these far-more-complicated primitives. (Existing Mach-O signing solutions just do ad-hoc signing and/or don't handle Mach-O in the context of a bundle.)
> Existing Mach-O signing solutions just do ad-hoc signing and/or don't handle Mach-O in the context of a bundle.
I can assure you that saurik's ldid[0] does. Or the updated fork that I maintain at ProcursusTeam/ldid[1]. You can use -K to sign with a cert. You can find full documentation in the manpage[2].
[0] http://git.saurik.com/ldid.git
> Plus, it's not Written in Rust (TM)!
It could be co-authored by Xi Jinping and Mark Zuckerberg in COBOL and hosted on NPM, and that wouldn’t stop me from using an open source codesign alternative that runs on Linux.
Can you elaborate? Since I mostly work on macOS stuff, I don't understand the workflow that leads to needing this sort of tool. What's the process of writing software for either macOS or iOS that doesn't involve needing macOS or iOS at some point? Like you probably need to test on those systems if you're supporting software for them, right? Is it because development is done on Linux, and only testing is done on, say, macOS? Sorry if this is a stupid question. I just haven't run into this sort of situation before and am trying to better understand it.
There are a lot of different reasons a project would want something like this. In general though, I think the most common is CI and build automation. Currently, build automation for Mac requires doing some shady stuff with VMs, or renting or buying and maintaining a physical Mac. Being able to do everything on Linux vastly simplifies the infrastructure needed to support that.
Even if you're just writing standard native apps with Xcode, being able to build, test, and deploy your app every time you push to the production branch in your Gitlab/Github repo is valuable. With codesign on Linux, you can build that on top of a simple, ~5mb alpine linux container.
> I think the most common is CI and build automation
Common practice for CI is to actually run tests, though. How would you validate your macOS/iOS builds are functional without still requiring VMs or real hardware?
If those instances are only for tests/validation it could simplify some overall pipelines quite a bit.
We have a pool of MacOS hardware to run tests, and another pool to perform signing. The latter is a security risk so very tightly controlled and locked down. They are a pita to maintain.
Being able to sign on Linux will allow us to re-use the existing signing infrastructure we use for literally every other platform other than MacOS. It'll be more secure and much less maintenance.
You could use this in addition to regular macOS machines. You can run tests from un-trusted branches, without worrying that something will expose the signing keys.
Then after merging, you have a second machine / set of machines that only does signing. It increases the difficulty of an accidental or intentional leakage of your sensitive keys.
(Note: I have not tested this yet)
That doesn't justify the build part.
CoreFoundation is (partially?) open-source and cross-platform now: https://github.com/apple/swift-corelibs-foundation
That’s not CoreFoundation, a C library, but a Swift reimplementation of that library, only to be consumed by Swift (import Foundation).
CF was always (approximately) a multiplatform and open source subset of foundation (which is the thing this isn't). I think this is a superset and extension of the original core foundation under more favorable license terms though.
The "original" CF can be found here in various revisions tied to macos versions, complete with theoretically useful Linux makefiles and apple public source license files: https://opensource.apple.com/source/CF/
Compare, for example, these two versions of CFBase.c:
https://opensource.apple.com/source/CF/CF-1153.18/CFBase.c.a...
https://github.com/apple/swift-corelibs-foundation/blob/main...
(Credentials: I worked on the swift Linux port and made an effort to use these very libraries before the swift project itself took on the task of porting CF to swift. While I was working on it we mostly just ifdef'd out everything to do with foundation when targeting Linux)
Bitcoin Core signs it’s own binaries in a deterministic environment. Not sure if they use xar, or what.
This is so fantastic. I wouldn't have believed it was possible had I not seen it here. I maintain multiple old OSX machines solely for this purpose; migrating to Linux for this step makes me feel so much safer for my CI build systems.
The only thing left to solve is to actually build cross platform. You can do with rust cross but you have to build your own docker images on macos first since apple licensing prohibits redistribution of binaries. It would save us so much money in github CI minutes.
This is actually a solved problem, using osxcross[0]. The experience is honestly very smooth, and we don't require any apple proprietary binaries. The only thing apple-proprietary is their SDK (containing the header files for compiling, and tbd files for linking), which can be downloaded from apple's website (at least if you have a developer account), or from various GitHub projects archiving them.
You mean building mac/ios apps on Linux?
That’s relatively easy to do, with the only questionable/legally gray area being access to the iOS/Mac SDKs. I’m not a lawyer, but it’s my understanding that the Google v Oracle case established that APIs aren’t protected by copyright. So if we can use the headers from e.g. the iOS SDK on Linux wtohout worrying about copyright issues (Apple developer agreements may be an issue though), it should be possible to autogenerate stub libraries for linking purposes.
With that, you can compile and link a binary for iOS or Mac on Linux.
Next, you’ll need to implement the packaging and signing portion. Packaging (generating an app bundle) is easy, but codesigning requires Apple’s codesign utility, which doesn’t work outside of Mac, even with a project like Darling (last I checked, anyways).
But if the OP does implement a working codesign alternative, then that means it should be possible to create an iOS/Mac app that can be distributed on the App Store entirely from Linux with no Xcode, virtual machines, cloud macs, etc.
The only gate left then is the Apple tax/signing certificate you need to buy for $99/year. But that might change with all of the antitrust pressure Apple is facing, including the recent EU law that would force Apple to allow competing app stores.
"I’m not a lawyer, but it’s my understanding that the Google v Oracle case established that APIs aren’t protected by copyright."
I'm no lawyer either but I have read the Supreme Court's decision in the case. It doesn't address whether APIs are protected by copyright. The Supreme Court generally makes the most limited ruling needed in order to resolve a case. In this case there were two issues at play.
-Are API's copyrightable? -If so, was Google's use of the APIs fair use under copyright law as a jury had previously ruled?
In order to make the most limited ruling needed to decide the case the Supreme Court first addressed the fair use question with an assumption that APIs could be copyrighted. The Court decided that Google's use did fall under fair use and so the case could be decided by the fair use question alone. They therefore didn't bother to address the question of whether APIs can be copyrighted.
As it stands right now there is a U.S. Court of Appeals ruling in the Oracle case that APIs are copyrightable which until the Supreme Court does address the issue means they are as far as the courts are concerned. Fair use is a case-by-case defense and the Supreme Court's decision doesn't mean that all use of an API is fair use. This is all very much still all up in the air and any use of the headers from the iOS SDK should be concerned about copyright issues.
Zig already does this I think, they committed the Apple headers to their repo.
We achieved this for Firefox macOS builds years ago. (Greg was there for that work as well. ) You do need to have an SDK accessible, but storing one internally and using it in CI was deemed acceptable. The impact on CI build times was hard to overstate.
We were already building our own clang binaries for various reasons, so it was mostly a matter of making an SDK available and ensuring that the right compiler options were passed in (since running clang on macOS sets a bunch of defaults that you don't get even if you pass -target x86_64-apple-darwin).
I came here to whine about Apple Code signing, and how this seems like a terrible thing to reproduce, until I realized this breaks free of Apple, and is hopefully much more sane! Well done!
It's only good when it's not Apple doing it?
You know you can go disable code signing verification on macos with a reboot and a single command right?
The comment you replied to means "reproduce" as in reproducible-builds[.org]. Needing to be on macOS to sign Mach-O binaries makes builds less reproducible than also having a Linux way of signing the builds.
Kudos to these developers. I wish them success.
If I can do a small derail: I and some contributors had this mostly working a few iOS versions ago with isign (https://github.com/isignpy/isign). This is befor notarization. Announced several times to HN but didn’t seem to be interesting to many people.
This was a spin-off from our work at a testing company. I’m not an iOS developer. (That may be why I could never see what to do with this.)
I tried to make it into something, but I could never figure out who actually wanted this. What potential uses are foreseen for rcodedesign?
- Weird custom signing. We did a project for a large financial company to make it compatible with hardware security module signing, but they never implemented it due to the pandemic changing priorities, and then someone discovered there was a little-known MacOS API to do something similar anyway, so our project was shelved. (We got paid though).
- “Alternative” app distribution. From time to time I am contacted by people who run alternative app stores, either distributing hacked versions of paid apps, or who run app store in countries under embargo from Western countries. (They have banks, their customers have iPhones, they want a way to distribute iOS apps). The ethical issues and legal risks seemed significant so I never pursued that. If someone wants to go do that, I guess that’s a use case.
- Build pipeline. Maybe, but you can’t fully build a iOS app on Linux so it didn’t seem like a win.
- Mass app production. I am aware of some app white-labeling concepts that used isign back in the day. There was a guy who had a generic app for local news outlets, and essentially uploaded it to the App Store dozens of times under different names, for new versions. isign was more scriptable (and I wrote a multi-resign option for this use case) but I didn’t see a way to turn this into a business.
- Testing on real devices. Even the original use case (re-signing apps to run on local device labs) has been mostly obviated. Hosting and scripting on MacOS has gotten better.
- ??? something else? I even reached out to saurik about this at a conference. Cydia worked on jailbroken devices, so it didn’t need signing. To my surprise saurik was angry with me. He said that many, many companies had figured this out internally so he was skeptical we would ever release it. (We did!)
So… again kudos, but what can we do with this? I’m not dissing it, I’m genuinely baffled. I also thought this would be important and useful but couldn’t figure it out.
> - Build pipeline. Maybe, but you can’t fully build a iOS app on Linux so it didn’t seem like a win.
This is my use case (I implemented notarization API support in rcodesign).
I build MacOS apps entirely from Linux using my own bespoke toolchain, based on osxcross. It can potentially be used to build iOS apps too (I tested it in jest some time ago, but I don't use it for anything serious). This is all done without any of the apple binaries. The toolchain uses nix to pull everything together, clang and other llvm utilities as a compiler, lld as a linker, rcodesign for signing and notarization, and the macOS/iOS SDK for headers and tbd files used for linking. And a bit of custom glue code in bash to hold it all together.
The reason I do this is simple: My CI is entirely Linux-based. I already cross-build windows binary from Linux. This makes CI so much easier, as everything can be containerized, and thus run on most standard CI offerings. I only need to bring Windows and macOS VMs to run test suites. Because VM bringup is slow and somewhat fragile, pushing everything to run in Linux containers is a huge win.
It also gives me a lot more control when things inevitably break. Being open source, rcodesign can be trivially changed, be it to add more logging, or add new features. For instance, rcodesign has a remote signing mechanism! This isn't something that exists in the normal codesign implementation.
I'm kind of confused by what you mean by "re-sign". The repo you linked to says that it's a tool to "re-sign" apps. Does that mean it requires an already-signed app to function?
Mac/iOS app bundles need to be signed, but so does the actual compiled binary within the bundle. The signing process, among other things, embeds entitlement information into the binary (permissions for using certain APIs). Can isignpy do this? Or does it only do app bundle signing?
From the perspective of app development, a tool that can perform full code signing on Linux is huge. Imagine how much of a pain (and expense) it is to maintain a CI pipeline when you need a physical Mac (since you can't legally run MacOS in a VM). There are more than a few cloud companies that offer dedicated Macs pretty much just for this purpose.
Regarding use case:
It could certainly be useful for macOS CLI tools. Languages like Go and Zig can cross-compile seamlessly from linux (and Rust is nearly but not quite there). With this you could compile, sign and notarize a ready-to-publish binary without ever touching a macOS machine.
I’ve always used gon ( https://github.com/mitchellh/gon ) for this, which is open source golang, but I don’t think it supports mach-o embedding. I’ll have to try this tool out.
Note that gon is a glorified front-end for executing processes like `codesign`, `altool`, and even `ditto` for zip file generation.
This Rust implementation, by contrast, has all the functionality implemented in pure Rust: there is no calling out to external processes for anything. You could drop the statically linked `rcodesign` executable into a Linux container with no other files and it would work.
That's not to discredit gon or its authors: it is a fantastic tool for streamlining common functionality. But the mechanism is completely different.
Never knew it wrapped shell commands - good to know!
Beyond clownish that apple didn’t just release this themselves. Kudos to the creators for doing the job apple should have done.
> There are probably thousands of companies and individuals who have wanted to release Apple software from non-macOS operating systems. (The existence and popularity of tools like fastlane seems to confirm this.)
I'm not sure the existence/popularity of fastlane supports the non-macOS operating systems argument since its primary purpose is as a wrapper around Apple's clunky/undocumented build and upload tools on macOS.
I have a customer that runs a dynamic analysis service. They add a library to the IPA and create a new one. They need to to sign it. It's automated on a Mac but they won't need a Mac anymore if they could run code signing on a Linux VM like the rest of their infrastructure. No more sending files and messages back and forth between the two environments. A huge win.
this feels like a job for lemur and smallstep
Does this mean I can now develop iOS (iphone/ipad) apps, start to finish, without asking Apple's
permission _at any stage_, and then sideload these apps onto my personal iphone/ipad?
(assuming as mentioned elsewhere that I can develop such apps on linux, without needing MacOS)
Will this upload to the app store?
What does notarization mean here?
Realistically? It's an additional static analysis hoop to jump through and another brick in Apple's walled garden in terms of trying to lock-out generic programs written by anyone who can't be arsed to fiddle with cryptography.
How to build GUI apps for iOS and macOS with Rust ?
Can this notarize apps made with other programming languages?
As far I can tell, you can compile the binary and sign your app.
# From a Git checkout
$ cargo run --bin rcodesign -- --help
$ cargo install --bin rcodesign
# Remote install.
$ cargo install --git https://github.com/indygreg/PyOxidizer --branch main --bin rcodesign apple-codesign
rcodesign can sign and notarize applications written in any language. From rcodesign's perspective, Rust is an implementation detail of the tool itself.
You could use Qt and cross-compile.
is there a public format for Code or Keysigning Servers? Like SCEP/EST, but for other stuff?
From what I can remember the signature portion is documented in CMS RFC and the file format is only available from Apple's open source libsecurity.