Git Submodules as a Package Manager
nesbitt.ioFYI A submodule doesn't have to use a gitfile and a corresponding `$GIT_DIR/modules/<name>` and there are good reasons not to. As long as the submodule has a '.git' it can be a symlink, or regular .git directory for a self contained embedded repo. You can still use the same gitlink in the parent repo representing it's commit id and git will still manage it.
If I have a 20GB submodule I'd usually just do a `git clone <url> <path> && git submodule add <path>` and it will be treated the same by git. But now I can just delete it and it's purged. And it is more portable less fragile in some ways because it's not de-referencing a gitfile. I prefer my repos to be more bottom-heavy and to not clog my modules folder.
I have a rough script that's the inverse of `git submodule absorbgitdirs` but it's a bit fragile.
It would be cool if there was some plumbing to expose this a bit more.
I know it sa toy, but if someone gets the bright idea to do this for real: please learn from npm and dont duplicate packages for every clone of a project. Share them between projects like every sane package manager.
> Share them between projects like every sane package manager.
I'd argue the other extreme is less sane. Pip installs everything system wide and you can't have different versions of the same package without venv.
Even early NPM was much much better than the pre-uv Python situation. (TBH even with uv it still feels hacky at times.)
Before uv there was poetry, and before poetry there was pipenv, before pipenv there was virtualenv. I wish people would stop portraying Python tooling completely wrong like that. uv is not the first tool to solve most problems in this space. It may be the best performing though and may be the best overall currently.
I think you’re underselling it. You’ve listed several projects with many thousand man hours in development.
It’s telling that in Python there have been multiple package managers over the years to fix the broken status quo. Typescript/Javascript notwithstanding, in other languages and environments you get the one tool that’s been there for over a decade without competitors.
Very few languages have suffered from the same package manager issues as JS/TS and Python.
Wouldn't yarn, grunt and gulp fall into this category?
> Typescript/Javascript notwithstanding
Also, gulp / grunt weren't package managers.
Very few languages have also suffered their popularity
> the other extreme
These aren't the only two options.
This
> Pip installs everything system wide
does not inherently mean
> you can't have different versions of the same package
Virtual venv existed before uv. So you could still just generate a virtual venv and then use pip to install packages in the local venv. (2007: virtualenv was released as a third-party tool).
The global repository should support versions obviously.
If you can point a package manager at a git repo and use it like package, this is accidental convenience. Source code repositories should be factored as source code, with a build step to transform them into packages. For some languages that means compilation, for others it means transpiling, minifying, or just copying files. The resulting artifact is a different shape, designed for consumption.
If you use git for packages, then your repo becomes the package boundary. You no longer have the option of producing multiple packages from one repo, or even one package from multiple repos.
> If you use git for packages, then your repo becomes the package boundary. You no longer have the option of producing multiple packages from one repo, or even one package from multiple repos.
What does this even mean?
I'd argue that having the source to build the thing is more important than the artifacts. Release artifacts are more of a convenience. If the thing doesn't build from the source given, what use is it?
Exactly my point. You assume the source is the thing you want to consume, but it isn't. Package registries are much more than a convenience and packages don't necessarily map 1:1 with source code repositories. Nor should they need to. You shouldn't have to build every dependency and care where their source code is.
Sure you want the source available, otherwise it's closed source, but ideally you never need to look at it unless you're a contributor.
git has really conflated the two concepts.
Correct way is to have everything as-if a source dependency, but a globally shared cache system that can automatically provide artifacts if that exact version of source has already been compiled.
Any system that can only consume manually built (and even in worst case, only tagged and globally pushed) dependencies is horrible for cross-module development, like large scale refactorings and the like.
A lot of the confusion and criticism surrounding submodules comes from people not understanding that you do not need to learn a lot of new commands to use them: you just cd into the submodule and use it as a normal git repo. Then you cd out and commit the submodule hash change if that's what you want to do.
Some things are annoying, like removing them. But in general they're extremely useful and designed basically exactly how you'd want: it's extremely consistent since to a good approximation, a submodule behaves the same as a normal git repo once you're inside it.
It's the only way to do certain things but the devx is burdensome. It might be easy enough to do a thing but the major issue is that you have to constantly fight wonky behavior when working in a team.
If someone adds a new submodule, you need to know to pull it or at least init it. You need to know the code is in a submodule and treat it differently, complicating integration procedures.
It certainly does not behave the same in that you certainly can't ignore whether it exists once it's set up.
I disagree, even when you understand them there are still big problems with them:
1. Git doesn't support them very well. Bugs are disappointing frequent. Try changing a submodule to a directory or vice versa and feel the pain.
2. They are often used to split up projects (especially commercially) but this makes testing much harder and cross-repo changes way way harder. Even changes that only touch the submodule become way more tedious because you have to always do another PR to update the pointer.
3. The submodules are referenced by URL, which means you can't easily move or copy projects that have them because they still point to the old submodule location. Sometimes they're even misconfiguration to always use ssh:// or whatever, breaking unauthenticated clones.
4. Switching branches where submodules have been added/removed is a mess.
5. Having to constantly remember `git submodule update --init --recursive` is a right pain.
6. When used as a crap dependency manager you easily end up with duplicates. A project I worked on had 12 copied of a common submodule via transitive dependencies.
IMO they're almost always the wrong solution.
Submodules aren't perfect and can be really annoying. I tried to avoid them at first.
But if you work with larger projects I think you'll see how they are useful.
For example, some projects might commit their node_modules or composer vendor folder as submodules so that they have an exact snapshot of their dependencies that can be quickly cloned in place and referenced without mixing it with the parent repos source.
Submodules are certainly not the only way to do things with git. Sometimes monorepos or something else makes more sense.
But if you care about maintainability then what are you supposed to do when the majority of your commits are unrelated to your project because you're using git subtree or something similar? Or the majority of your files are also not directly relevant to your project but need to be there?
What about stuff like optional tests or maintenance scripts or even binary dependencies that you want to have closely associated with your project but don't want to clone every time?
My thinking is that the porcelain is incomplete but very open. The way submodules are represented can be added on to.
And the plumbing is consistent enough that you easily script around it.
You should check out the Datalad project to see some really clever usage of submodules.
And the url key can be annoying; I usually try to use relative paths in my .gitmodules folder when possible. You're actually free to use remote helpers and relative paths to get around hard a github url into it.
You can also embed and read arbitrary keys from your .gitmodules file with git-config -f which is really useful. You don't even have to use a .gitmodules file, in theory.
Well indeed. I think the ultimate point is that Git is severely lacking here. There are clear use cases (you listed some) that drive people to submodules or LFS, but they are both shitty solutions.
We need Git (or something else) to provide a third solution that actually works. I'm really hoping that JJ realises that this is a problem that needs to be solved but I'm not holding my breath.
> Git doesn't support them very well. Bugs are disappointing frequent
I can't remember any "bug" in the submodule machinery. If you find one, report to the Git mailing list, they very receptive.
> Having to constantly remember `git submodule update --init --recursive`
Set this config: `git config submodule.recurse true`
> IMO they're almost always the wrong solution.
Yeah... but sometimes they are the only feasible solution, especially when the dependency needs to be shared across projects that use incompatible dependency managers and Git is the only common denominator between them...
> Set this config: `git config submodule.recurse true`
Well that does save a flag at least so thanks! But I still have to run the command.
There is actually a flag to do submodule update automatically when you checkout or switch branches but DO NOT USE IT! It is known to be buggy and can break your .git directory in a way that is pretty much "delete everything and clone from scratch".
I just have a feeling the developer-experience wouldn't be the same - but that's just a temporary issue perhaps?
This appears to be ai-generated based on my impression, pangram's "100% of this text is ai", and the blog's posting history.
So a fully automatic Gatling Footgun.
I find https://github.com/andrewmcwattersandco/git-fetch-file more useful. It's manifest is nearly the same as a .gitmodules file, but having specific control over what files you want to pull, or what commit or tag, or just the equivalent of latest is way more practical day-to-day.
How does this compare to `git subtree`, which is part of git itself ?
This isn't really like subtree it's like submodules, but for specific files. This could really be useful, you may want to keep track of a specific file in a repo but not need it as a submodule.
There's a whole umbrella of similar-ish utilities. The one that I think does almost the same thing is: https://github.com/mullvad/git-carbon
Yes, basically this. I think git-fetch-file has been used in some larger projects based on Googling it, so it seems to be different in that it's been vetted by other orgs, but like Mullvad, theirs is built by a corporation.
It seems like something more in-between. Like a git-based npm install, but generic to any repository or language.
It's also "pointer" based, but like vendoring, it's up to you whether or not you want to keep files in the repository. It also seems to be intentionally built for the use case where you modify something from upstream, because if you do have local changes, it detects them and won't overwrite them without your permission.