Show HN: ied – an alternative package manager for Node
github.comHey everyone!
I made this. I'm happy to answer any questions, but please bear in mind that this is a WIP. There is still a lot of work to be done, although feature parity with npm is not the goal.
Upcoming features are:
* Nix-like rollbacks * built in registry server * discovery + installation via BitTorrent DHT
Would love to get any feedback!
Is your plan to remain compatible with `npm` projects?
I could see a case where I'd use `ied` to install deps for the speed, but continued to use npm for other features and interop with others.
This could be very useful - but would need shrinkwrap support along with the stuff on your TODO to make it viable for my current work projects :)
Although using ied and npm at the same time for installing dependencies should work, you might run into some weird corner cases.
Therefore, npm and ied are compatible in the sense that you can use ied to install packages and npm for publishing them. At least that's the level of compatibility that is currently supported.
shrinkwrap is definitely coming :)
Good to hear, I reckon I could quite happily use ied for all the installing stuff if it's going to produce the same tree as npm, but perhaps not support all of the crazy corner cases.
For the discovery and installation, you might be interested in looking into using IPFS for it. Kind of gives you all the features you want out-of-the-box. Also, another guy is working on mirroring the npm registry in IPFS, might be interesting for you: https://github.com/diasdavid/registry-mirror
Hi Alexander, this is really great!
I've started `registry-mirror` to demonstrate how a Content-Addressed file structure and P2P discovery, can bring a lot of speed improvements, specially when the bottleneck is low bandwidth/latency to the backbone, by connecting to more local peers that have the content that we are looking for.
Right now, the goal with `registry-mirror` is to have a very large IPFS node in the network with the entire npm and that keeps replicating it, while end user machines only download the modules they need (and if they agree, provide them to the network as well). Each end user will be able to get the latest state of the registry, through a IPNS hash, a mutable pointer, that changes each time the mirror is updated.
The nix package manager model layers perfectly on top of IPFS' MerkleDAG (https://github.com/ipfs/specs/tree/master/merkledag), it can be a very awesome transport for ied. An example of a package manager that uses IPFS to distribute the packages is GX https://github.com/whyrusleeping/gx - Still a WIP.
If this is interesting to you, join us at IRC Freenode #IPFS, it would be great to bounce more ideas! :)
This looks very interesting!
Also related: https://github.com/dominictarr/npmd
if you can make this work in a way that it's capable of reinstalling a precise snapshot (like using a Gemfile.lock in Ruby's Bundler world) and keep it stable then I once again will have faith in JS package management.
You know npm shrinkwrap does exactly this, right?
Does it support installing packages globally and then having them actually be available, without having to install them locally as well?
This is my biggest bugbear with npm, as it pretty much rules out doing shell scripting with node.
If you keep node in ~/bin/, you could symlink your global npm path to ~/bin/node_modules/ .. just a thought here.
I mean if you keep your shell scripts in ~/bin/ (which is where I happen to keep mine)... I use straight bash scripts most of the time, but it's usually easiest for me to use node/js since my environment already has it than it is for me to try to remember perl, learn python or use something else.
Making using a shrinkwrap not awful would be amazing.
Having fast immutable shrinkwrap installs, and the ability to upgrade or selectively upgrade packages in the shrinkwrap vs. the semver in package.json would be a huge win.
Great to hear that! shrinkwraps + rollbacks are definitely coming!
npm install --save
In npm v3 updates the shrinkwrap as well as the package.json
That seems so cool. One question is how compatible is with npm? Can I use it as a drop-in replacement ( talking about the CLI tool )?
Mostly, yes, but there are some features that are still missing:
`ied publish` and `ied version` is coming next week. I'm also thinking of adding scoped modules, but I'm not sure about that yet.
You can also configure a private npm registry to be used: https://github.com/alexanderGugel/ied/blob/master/lib/config...
Cloned a popular project (babel) and on a MacBook Pro:
npm install 52.28s user 8.08s system 73% cpu 1:22.41 total
ied install 10.22s user 4.36s system 142% cpu 10.230 total
Awesome! Glad it worked that well! Please let me know if you run into any bugs: https://github.com/alexanderGugel/ied
Happy to open an issue, but any plans on supporting git+ssh urls? For work projects, we have some modules installed via git and when I tried to install, I got the same error described here: https://github.com/alexanderGugel/ied/issues/2
Very promising project!
+1 here, git+ssh and git+http(s) are essential imho... its' the easiest way to sidestep the need for an npm server for some internal projects/libraries.
Agreed. It's definitely on the roadmap. Should be done in a couple of weeks.
You say you are creating a more performant NPM. Is it impossible to fix the current NPM? Why not submit fixes back to NPM? Why create a new project?
The irony of this question should be apparent if you search for packages on npm today. How many "new projects" are there?
In answer, why not create a new project? NPM INC controls npm, hasn't contributed it to the node foundation (despite playing a pivotal role in creating said foundation), and hasn't been especially good at taking contributions recently.
A new project dodges all those existing problems, demonstrates alternate approaches are both feasible and compatible, and destroys the myth that npm is fundamental to node, rather than simply the first of many package management systems that take advantage of node's import semantics.
I wonder what reasons there are for not putting npm in a foundation in the same way that happened to Node itself? Surely such a critical piece of Node infrastructure shouldn't be controlled by a single (for-profit) company.
Works is ongoing, but npm is pretty complicated at this point
Its install process is very different from the one npm uses. E.g. just look into the node_modules directory produced by npm vs the one produced by ied.
Also the way it wires up dependencies (using symlinks) has been called "non humane design" by npm... so I thought it would be easier to just start a new project... it's not a lot of code/logic needed there actually... :)
npm v3 already uses the 'flat by default' design, so the node_modules folders aren't that different.
They are extremely different.
I made a quick comparison here: https://gist.github.com/alexanderGugel/a10ed5655d366875a280
Basically ied uses symlinks in order to resolve circular dependencies, while ied exploits the fact that require "falls back" in the directory structure.
Understood. Though I believe your comment (which compared ied with itself) should actually read:
> Basically ied uses symlinks in order to resolve circular dependencies, while npm exploits the fact that require "falls back" in the directory structure.
Yup. Thanks.
NPM has decided to risk the life of the entire Node ecosystem by taking on funding. Additionally, they aren't particularly good at npm--only after the much-awaited v3 has it approached a decent piece of software.
I never thought to myself, "wow, npm is slow." Is that really the core problem this is solving?
NPM 2.x is actually very slow.
Part of that is just the monstrous number of files involved. For example, one of our projects has 48 dependencies, which installs 24,421 files under node_modules. NPM could probably benefit from managing each dependency as an archive.
We do atomic deploys and try to make them reproducible, so for each deploy we do a fresh install from npm-shrinkwrap.json, but even when all the modules are in NPM's local cache it's very slow at copying everything.
Unfortunately, NPM doesn't version the node_modules folder (a package becomes ./node_modules/mypackage/...)), so you can't reuse it. The NPM cache is versioned ($cachedir/mypackage/3.23/...), but can't be used directly. It would be much better to skip the cache altogether, and have node_modules embed version strings (./node_modules/mypackage-3.23/...). Then you could easily share the folder across builds.
NPM is also pretty brittle. We frequently have deploys fail because of transient network errors (repository timing out or similar) that cause NPM to fall over. The dreaded mysterious "npm ERR cb() never called" error still hits us weekly.
(Speaking of reproducible builds: NPM lets people unpublish packages. Sometimes old versions just disappear, presumably because they were unpublished. 6 months later you want to deploy a certain app, and you find it depends on some package X, which deep in its dependency graph relies on package Y 0.3, but 0.3 is gone from npmjs.com, so you have to upgrade for no reason at all.)
Well, you're definitely in for a treat with NPM 3. And I don't mean that in a good way.
How so?
Have you used any other package manager? I'm not completely sure if it's just Node's dependency hell, but npm feels so incredibly slow compared to Cargo or bundler or something.
Never profiled it but I would guess that is largely due to the npm's main problem: It doesn't locked dependency versions. As a result it probably builds the dependency graph a lot more than it should.
For me this is the main thing I'd like to see fixed. Not only does it build more than it need to, it can result in broken builds. When my dependency doesn't update, I don't expect its dependencies to update (especially when I don't have control over setting the versions of the dependencies of dependencies). This has bitten me more than once.
It really baffles me that most dependency management systems get this wrong. You'd think it would be consideration #1.
I work on a fairly large frontend project where everything is modularized. So we have maybe 50+ dependencies that are our own. These constantly change. I detest changing branches and having to run npm to get the correct version of stuff. Takes minutes before it even starts installing packages.
Wow. I think to myself "npm is slow" several times every working day, no exaggeration. I am currently on HN because npm has been busy for about 5 minutes. Maybe we use npm very differently.
> The easiest way to install ied is using npm
Bootstrapping ftw.
The initial set of dependencies is being installed via npm, the it installs its own dependencies via ied if told so: https://github.com/alexanderGugel/ied#installation
This is a "cool" feature during development, since it's a nice proof of concept.
Originally I checked in the node_modules directory, but then reddit was shitting on me as usual (yes, you shouldn't check in node_modules in an actual app, but this is PACKAGE MANAGER!). As far as I know, npm has also its own dependencies check in + a ton of packages as tarballs for tests, so I might do that later.
Dude, don't listen to reddit when it comes to managing your own open source project.
I was just thinking about that yesterday. I write my own projects so that I don't have to dance the political dance and make my colleagues happy. My own projects are for exploring my own ideas. Advice is always appreciated, but if someone wants to dictate how I write code in my own projects, they better damn well pay me (a lot).
To the OP: Don't let people bully you. Many people have strong ideas and will want you to do things their way. You aren't going to make everyone happy, though. Somebody will be pissed off no matter what you do (if you are popular enough -- normally people won't pay any attention to you ;-) ). "Because that's what I want to do" is a completely valid reason for any decision on your own project.
reddit gets mentioned a lot in these parts. Is there a subreddit i don't know off?
> produces a flat node_modules directory
Finally. npm's node_modules makes Node on Windows unbearable from time to time.
Shut up and take my money!
npm v3 already does this by default.
Unfortunate naming ie https://en.wikipedia.org/wiki/Improvised_explosive_device
Damn. Didn't think of that.
In fact I literally just renamed it: https://github.com/alexanderGugel/ied/commit/84628b3c871c85d...
Originally it was called mpm, but I figured that would have been pretty confusing, but it looks like the new name isn't necessarily better.
I'm pretty terrible at naming. Any suggestions are more than welcome!
I think it's fine as it is. Could have chosen a worse name for sure! And, managing dependencies is like dealing with explosives anyways...
> And, managing dependencies is like dealing with explosives anyways...
Only funny for anyone who hasn't been affected by one, or had friends or family who were. Still better to avoid names with negative connotations (and search for them first to check).
> Any suggestions are more than welcome!
A few ideas:
bpm - Better Package Manager
edge - the thing that connects nodes
jpm - Javascript Package Manager
ppm - Peer Package Manager
fpm - Functional Package Manager
ayp - All Your Packages
nnm - New Node Manager
As I said earlier, it wasn't my intention to name it after a weapon.
That being said, it's a three letter name. It's very unlikely NOT to run into naming conflicts here.
edge - taken by Microsoft
jpm - JPMorgan
ppm - taken by Perl package manager: https://en.wikipedia.org/wiki/Perl_package_manager
fpm - taken by Effing package management: https://github.com/jordansissel/fpm
bpm - beats per minute
ayp - terrible to type, although taken by "Adequate Yearly Progress": https://en.wikipedia.org/wiki/Adequate_Yearly_Progress
nnm - What happens when it's no longer new?
Just in Germany for example there are a ton of companies called ISIS (just google "ISIS GmbH"). Being offended by a three letter shell command seems a bit over the top to me to be honest.
Edit: I won't respond to further comments on the naming issue. It wasn't my intention to name it after a weapon. As I said earlier, I will change the name as soon as anyone proposes a better one.
Yeah, but when people do a google search for IED, what are they going to find?
I'd worry less about offending someone because they had to type it, and more about SEO. I'd stick with things that don't have pictures of gore and destruction on the first page of search results.
So, that's a really good reason to not.
Also, your response was shitty... You specifically state that you chose IED because it's "easy to type"... but when someone says "Hey, that's what we call bombs that insurgents use to kill people with", you reply "Yeah, but the alternative is an acronym used by JP Morgan... People getting offended by me naming something after a way to kill people are being over the top". If "IED" happened to actually mean something, fine, make a case... but it isn't actually easier to type than anything else... If you want that, name it ASD, which shouldn't have any conflicts, as the top google search result is Anchorage School District... and I couldn't find any conflicting package names.
I get it, people are constantly picking on things and suggesting that they need to be more PC... but in this case, there is absolutely NO reason for you to stick with the name IED... and several reasons to change it (SEO, offensiveness, typability).
Also, good news for german companies... The news has started using different acronyms for ISIS... I've seen IS and ISIL in regards to the paris bombings, as they're more true to the literal translation.
Another Stupid Deployment (ASD)... just suggesting a corresponding name to work with.
> That being said, it's a three letter name. It's very unlikely NOT to run into naming conflicts here.
It isn't critical to avoid all possible naming conflicts, only to avoid 1) other command-line tools and 2) names in poor taste. Thus, only the conflicts with other package managers really matter.
I wouldn't worry about naming conflicts per se, just unfortunate or misleading ones (e.g. company names).
I think "bpm" is a clever name for a faster npm client because "beats per minute" is a speed measurement and thus associated with something going really fast.
IED on the other hand has the association of people dying or being crippled by terrorists.
Only funny for anyone who hasn't been affected by one, or had friends or family who were.
Oh, lay off the holier-than-thou moral outrage. I bet you could mine for naming collisions in a space of three characters like this all day, or really all sorts of project names.
With two letters it gets even better. I was offered ss.com but passed, later I thought I should have taken it and turned it into a site documenting the atrocities committed by the SS and to strongly attack neo-nazis by showing them for what they are. Missed opportunity.
SS can just as easily be ss(8), Social Security, Secret Service (actually called USSS to deliberately avoid that), etc.
man 8 ss tells me it was written by Alexey Kuznetosv. I should strongly consider informing him of his trivialization of the NSDAP's atrocities.
holier-than-thou moral outrage? He or she is just expressing an opinion that happens to differ from yours.
I don't think the name is supposed to be funny. If you're afraid of naming things something just because someone, somewhere in the world have been affected by what you're naming your thing, you'll run out of names.
So yeah, throw your hands up and don't even try. It's not even worth trying. No, don't even put in a sliver of effort.
Its more the case that the set of people that could take offense to something is effectively infinite.
Say the programmer has english as a second language, its really easy not to know "common" things like this. Taking outrage in these cases is not in very good faith as the presumption is that the name is the same thing as the acronym in question.
Lets say they chose to name it baa, and people reacted to it thusly:
Sometimes people CANNOT be pleased or appeased or even bargained with when they respond to things from the pathos mindset. Make an effort sure but recently people seem to be going out of their way to be offended. As a fan of debate and arguing I hate that the presumption in these debates presumes bad intent. That style of thinking eventually leads to a very closed mindset.- what is this implying all users are sheep! needs to be changed/boycotted - ... - this makes no sense, name it after its function xyz. - this is a horrible name, I'm going to encourage everyone I know not to use this
I always thought the name of git/github was mildly offensive to me... but never really enough to speak out about. Though I think that ASD mentioned above (I added "Another Stupid Deployment") isn't a bad name.
> edge - the thing that connects nodes
Or "leaf" for singly-connected nodes. Or "graf" for that matter.
pkn - package node?
Super easy to type, and available on npm. https://www.npmjs.com/search?q=pkn
yanm
yapm
This post will attract those of conservative/libertarian mentality who like to stand for our right to use language that may offend people, but I just want to add another vote for changing the name.
I am not particularly sensitive to these things, but upon reading the name, an image came to mind of a soldier leaning against a concrete wall on a dusty urban street, leg recently blown off, bleeding out, in shock. That image is now permanently associated with your project for me :)
For comparison, imagine if you'd called it "iud". At least in American society, this is more readily agreed to be offensive because of our sex taboo and gender inequality.
For a lot of people, the thought of war evokes deep visceral feelings of horror and fear. Such people will be less likely to use your project, which you could argue is a form of discrimination, and in any case doesn't benefit anyone.
I had a project that got really popular and went through like 5 name changes. It was a horrible experience. My advice to anyone who is ever involved in naming a software project:
spend no more than 2 minutes picking a name, and stick with it
This is why developers should try to avoid introducing dependencies. Not at all costs, of course, but typical project in JavaScript, Go, Ruby, or Python has way, way too many dependencies, while little of them have any significant benefit.
What's the downside? I think it's great. More modularity + distributed responsibility = better.
Delegating things to a library you give up control over any bugs the library has. You can't simply fix the bug like you would do to your code, you either need to fork the library (maintaining a fork has cost), patch it (maintaining the patch is not free), or send fixes to its maintainer -- or any combination of these.
Using many dependencies also makes your application a fractal of dependencies. Library you use has its own dependencies, which have their own dependencies, which have... And so on. If you happen to want to put your application to DEB/RPM packages properly, host your app's dependencies locally to isolate from network outages, or do virtually anything non-trivial with your app, you hit the list of dependencies very hard.
It's not to say dependencies are evil; like any generalization, this would be simply stupid. But they have their cost, and it's large. You'd better make sure a library really simplifies your life before adding it to your project.
And writing custom code to duplicate a library doesn't have a cost? (compared to forking).
I find that using tools like npm, nuget, gems, and the like along with github for core libraries makes managing dependencies FAR better than relying on an internal copy of said package in my project, which is just as risky as an internal fork. Yes, there's a chance the package system can be down while you are wanting to deploy.. that's why you prepare on a staging server, and duplicate that to production. Hell with things like docker, you create your container, and run it with environment variables for which environment it's running against.
> And writing custom code to duplicate a library doesn't have a cost? (compared to forking).
Of course it does, but again: it's you who controls the code, not the library's upstream. You can easily patch and adjust whatever necessary, without being bothered with future merges from upstream.
This aspect grows in importance as the library gives less and less benefits. For example, I wouldn't hesitate to use Flask or Django for a web application, but I would think very hard what good is going to give me logging library other than Python's built-in.
> I find that using tools like npm, nuget, gems, and the like along with github for core libraries makes managing dependencies FAR better than relying on an internal copy of said package in my project
npm and gems give you dynamics very different from seeing dependencies by yourself. You see dependencies as very cheap (which is not true, it's just the cost is hidden from you), so you don't bother with thinking if you really need the new dependency. If it can shorten your code by twenty lines, you see it worth.
My suggestion: NIP — a recursive acronym meaning "NIP Installs Packages".
How about MOL - one letter back for each letter, as HAL was to IBM?
Leave it as it is. It will be much easier to memorize.
Fully disagree. Really unfortunate/distasteful name - distracts from what I'm sure is a great project (see: conversation in this thread).
It's just a three-letter acronym. There aren't that many of those, it's inevitable that sometimes there are collisions.
How about "package"?
'$ package install ...'
Did you run your name by /anyone/ whatsoever before posting this? I can't imagine that if you'd say, asked three people, one of them wouldn't have laughed at the very suggestion. IED indeed. This is the sort of silly mistake one can only make if one is acting in isolation: to me it's super important in life to be checking in with people constantly when making decisions like this. Typically a couple of people will do it: after that, for me in my circle anyway, the returns diminish. What a waste of people's time and energy on this forum, just because you didn't bother checking with people (I'm assuming). It's kinda selfish of you, in that sense. Well, good luck renaming it, and good luck figuring out where you went wrong.
Also, 'ie', the only until recently mortal enemy of javascript.
Guys NPM is not the problem, Javascript is.