Npm operational incident, 6 Jan 2018
blog.npmjs.org> Unfortunately, the process was complicated by well-meaning members of the npm community who believed that a malicious actor or security breach was to blame and independently attempted to publish their own replacements for these packages. Ensuring the integrity of the affected packages required additional steps and time.
That is such a bad response to this.
The problem isn't that "well-meaning members of the community" decided to upload packages. The problem is that when their system decides that a package shouldn't be up it completely removes the package, as if it never existed, and allows the namespace to be reused immediately. Those "well-meaning members" should not even be able to hijack packages this way, as it means the people who aren't "well-meaning" can also do it.
What should happen is that they block downloads of the package while they investigate. That way people who attempt to download the packages get a meaningful error and people are unable to hijack the package name.
> The problem isn't that "well-meaning members of the community" decided to upload packages. The problem is that when their system decides that a package shouldn't be up it completely removes the package, as if it never existed, and allows the namespace to be reused immediately. Those "well-meaning members" should not even be able to hijack packages this way, as it means the people who aren't "well-meaning" can also do it.
Hasn't this been an ongoing issue with npm since pretty much its inception? I remember reading articles about this vulnerability and the hijacking of packages that were taken down temporarily years ago.
How has this not been dealt with systematically yet?
Yeah, about two years ago NPM stole a package name from an existing user and gave it to a company to use. The user then said that if he can't trust NPM to actually treat package naming fairly then he was just going to delete all of his packages[1]. This broke a ton of packages on people (in part due to "left-pad" disappearing), so the community went ahead and registered/uploaded all of the packages to NPM again.
Afterwords NPM came out with a blog post[2] where they went out of their way to take as little blame as possible and basically said it was the developers fault. They said they "stand by [their] package name dispute resolution policy, and the decision to which it led"- basically ensuring that no developer should ever trust their repository in the long term, as they'll happily hand over any package name to a corporate entity if that entity asks for it.
The weird thing is that they claimed to make it impossible to "unpublish" packages, so that developers could no longer rage quit their site, but apparently they didn't extend that new requirement to their own "security" systems.
[1] http://azer.bike/journal/i-ve-just-liberated-my-modules/ [2] http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm
This is exactly the incident I was thinking of, thank you.
It is bizarre that npm hasn't resolved these kinds of situations by now. Makes me glad I've continued to stay away from that ecosystem.
I don't think the dispute resolution process makes them untrustworthy. It's an annoyance. It's the unpublishing that's a huge security flaw vector, and the fact that they didn't actually fix what they claimed is pretty bad.
To me it's a trust issue. Their dispute policy breaks trust in two ways-
1. As a developer I can not know with certainty that a package I publish will remain published under its current name.
2. As a consumer of packages I can not trust that a library I am using won't get changed to a different piece of code due to someone else thinking they deserve the name better.
What you say is also a problem. The fact that they claimed to have solved the unpublishing problem when they apparently hadn't is pretty huge, as is the fact that the flaw exists to begin with. Unfortunately NPM is just not a trustworthy company.
I agree as well. The guy published his kik package well before Kik was even a company. There was no reason for the npm team to resolve the situation they way they did. If someone already owns your Twitter ID or Facebook page name, well they're closed private services. Same if someone buy's your domain name (see the case of Nissan).
You don't have a right to a name on every service on the planet just because you trademark it somehow.
> 1. As a developer I can not know with certainty that a package I publish will remain published under its current name.
You can as long as your package name isn't trademarked or likely to confuse users installing the package.
> 2. As a consumer of packages I can not trust that a library I am using won't get changed to a different piece of code due to someone else thinking they deserve the name better.
I'm actually fairly sure npm won't blindly hand over a package that is depended upon, to another entity. When they handed over 'kik' it wasn't in the same league as 'left-pad' which was widely depended upon.
> What you say is also a problem. The fact that they claimed to have solved the unpublishing problem when they apparently hadn't is pretty huge
I agree it sucks, but the fact is they 'prevented unpublishing' to bug-fix one vector for this problem, but then introduced a bug in process that appears very similar to unpublishing. If you've never had this sort of thing happen to you as a software dev, (had some stakeholder question 'but I thought you'd fixed X') you're very very lucky.
> as is the fact that the flaw exists to begin with.
Easy to criticise in hindsight. At the time of left-pad, several other package registries (e.g. PyPI) also allowed unpublishing.
>> 1. As a developer I can not know with certainty that a package I publish will remain published under its current name.
> You can as long as your package name isn't trademarked or likely to confuse users installing the package.
Trademarked where exactly? You know, there's quite a lot of world beside US.
> I'm actually fairly sure npm won't blindly hand over a package that is depended upon, to another entity.
What makes you trust them in this matter? They haven't displayed such behaviour, and their behaviour up until now slightly suggests the opposite.
>>> 1. As a developer I can not know with certainty that a package I publish will remain published under its current name.
>> You can as long as your package name isn't trademarked or likely to confuse users installing the package.
> Trademarked where exactly? You know, there's quite a lot of world beside US.
And, if I recall correctly, trademarked when? Wasn't leftpad.js's author using the name kik well before the company Kik existed? So you don't just need a name that's not trademarked _now_, you need to pick one that no-one else trademarks sometime in the future (in whatever jurisdictions the npm people care about)...
> and their behaviour up until now slightly suggests the opposite.
Please elaborate. Afaik 'kik' wasn't significantly depended upon, and people using the old kik could still install it [1] (had the leftpad author not unpublished it), and that is the only example I'm aware of of npm handing over a package name.
[1] http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm
I cannot believe anyone will defend npm over it.
There is no scenario where it is OK to hand over a namespace to someone else. At worst, it is acceptable to make a namespace unavailable to anyone.
I think npm is completely unable to exist as an organization and should disband immediately.
Trouble is, if they do that, someone else will quickly replace them, take over the same function, and users will likely not notice that they are now trusting a different unverifiable entity to ensure package integrity
Users will continue to trust what comes packages with node. Nothing needs to change from a user's perspective. They will still type npm ... But it will point to a new infrastructure (and more importantly led by nobody currently leading npm).
If node wants to do this, I think they totally can. There just needs to be the will to cut off and blacklist the current npm team.
How many times does it have to happen to warrant concern?
Trust, once broken, isn't quickly restored.
Hypothetically, even if it's just the author of kik (I don't know if it is), isn't that still unfair to them? They might have used it on client projects. Why would I want to use your registry if you're going to break all of my software because a corporation wants a my package name?
Pretty sure "allover" isn't "significantly depended upon" - how would you feel if YC gave that account here to me just because I went and paid for a trademark on it?
I have no investment in this username, so I wouldn't care.
And "significantly depended upon" doesn't make sense in your example, but is relevant in terms of a package in a registry.
Trademarks only apply within a common industry. Kik the company isn't in the business of creating NPM packages. It's perfectly legal to use the same name for something unrelated to messaging even after the founding date of the commercial entity.
I'm not arguing what's legal. If your package name is a trademark, thinking you're entitled to hang on to it is naive. This is well understand with domains, so why do you think a package registry should be different?
Just had a quick look - the "kik" package isn't in even use anymore - the whole thing was just drama theatre from by a messaging app I've never heard of. Despite it having 200 million users. Apparently.
> a messaging app I've never heard of. Despite it having 200 million users. Apparently.
Yeah it's tough for me to keep track of which apps are/aren't significant because I'm not in the loop with every community out there.
I'm guessing the users are distributed among small pockets in a network of people who all know each other -- like how in the 1990s there were pockets of "MSN Messenger" and pockets of "Yahoo Instant Messenger" and pockets of "AOL Instant Messenger" users randomly scattered around various schools. OR like how Orkut grew so precipitously in Brazil.
Or it's a generational thing -- I think there is a certain age group of people in the US who used Yik Yak, another who uses tbh and co, and so on.
I would generally trust Facebook employees to tell me which apps are "real" / growing since they have Onavo surveillance and other sources (https://www.wsj.com/articles/facebooks-onavo-gives-social-me...). But aside from those data, I'm not sure how else to keep track of which apps are "real" / big.
Kik is mostly popular outside the west, it is pretty popular in South East Asia.
In the US Kik has a pretty active userbase; but it's demographics skew older, poorer and more female and more rural. So that might be why you haven't heard of it.
Are you sure? This wasn't my impression and what I'm seeing from a quick Google is pretty different on multiple counts:
"Quite surprisingly, close to 70% of Kik’s users are men. It is most popular among young men between ages 20-24. Vine and Flickr are popular among men too, though their users tend to be a little older: 25-29 and 35-39, respectively. Not so surprisingly, Instagram and Pinterest are more popular among women than men. Women also prefer Facebook Messenger over WhatsApp and Kik"
http://www.vertoanalytics.com/the-demographics-of-social-med...
Kik is pretty popular among camgirls. Usually paying for a subscription to them gets you added to their contents on Kik so you can chat with them.
I wouldn't associate that with older, rural people. More female, sure... but then you've also got their male followers.
Older?! Kik’s demographic is far younger than anything.
I hope this leads more developers to begin using restrictive trademark terms for their work, like Mozilla does with their software (e.g. you cannot fork Firefox and still call it "Firefox") or what Tuomo Valkonen did with ion3.
And for that matter, this sounds like a pretty strong disincentive for developers to make their code open-source at all. I'd rather just release proprietary software under a license that allows me to revoke individual entities' ability to distribute my work on a whim to prevent something like this from happening. And I mentioned Tuomo above for a reason--his experiences with ion3 and dealing with the open-source community drove him to a similar decision, and he quit writing open-source software and developing for Linux as a result. By the time he quit, he had become an active opponent of free software.
IIRC, Tuomo was also dissatisfied with the state of Linux on the desktop in general.
A huge chunk of that was that he was fed up with stable distributions shipping older versions of his software, so he'd get inundated by bug reports from users who were reporting things he'd already fixed. He argued that distros like Debian Stable and RHEL were actually more prone to bugs because of this. He wanted distros to use either rolling release or the Microsoft/Apple model, where the OS vendor only ships the base OS and all other software is acquired from third parties.
Because of this, he ended up putting in a termination clause in his license:
> 3. Redistributions of this software accessible plainly with a name
> of this software ("ion", "ion3", etc.), must provide the latest
> release with a reasonable delay from its release (normally 28 days).
> Older releases may be distributed, if the full version, or some
> other explicit indicator, such as the word "ancient", is part of
> the name that the package is accessed with, or if this identifier
> is completely unrelated to a name of this software.
+1
I think it's shocking how NPM tries to subtly shift fault to "well-meaning members of the npm community" for costing NPM "additional steps and time". Shame on npm. This shouldn't have even been possible after the azer/kik issue a few years back.
I'm not a javascript dev, but isn't NPM volunteer run? It seems odd to be shaming people who do something for you as a volunteer free service.
It's a company: https://www.npmjs.com/about
It's been literally years since node-forward got its talk about signing packages [1] with a lot of pushback from the npm team. Every time a new typosquatting article shows up, there's some more waffling by npm. left-pad happened to much consternation. Now this.
I used to really care about trying to harden the Node ecosystem, and last year it was one of my main goals. I tried to send multiple vulnerability reports, do mass static analysis of npm packages, and wanted to contribute more to the ecosystem, but the consistent ambivalent reactions of much of the community that I talked to turned me off of the project entirely. If npm wants to continue to be a security dumpster fire, let it burn. Node is a waste of security researchers' time and an honest goldmine for black hats looking to compromise relatively powerful novice webdev hardware.
I don't see it changing anytime soon. npm is a business that isn't focused on security. These things keep coming up, and yet npm install metrics I'm sure aren't decreasing. Until they face meaningful competition and/or the rest of the Node community begins to give even half a care to security outside of this forum, there will be no incentive for anyone to do anything about it. It's easier to play PR, give a little lip service to it and dodge the problem than it is to add any friction to their potential growth.
This is your occasional reminder that package signing is not a panacea, and as typically proposed for community package repositories like npm, PyPI, etc. would likely do absolutely nothing.
For example, people often insist in the Python world that PyPI should support package signing. But it already does -- you can generate a signature for a package and upload the signature with the package. Django does this, and has been doing it for years. You can also get package download/install tools that will check the signature. But then what?
What people really mean when they say there should be "signed packages" is that there should be a whole bunch of invisible infrastructure (set up by... who, exactly? Maintained by... who, exactly?) to decide which PGP keys are authorized to sign releases of which packages. And that's close to an intractable problem for an anyone-can-contribute community repository like npm or PyPI.
This is a very important point. I work for a company that publishes client libs for many different package indexes (although not npm). This is a fairly well automated process, but it takes minutes (if that) to push a new version to pypi, rubygems etc, but at least a few hours of fiddling about to get something on maven, which of course has this security infrastructure.
An analogy might be drawn with the app stores. We all know it is massively easier to get stuff in the play store than the iOS store. We all know there is a shit ton of spam, malware etc on the play store and not really in the other. But it's also much easier to contribute to. It's a trade off. Security is important, but sometimes I feel that people are unwilling to treat it as an input in a basic cost benefit analysis, instead turning it into a kind of absolute value. I accept that it is not treated seriously enough by many in the community, but overcorrection is not the answer.
Of course, other relatively 'open' package indexes exist that do not have npm's typo squatting issues, so there are other design issues at work in this particular case.
Do you suppose FB+Yarn is in a position to compete? Yarn can implement support for optional package signing. From the consumer's perspective, one can choose to be alerted whenever the "main" package signer (usu. developer) changes, or simply to accept only packages verified and signed by a group of trusted third parties.
Nit: leftpad was not the issue. Kik was the issue. Npm incredibly mishandled the situation. I agree with the rest of your analysis.
I think the community should fork npm repository. Anything of value is free and open source anyways. Why does node continue to support npm people?
"Those who cannot remember the past are condemned to repeat it" -- George Santayana
Now, it is easy enough to lob an aphorism but an event like this means that lessons previously learnt have been ignored (for whatever reason).
If you are going to build something that replicates or advances on previous systems the least you can do is study those systems to see what they did right and what they did wrong. In my opinion the Javascript ecosystem has produced some wonderful new systems that do things that have never been done before but in doing so they have wilfully ignored the past and didn't learn some basic lessons. In doing so they muddy the waters and in the end bring down the reputation of the whole ecosystem. I guess then it is not a huge surprise that a sizeable number of people feel that an unusually large number, as a percentage, of Javascript projects come and go and keep reinventing the wheel.
Very displeased about this response.
> In yesterday’s case, we got it wrong, which prevented a publisher’s legitimate code from being distributed to developers whose projects depend on it. We identified the error within five minutes and followed defined processes to reverse this block. Unfortunately, the process was complicated by well-meaning members of the npm community who believed that a malicious actor or security breach was to blame and independently attempted to publish their own replacements for these packages.
No. Assuming everything in that excerpt is true (and I happen to know it's not, but that's not even relevant here), that wasn't the problem.
The problem is that NPM allowed packages to be re-uploaded by new authors after the initial versions had been spam filtered. Especially since allowing packages to be re-uploaded by new authors was the core issue of the left-pad debacle, and the one thing NPM said they'd fixed in response.
Let's summarise here:
1. NPM has a big issue
2. They claimed they had fixed it
3. They had not
4. In their post mortem they're pretending the issue doesn't exist
This guts any remaining trust I had in npm. Even if I wanted to trust them, they're not even admitting the problem exists; how am I meant to believe they're finally going to fix it? They've stopped even promising to fix this, and moved on to lies and denial.
Unacceptable. Literally. This is pushing me away from the node ecosystem because I am not prepared to accept this sort of weaponized incompetence from the primary package repo for node.
With all that money they've raised, it's puzzling why they don't take PR seriously. I guess they have a mentality of "where the fuck else are you gonna go, the node community is tightly coupled to our service"
It's almost as if the tech community has to learn once again centralized systems with a trusted authority at the top can't be trusted when that trusted authority makes mistakes.
It's kind of unbelievable that NPM is a for-profit company that serves the node community for free. You can't possibly expect them to make the right choices when the community isn't whom they are directly responsible to. NPM should be a nonprofit foundation, a coop, or some other organization that has its operations funded by the community and has elected positions.
Their lack of financial or other interests in the community directly explains their policies.
Honest question: what's the alternative?
I've mentioned this elsewhere in this discussion, but: https://github.com/ramasilveyra/gitpkg
I've been using it a bit, and it's really nice. Yarn (and the npm command line tool) already provide nice methods to install packages from arbitrary locations, including git repos; this provides a nice mechanism for publishing to git repos. It's a super simple solution, but then, it's not really a hard problem.
Publish you package on an http reachable git server and use that as location for your package.
Distributed systems. Mirrors. Tools that are agnostic about sources for packages.
There's no reason that we must have a single, centralised package repository for a language and the dominant tool chain for that language relying on it by default.
nix (pkg manager). Can give you an insane amount of control of how your project builds, especially if you use nixos, but with great power comes great responsibility XD.
>This guts any remaining trust I had in npm. Even if I wanted to trust them, they're not even admitting the problem exists; how am I meant to believe they're finally going to fix it? They've stopped even promising to fix this, and moved on to lies and denial.
The cold reality I'm realizing is: what are you going to do about it? If you have tens of thousands of lines of javascript written using dependencies from NPM, what is the next step?
Very disheartening to see that NPM has not grown from the kik or left-pad incidents. Users should not be able to republish on top of old package names without some kind of intervention.
Of course, NPM's response to the kik/left-pad problem was also pretty awful. Make it so users can't delete packages. Cool. For those of you using NPM's private offerings, this also applies to you, so hope you don't care about cleaning up your private NPM registry content.
Getting pretty tired of this. Their open source operation seems to suffer from poor handling of community and technical issues, at least from a high level. Their private registry operation is very lean on features, and also suffers from very confusing limitations. I'm surprised at how long it took there to be read-only API keys. Until last year, you literally had to give your CI keys that could publish to your organization if you wanted access to private packages. And you had to pay for an extra user for the privilege of doing it wrong. It's fixed now but it still blows my mind that it took so long. Aside from faster installation, it's actually a lot better to just use private Git URLs instead of NPM's private offerings.
NPM will never properly take the fall. They will take just enough blame to seem responsible and then shift the majority of the poor decision making onto other actors in the community in their explanations. Third time in recent memory.
> no malicious actors were involved in yesterday’s incident, and the security of npm users’ accounts and the integrity of these 106 packages were never jeopardized
Maybe not in the incident itself, but the sheer fact that many of the packages were replaced by other people constitutes a jeopardization of applications that depend on NPM. The only reason why some big package didn't get replaced with code that exfiltrated data from production or subtly backdoored it is sheer luck.
it's beyond disingenuous and flat out dishonest for npm to say the integrity of the packages was never jeopardized. within minutes there were reports that packages had been replaced with questionable and malicious content.
https://news.ycombinator.com/item?id=16087079
this is probably a good indication of how a much more serious security event would be treated by the organization.. just sayin
At the very least, Npm should explain why we can be sure the packages now offered are exactly the same as they were before the incident, if, in fact, it is in a position to make that claim.
Yeah, I have no idea how they can even pretend to claim that.
How can they claim no malicious actors were involved when packages such as duplexer3 were apparently replaced with undesirable code as reported in https://news.ycombinator.com/item?id=16087126 ?
This is not a claim or a fact, that's probably an uneducated guess, npm is broken on so many levels they can't say anything for sure. [1]
npm is full of fundamental bugs in their software, workflows, architecture, and community. We learnt this in March 2016 when one developers got a copyright claim and had to remove their package. [2] Why is it even possible to remove a package? Why is it possible to create new package with the same name? Almost no software published there have unit tests, asking developers how they tested it in a GitHub issue leaves the issue open with no response. Everything is centralized on GitHub, which was proven to fail many times when GitHub was down in a region or globally due to a DDoS. Why is Node allowed to exist? We're not in 1998 anymore.
[1] https://news.ycombinator.com/item?id=16092584 (comment above)
[2] https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/
We learnt this in March 2016 when one developers got a copyright claim and had to remove their package. [2] Why is it even possible to remove a package?
The earlier problem you mentioned didn't seem to be about copyright. But if it had been, the answer to why it's even possible to remove a package is: because if you don't and you knowingly continue to host copyright-infringing content you can expect a very large penalty in court.
At least get the company name right if you have to make guesses about what they know internally and what they don't. (lol, and edited it silently so my comment looks dumb now)
>We don’t discuss all of our security processes and technologies in specific detail for what should be obvious reasons, but here is a high-level overview
> they know internally and what they don't.
They know internally that they do a lot of things badly and this will result in more disasters for npm and JS communities, that's for sure. Check out how GitLab fixed their backup failure. They made new software for testing backups, they live-streamed their work on youtube, they made at least 2 write ups on this case. They were totally transparent about their mistakes and fixes, everyone applauded them for this. What did we get from npm?
Are you saying that displaying a Bible quote, modified to refer to the fact that duplexer3 is gone, is "malicious"?
I believe this was one of the cases of a community member stepping up to publish a package quickly so that malicious code wasn't published.
Presumably they make the claim that none of the code was malicious because they checked.
from the blog post:
> the integrity of these 106 packages were never jeopardized.
are we operating with different definitions of jeopardy here? 106 packages were absolutely at risk of harm during this window. The fact that some community members stepped up is irrelevant, a bad actor could have done a lot of damage here. I think this blog post is completely disingenuous, and doesn't make me trust npm.
I was replying to 0x0's claim, not making a broad statement about npm's policies.
0x0 claimed the code uploaded as duplexer3 was "undesirable code" and implied that it was a "malicious actor". It was completely utterly benign.
The last paragraph is good! The other paragraphs are bad. The two main ways they are bad are:
1) A system that detects "spam" and then allows for the complete removal of packages as if they never existed, allowing anyone to replace them, should never be described in the neutral terms used in this post. This system appears to be an existential threat to the company and project, and one of its largest mistakes. It won't take a long investigation to figure that out; it should be obvious today.
2) The claim that the security and integrity of these hijacked npm package names was not jeopardized appears to be 100% pants-on-fire false. If it is not false, I think npm's users are owed an explanation today of why it's false, rather than a bare assertion.
I know it's hard to be in the hot seat. No animosity to any of the humans involved.
It is really another embarrasment for NPM, considering that this is not the first time something similar happens (see: left-pad issue, 23 march 2016)
So they learnt zero from it.
Then, WHEN ARE THEY GOING TO SIGN THOSE PACKAGES? Zero, nothing. They didn't learn from mistakes, they don't listen. NPM is still open to all sorts of malicious use.
Besides NPM problems, yesterday many packages won't work because package "pinkie-promise" wasn't available. This is the full effective source code of pinkie-promise:
module.exports = typeof Promise === 'function' ? Promise : require('pinkie');
Someone should grab the 100 most used packages which contain less than 30 lines of code, merge it into one package and call it 'npm standard library'.
I have no idea why there are negative comments to this suggestion. I do a lot of Java development and Apache Commons as well as Google Guava are two of the most popular libraries.
The problem would be maintaining and versioning this standard library.
Guess I'll be that guy: https://xkcd.com/927/
I’ve gotten to the point where I recognize certain XKCD urls and no longer have to click on them to know which one it is. The ol’ 927, I see.
So the kiddies haven't learned, huh? The whole left-pad debacle was because of a package that has 1 single function: it pads your string to a certain length by prefixing characters. (Writing this I wonder who made right-pad). And it broke almost everything.
> who believed that a malicious actor or security breach was to blame
This was a security breach. Their anti-spam system should block repos by freezing the module name and returning blank files, not by deleting the entire module and subsequently allowing anyone to upload new modules. This is leftpad all over again.
Update: Woah, so I was checking out my old NPM namespaces and apparently someone took control over https://www.npmjs.com/package/filesaver.js
It's not a "breach" technically. No malicious person caused the packages to be removed by circumventing authentication/authorization systems. It is a security issue though. A bug and serious security issue, yes, but not technically a breach.
A breach didn't cause the issue, but there absolutely was a breach in response to NPM's incompetence. https://news.ycombinator.com/item?id=16087126
I think it's time to replace npm in default Node installs. They've shown a history of negligence for package handling and procedure. Why should Node continue putting the trust of such a substantial part of not only their ecosystem but the JS ecosystem as well into such unreliable hands?
the official node.js Docker image comes with yarn as well as npm. but this is an issue with the npm registry, not the package manager shipped with node. both yarn and npm use the npm registry.
EDIT: it may be worth debating why npm (a binary produced by npm, Inc) is shipped with node.js at all, but that's upto the node foundation
I’m pretty sure npm is shipped as Javascript source by node.js
ah yes, you are correct. it isn't a binary
From what I've read (I've not yet worked on anything that used npm), it appears that npm uses a single shared flat namespace for all packages. If I publish a package named "order_pizza" nobody else can name a package "order_pizza", unless mine gets deleted first, and then they can...and anyone who was using my "order_pizza" will now silently get the other developer's "order_pizza" the next time they build.
Anyone know why they did it that way instead of giving each developer a subspace of the namespace that only that developer can publish under?
I'm thinking of the way it works on Java, where my package would be net.tzs.order_pizza, or GitHub (and numerous others) where it would be tzs/order_pizza.
I think it's annoying but also toxic to an ecosystem. You have to name your packages the same way you look for a domain name, people can squat on names, and it causes drama like how the 'kik' package was stolen from a developer and given to a company who never used it.
Nobody would've cared it the package was "tzs/kik". For example, the Elm package system does this right.
Another example of this failure is the Reddit subreddit system where a subreddit like /r/bitcoin makes it seem like the authority when it's a carefully moderated and biased subreddit. And nobody can really compete with it without getting a worse subreddit name like truebitcoin and bitcoin2 or something. Would be much more clear if it was /r/@theymos/bitcoin and /r/@spez/politics.
The failure is yours alone if you actually consider Reddit an authority on anything.
Downvoters: I of course meant an authority on anything ELSE than memes and karmawhoring. Sorry for not being clear about that.
Okay.
And you might say the failure is yours alone when you overeat, yet 30%+ of the USA is overweight. This little platitude only makes you feel better about yourself but isn't a good basis for designing something.
That's interesting. Unless I'm being forcefed, then who else is to blame? I have free will after all.
> Anyone know why they did it that way instead of
Root causes wise, my impression is that software language engineering has a knowledge integration and communication problem. Lambda the Ultimate doesn't have a wiki. :) Academic papers provide a foundation, but a lot of practical professional knowledge is scattered, siloed in language communities, and time-consuming to access. Gathering best practices for language design and implementation isn't something we do well. So language creation projects often look like team and/or community "learn by doing" education exercises. Even with good people at leading companies... I had a conversation at redacted once, "why did you make (disappointing) design decision X?", "mumble mumble", "<facepalm> that's a (widespread) misconception, a misinterpretation of paper P (which admittedly could have been clearer), that's been kicking around for several decades now".
My fuzzy impression is the npm repo was not originally a mission focus, but rather a "don't be python" add on goal. Absent organized best practices, low-priority design decisions are often not made well. Perhaps Ruby, Perl5, and/or Python were copied, without attending to those communities experiences, and the implications for an even more fine-grained code sharing community.
This person on reddit apparently suggested it back in 2013: https://www.reddit.com/r/webdev/comments/7oljbe/xpost_rjavas...
Relevant excerpt:
> I proposed required namespaces (@user/package) as a requirement in 2013 and the npm developers acted like I had 2 heads.
In theory, deleted packages can't be taken over, their name remains reserved unless the company assigns it manually via some administrative process. That is, unless there are bugs which can actually delete packages, which seems to be the case here.
Namespaced packages/repos have their own problems (figuring out which is the "good" one after some time and a few forks happen)
>Namespaced packages/repos have their own problems (figuring out which is the "good" one after some time and a few forks happen)
Most if not all of these packages are hosted somewhere like Github. It's not hard to tell which repo has the most stars, fewest issues or most recent commit, that work's already done for you.
But, we're talking about Node packages here - many of which seem to be trivial, or single line functions. How many forks of left-pad could one expect to find?
> How many forks of left-pad could one expect to find?
Javascript github repos: 70 of left-pad and 29 of leftpad. Some are jokes, but still...
If you try the C++ package manager conan.io you will likely run into similar troubles. See for example https://github.com/lasote/conan-zlib/issues/23
Fair enough... that's a lot more than I would have expected. But I still think on balance there are more benefits to using namespaces than drawbacks. Without namespaces, the same problems of quality control exist, but the decisions are being hidden from you or made on your behalf by a central authority, or you're just stuck with whomever got to the name first.
Organizations on npm can publish "namespaced" modules, that look like `@orgName/moduleName`. It's just optional.
That is a new facility.
This npm blogpost about them was published almost three years ago: http://blog.npmjs.org/post/116936804365/solving-npms-hard-pr...
> and anyone who was using my "order_pizza" will now silently get the other developer's "order_pizza" the next time they build.
Nope.
What if the project that requires order_pizza has no restriction on what version it pulls in?
- It still wouldn't be silent or automatic - npm-based projects have a package lock
"We don’t discuss all of our security processes and technologies in specific detail for what should be obvious reasons..." - Security by obfuscation? That's not security. Your protocols and processes should stand up even if made public.
It's standard operating procedure to elide the details of one's security posture.
Would you expect Google's security team to provide a detailed list of all of the security procedures in place to prevent access to their network?
Rejecting "security through obscurity" just means you shouldn't rely on obscurity alone. Obscurity + good security > good security alone, since it increases the threshold of time and ability that any attacker would need to bring to an attack.
I saw a bit of a talk by a google engineer that laid out their overall strategy for securing the android app ecosystem. I don't think it's a hard and fast rule.
Yeah, you make a good point. I'm not outright rejecting keeping your secrets close, but it doesn't instill a lot of confidence for me.
After left-pad I think we'd all like to see a strong, well documented methodology to keep things like this from happening again. The broad strokes can't be any more clever than what the rest of us would expect, so why not at least provide a basic idea of what you're doing and then we could trust the system a bit more.
To me it sounds like they are talking about their automated systems to scan for spam and malware. While I normally would support your stance I see why they would act in this way, such systems often only work as long as they are not made public.
Sure, this is "security by obscurity" in the purest sense, but its pretty common throughout the industry to not publicly disclose one's security process. Those details make it easier to circumvent.
What is your home address and phone number?
> What is your home address and phone number?
Neither of those are security systems. That's like asking for their password and claiming you've proven security through obscurity.
The idea behind dismissing obscurity is that if everything but private tokens were exposed, your system should still remain secure.
If you think that your address and phone number are secret information, consider how easy it would be to find yours from the information in your HN profile. I'm pretty sure I found your entry on whitepages within a few minutes and I'm barely motivated to find it.
I think you are confused as to my intention
I thought you were asking a rhetorical question to show that obfuscation is a valid security measure that we all use. What was your intention?
The correct question should be: "what firewall and IDS do you use?"
I left an Ethereum related project precisely because of the house of cards feel of it being built off JS and npm modules. They're a great example of how a project can be exploited by a malicious module which could proceed to extract all the tokens.
If my hat is black, I'm writing a daemon which monitors relevant npm modules and uploads subtlely modified versions if and when the possibility to do so occurs again. Particularly, but not only, targeting crypto currencies.
> If my hat is black, I'm writing a daemon which monitors relevant npm modules and uploads subtlely modified versions ...
I started sketching out one yesterday, and then thought "What am I going to do with this? Probably just get in trouble" so I stopped.
If I was managing npm-using systems, this press release would not put my fears to rest. They need to publish a full review of the contents of packages replacements that were uploaded (even if just for 5 minutes), and publish those replacements in a safe form so they can be reviewed personally by any concerned npm user (any user - not just those who downloaded the replacements through npm itself).
Lesson : in a name space, never delete records. Mark them as unused, invalid, inactive, "deleted" but never actually delete.
Independent of npm's issues now and in the past, how vulnerable are other package managers to similar problems? There were typosquatting issues with pip, replacement issues with Rubygems, probably others that I don't recall. What's the current state of things for the more commonly used languages?
Maven/Gradle don't allow un-namespaced packages; nobody knows for sure how deep up their arse npm's designers have had their ass when they made non-namespaced packages possible (even the only way in the beginning; I don't know if it's still the default).
Additionally, there's the problem of npm's fine-grained dependencies (that have become a laughing stock in itself), in my Java large projects, while I don't know every single dependency, I know all the "vendors", i.e. trusted upstream namespaces I use, and their transitive dependencies. There's the occasional non-Eclipse, non-Google, non-Apache dependency (say, XStream), but I am aware of it and fix it to a specific version. npm with its thousands of dependencies by hundreds of pseudonymous developers for simple projects is an abomination.
`Crates.io` (the Rust package ecosystem) doesn't have namespaces and operates after the first come first serve principle. They won't resolve namesquatting issues so something like the left-pad debacle won't happen. I think their policy[0] is quite sane. At least I'm not aware of any issues of that kind regarding crates.io.
For typosquatting: I don't think that problem can be easily solved by package registries. They could warn you if there is a package with a similar name (and more votes, downloads, what ever) but aside from that, the developer should double check the dependency names and that's it.
Npm isn't ready for production use. If you're going to use it you either need to ship the modules you need or run it against a private repo of modules.
I agree in principal about not relying on npm during deployment. For a novice, what is the best way to ship modules?
My first thought was “What about including node_modules in your version control?” Downside is that the repo would be much larger, but if it’s going to expand to that size on the client after running “npm install”, maybe that’s not a big deal? But it feels kinda wrong to do that...
Committing your `node_modules` is a bad idea. The best answer is to either use an offline cache of package tarballs that have been committed to your repo, or an NPM caching proxy.
Yarn has an "offline mirror" feature built in, and Shrinkpack is a tool that can create an offline mirror based on an npm-shrinkwrap.json file.
I wrote about using Yarn's offline mirror here: http://blog.isquaredsoftware.com/2017/07/practical-redux-par...
I know you can use JFrog/Artifactory to cache pulls of Jar files from maven repos, as well as Docker images. Does JFrog support node?
Yes, JFrog Artifactory supports npm and so does Sonatype Nexus. More info:
https://www.jfrog.com/confluence/display/RTF/Npm+Registry https://help.sonatype.com/display/NXRM3/Node+Packaged+Module...
I haven't actually used an NPM caching proxy myself. It looks like Artifactory does support acting as an NPM proxy. The other tools I've heard of are Sinopia, Verdaccio, and local-npm.
Addy Osmani has a related article at https://addyosmani.com/blog/using-npm-offline/ .
yarn's "offline mirror" blog post[1] has been super helpful — I was able to set one up in less than 30 minutes.
That would break when going cross platform for modules with native deps.
See npm rebuild https://docs.npmjs.com/cli/rebuild
As others have said countless times on the original thread: new packages from different users should never have been allowed to replace the missing packages.
Even if a package is removed as malware, a user should never unexpectedly download the work of a totally new author that they haven't vetted.
Some of the protections should be done on the user side, but that's no reason for NPM to have dangerous policies.
> but that's no reason for NPM to have dangerous policies.
Oversight != 'dangerous policies'. Seems like they thought they had this fixed.
They need to pull their socks up and start signing these packages
Been discussed. Repeatedly. For years.
Can't understand why it hasn't happened still.
Can't understand why it hasn't happened still.
https://github.com/npm/npm/pull/4016 - I read this as "it´s hard, and we don´t care." took them forever to look at and reject a patch that would have gotten the ball rolling
What would signed packages have achieved in this incident?
Who needs to sign the packages? The developers of the packages, right? Otherwise developer credential compromise subverts the entire point.
Which keys are authorized to sign for which packages? How to prevent credential compromise from affecting those authorizations?
What's the difference between a signed legitimate package and a signed malicious package?
If they introduced package signing, would packages would adopt it fast enough for users of packages to only use signed packages?
What happens when signing keys are lost or compromised? Do we need to use countersignatures from timestamping services as with other forms of code signing, so that CI systems do not break if a key is pulled?
I think this is very much not straightforward. npm+pgp may be well intentioned but seems grossly inadequate.
Android and the Google Play store works pretty well. The systems "locks on to" a public key signature so further updates for a given app must be signed with the same key. That would fix the problem with reproducible builds here. Only adding new top level packages to a project for the first time is at risk then.
No one is asking for perfect, just an ssh style warning of "the key of the system you are connecting to changed from last time" would be a good start. Implementing that is trivial, straightforward, requires zero setup and would have easily prevented the copycats here.
A bunch of folks here are complaining that the explanation isn't good enough, doesn't go into enough detail, etc.
Literally the second sentence is:
> Early this coming week, we will share a full analysis and technical explanation of the incident.
I'm looking forward to reading the final post-mortem, but in the mean time I really appreciate their update.
The only thing missing from this post is 'we take security very seriously'
Am I missing something in thinking that a reasonable solution is to temporarily block downloads of potentially malicious packages? Why remove them?
EDIT: or better yet, don’t allow people to download the update, just keep users on the old, ostensibly safe version until everything gets sorted?
I'm curious of the manual review process. Is this synchronous, i.e. immediately when publishing a package? Or is it after the fact, where suspicious code code have already been distributed? There are plenty UX trade-offs in either direction, of course.
I've used NPM only a little but this scares me. What are NPM alternatives?
This question is important: what are the alternatives to npm? Surely someone has something out there with code signing and single-use package names for node/JS?
Verdaccio [https://github.com/verdaccio/verdaccio] looks like a viable open source alternative. Presumably you could use it to download the packages you needed, then disable proxying to the NPM registry.
There aren't any. Yarn is a better package manager than the npm client, but the problem is in the npm registry, and that's the same for both yarn and npm..
NPM shouldn't scare you. Simple fact is you should not be relying on ANY package registry at the time of deployment.
> NPM shouldn't scare you.
It absolutely should, just like any dependency on any other third party code or servers. Especially when they regularly have incidents like this.
> Simple fact is you should not be relying on ANY package registry at the time of deployment.
This is true, but doesn't prove the previous claim.
> NPM shouldn't scare you.
>> It absolutely should, just like any dependency on any other third party code or servers
There's no need to be scared of any of those things if you understand the trade-offs and risks.
>> Especially when they regularly have incidents like this.
They don't.
>> Simple fact is you should not be relying on ANY package registry at the time of deployment.
> This is true, but doesn't prove the previous claim.
The 'previous claim' follows from it. If you don't treat a package registry like an essential part of your own infrastructure, there's no need to be scared of it.
These days we deploy whole virtual machines from image reopsitories.
Lots of people are piling onto npm here. This reads to me like a fairly simple unintended consequence of what seemed like a good approach.
Automated spam filters help to avoid dodgy packages. Spam filtering operates on heuristics so it’s sensible to not publicise how they work.
The automated spam filter kills dodgy uploads, as these mostly happen on previously unused names a decision is made to not have the spam packages’ name remain taken. Among other things this stops the spammers from leaving all the good names blocked.
The spam code gets a false positive and the above logic kicks in, leaving previously used names now available. This is noticed and corrected within a few hours.
The npm team will likely improve their spam filtering heuristics and also ensure that formerly good packages that get spam flagged do not release their names - they have indicated roughly this on Twitter.
This only just happened, and it’s the weekend, so I’d expect a full write up will be released during the week when they’ve had time to do a post-mortem and work through the salient points.
> This reads to me like a fairly simple unintended consequence of what seemed like a good approach.
Yes, but you need to keep in mind:
1. This stuff is hideously important because getting it wrong is a incredibly serious security hole. Letting people publish a package with a name that other people are already requiring is a defcon 1 emergency.
2. They already had a major issue with this and said they had fixed it. You get, at most, one time when you can say "whups, we didn't realise how big an issue this was!".
3. It's their job to get it right. This is, literally, npm inc. It's not some hobbyists, or a service being provided by people in their spare time.
So when you say:
> The automated spam filter kills dodgy uploads, as these mostly happen on previously unused names a decision is made to not have the spam packages’ name remain taken.
I have to disagree. They needed to check how many times the flagged package had been downloaded, and after the left-pad debacle, they knew (or should have known) that.
They're not taking this seriously.
maybe everyone should wait for the full report before jumping to conclusions?
from what it appears, the reason someone else was allowed to register the same package name was because their database didn't pull in the entries for that package. an error? yes. an issue? yes. but it is not a repeat of the same incidence of left-pad.
i'm not going to pass judgement until i read the full report. i think at the very least, whatever mechanism they were using to copy/backup the database seems to have an issue with deleting entries and needs to be addressed
Others may be interested to note this blog post from about a week ago by the CTO relating to npm's outage handling [0]. I'm not sure if I would classify this as an outage, but this seems to be the official narrative [1]. Given that status page was updated relatively quickly, I'm guessing that at least something like this procedure kicked in.
I am a little shocked at the brevity of the procedure. I am guessing by the blog post pointing out the brevity that I am meant to be shocked, but if I turned out that 310 word procedure for downtime I'd be pretty confident it was not good enough. I'm not confident that I would be able to read that document and understand what to do if I was the one on pager duty. Maybe it would be different if I worked there and perhaps the procedures are more granular and each of the stages well explained in other procedures. Maybe.
I'm finding it hard to understand what npm does that is so different to other languages' package managers that it can have so many problems and such bad public relations. Linux distros have repositories that handle packages with far greater sizes and quite a few are purely voluntary efforts. They also manage to include package signing as well. I've known of a few hiccups, but none that been handled quite so bad as to completely undermine my trust in the tool altogether.
Is there any compelling technical reason why we couldn't have a tool that works more like go get for node? For those that haven't used it, you just point it at a git repository.
[0] http://ceejbot.tumblr.com/post/169045932039/how-to-handle-an...
I'm trying to figure out whether node2nix [0] could help avoid package name/version hijacking vulnerabilities. Node2nix can convert a package-lock.json file to something called a Nix expression [1], like this [2], where each dependency has a checksum. The purpose is to make the Nix expression deterministic so that each Nix package is reproducible.
[0] https://github.com/svanderburg/node2nix
[1] https://nixos.org/nix/about.html
[2] https://github.com/svanderburg/node2nix/blob/22d0c19575ecd19...
Edit: fix link formatting
Why does a spam system delete packages that have already been in use for quite some time? I could understand if it blocked some newly updated ones, but it seems like it has deleted already existing packages that also were used by other packages?
With this and leftpad etc., I think it's time for the tech community to remember Java and Oracle, Sourceforge, etc., or look it up if you are too young to remember. It is beneficial to learn from history instead of making the same mistakes all over again.
Centralized trusted authorities cannot be trusted if they are private companies who don't answer to you. If you aren't paying them and you don't have a management position in said company, things like this will happen.
Everybody is mad/disappointed at/by the NPM for their handling...
Does this have any real impact on the NPM package adoption though? There is no alternative node module package directory, is there?
It seems like any action they could have taken would have had exactly the same effect (well, maybe some prospective products don't adopt JS backends, but you have all webpack projects locked in pretty tight, right?)
I've been using gitpkg (https://github.com/ramasilveyra/gitpkg) lately; it works pretty well. Basically lets you use a git repo as a package repo, instead of npm.
Very little about how I use npm requires it to be centralized; I don't ever search npm looking for a package to install; I find a package I want and then look to install it.
Couldn't you just use github IDs in your package.json directly? npm and yarn both have support for it.
Short answer: Sure, but it works much better with gitpkg.
Long answer: Sometimes you can do that, yes. But you'll run into issues with "modern" frontned toolchains and complicated packages where you want to write ES2017 code, transpile it with babel, bundle it with webpack, make a universal lib, etc., and push the result to your package repo, but not commit it to your git repo.
Gitpkg handles all the wierd corner cases, and lets you keep build artifacts out of your git repo, and raw source out of your package repos. And if you use github you can even use the github IDs in your package.json, but the result is going to be a lot better. If you're just writing a plain one file package for use in node that doesn't need any processing, eh, no need for gitpkg.
Such explanation looks misleading to me. Even having the module detected as potentially malicious, why was it put into a state that such module doesn't exist, letting others publish package with the same name. For me it looks like there is something wrong at the npm side internally with the processes.
How difficult is it to run your own private npm repo? Looks like that's what security-conscious folks should be doing, given this response. Any pointers/gotchas/battle stories much appreciated.
It's doable, and one very good option is Sonatype Nexus https://www.sonatype.com/download-oss-sonatype
The one issue I've seen over the years is it seems npm Inc does not officially sanction on site npm repos (presumably to help monetize their private hosted repos), and finding info on how to set up a private repo is lacking.
Just when I started thinking NPM had turned the ship around after the previous incidents. Given the amount of people who depend on their service, this is extremely amateur.
The pleasures of working with Node.js
Is it related to a package that write logs in colors and send visa card numbers to hackers ?
No, that article was a warning that a package like that could exist in the wild and nobody would notice. It doesn't actually exists, as the author states in the blog post.
Since everything uses HTTP these days I think they could use one of the 20x codes to signal when a package has been affected by spam management or suspension.
That way the official clients would support it but you could easily ignore it if you wanted to.
Has the community discussed adding cryptographic signatures to npm packages? It's generally a good idea in any case, but this outage further highlights the security risks of a public registry where packages are not signed.
Looks like they don't allow comments on their blog posts??... I guess with good reason.
They use Tumblr as their blog platform, and Tumblr AFAIK doesn't use comments.
> We don’t discuss all of our security processes and technologies in specific detail for what should be obvious reasons, but here is a high-level overview.
Isn't that the opposite of good practice? You shouldn't rely on obscurity. It's better to have the security processes out in the open so that it can be audited and flaws pointed out.
Yeah, I found this an important aspect of the article. I don't know anything about the culture at NPM, but this feels related to the mismanagement of communication with consumers/the public about the incident itself. i.e., being secretive for no good reason.
Discussing anti-spam/abuse systems in specific detail would be fairly unusual, can you name any larger operation that does?
There's a lot of questions open that are non-specific to that though, so I'd agree that they could be sharing more.
I think it's
> We don’t discuss all of our security processes and technologies [with each other internally, so dumb stuff keeps happening]
I think a major security improvement for npm and other package managers using almost exclusively open-source/ non-compiled code is to require the source code be linked from a popular open-source platform like Github and then to take the package directly from there, to ensure the code can be audited and nothing else can be snuck in.
I've been meaning to set up a blog with hexojs for a few weeks now, and was seriously all set to install node and npm (don't do too much with node...), and start tinkering yesterday, and then I checked hacker news, and was like 'well sheeyit'...
I guess this would be safe enough probably:
Pay attention to the tags they themselves added to their report. Namely, "#SPAM IS WHY WE CAN'T HAVE NICE THINGS". Oh npm...
noob question : would using yarn have mitigated any of the risks of missing/broken packages ?
No, both Yarn and npm use the npm registry.
idea: Peer reviewing packages with the ability to suggest a better name.
A manual review misjudged 106 packages at the same time? That seems... dubious. Unless they have a bulk removal based on meta data, instead of careful analysis. Which would also be pretty dubious.