Matrix.org Security Incident
matrix.orgThe attacker seems to have responded:
https://github.com/matrix-org/matrix.org/issues/357 edit: just saw the rest: https://github.com/matrix-org/matrix.org/issues?utf8=%E2%9C%...
"[SECURITY] SSH Agent Forwarding
I noticed in your blog post that you were talking about doing a postmortem and steps you need to take. As someone who is intimately familiar with your entire infrastructure, I thought I could help you out.
Complete compromise could have been avoided if developers were prohibited from using ForwardAgent yes or not using -A in their SSH commands. The flaws with agent forwarding are well documented."
Did the blog get hacked (again?) in between this being posted and now? It has what looks like password hashes and `uname -a` from every(?) server in their infrastructure.
This is about as bad as IR can get: you realize you got hacked, you re-build your entire infrastructure and publicly say it's fixed, and then you get popped again...
i think so. here is a wayback machine link to the previous blog post https://web.archive.org/web/20190412000400/https://matrix.or...
it stated "Having fully flushed out the attacker [...]" which i guess turned out to be false :-/
also im getting invalid HTTPS certs on the blog now. for some reason im getting a cert that looks like its for github.com ?
edit: now im getting a lets encrypt cert on matrix.org, but a cloudflare SSL error page when i go to www.matrix.org ? the lets encrypt cert looks like it was just issued about an hour ago.
edit2: i guess both with and without www. are lets encrypt, but the with www. cert was issued back in february (and gives a cloudflare SSL error page), while without www. was issued today. (and gives the current hacked message)
Here is the message of the hacker, after the matrix.org admins allegedly cleaned everything up and wrote the original blog post: https://web.archive.org/web/20190412055614/https://matrix.or...
The most favorable reading of the current defacement page is that the attackers still controls the DNS, but no other parts of the infrastructure.
Otherwise, the page probably wouldn't run off github.
Assuming the GitHub issues are from the actual attacker -- and I see no reason to doubt they are -- this is very troubling:
https://github.com/matrix-org/matrix.org/issues/363
Compromise began well over a month ago
Yikes. That's a long time for a compromise to go unnoticed.
Not really, the average in the industry seems to be floating between 70 and 400 days depending on the source of your stats on the topic (different vendors and reports use different stats for this)
> As we had to log out all users from matrix.org, if you do not have backups of your encryption keys you will not be able to read your encrypted conversation history
That seems like a fairly bad usability/security design?
content before it gets fixed:
Time for actual transparency.
[list of servers, uname -a for each]
root@[name]:/var/lib/postgresql# df -h
[list of partitions]
$ cat users.txt | grep [name] | head -n1
@[name]:matrix.org|[hash]
$ wc -l users.txt
[~6M users]
See you soon.
Here's the source of that page: https://github.com/matrixnotorg/matrixnotorg.github.io/blob/....
account got banned :) (it's on archive.org though)
Why was Jenkins running on a production server?
it was not, read again
But it does seem to be the case that the same SSH key pair that was used to access Jenkins also provided access to the production infrastructure. Unless I'm misunderstanding the nature of the attack.
It seems the issue was developers using SSH agent forwarding which was abused to access the production environment.