AS13335 doing SSH scanning
encryp.chThis is not unique to Cloudflare, Level3 or any of the other networks I've seen mentioned in this thread. The entire internet scans all the common ports and this should be expected to continue indefinitely. If this is causing alerts from your logging, there are a few options:
- Implement a firewall on your instances.
- Implement port knocking for sshd.
- Move sshd to a non standard port to avoid the nmap/bot noise.
- Only log successful logins.
- Any combination of the above.
There are pros and cons to each item so you would have to decide which combination is the most appropriate and least friction for your organization. If leaving the port exposed consider enforcing key based authentication and disabling password authentication. For high risk accounts such as monitoring accounts that use passwordless unrestricted sudo, you can even restrict what networks the keys are valid from.
https://github.com/jftuga/gofwd
I created gofwd, a cross-platform TCP port forwarder with Duo 2FA and Geographic IP integration. Its use case is to help protect services when using a VPN is not possible. While it's use cases are limited, it works great for me when I need to connect to my home network.
The overall elegance of this solution is that no additional software is needed. As long as you are within your predefined geo-ip location, have your phone, and know your hostname/ip address (and port number), then you will be able to access your system remotely.
I've been happy with having my firewall count SSH connection attempts, and if you make too many attempts in too short a period of time, it just blacklists that IP for 24 hours.
My first 'defense' was just moving off of port 22. I had planned other things, but to be honest, the attempts went from hundreds a day to zero, so I never bothered doing more.
The first package I install on public facing servers is fail2ban.
After starting it, everything becomes much calmer. Nice thing is, fail2ban can protect much more than SSH.
I do the same, but only 10 minutes after 5 failed logins. That's enough for bots while not really getting in your way if it accidentally blocks the hotel you're in.
Combined with only allowing key based login, password is disabled.
I've recently added 2FA as well. Super easy to do.
I wrote a blog on how to do it a couple years back if anyone is interested in a "tutorial": https://2byt.es/post/totp/
It's for the Pi as that's the target audience but it should apply generally for OpenSSH.
There's also a sister post about improving your "first factor" for those still using passwords: https://2byt.es/post/totp2/
My experience was similar. I opened a public SSH once and there were many unsuccessful login attempts. The way we resolved it was by installing VPN and closing the IP for public access.
My favorite is to implement a tarpit while moving the ssh port.
> My favorite is to implement a tarpit while moving the ssh port.
For those who, like me, are unfamiliar with the concept of a SSH tarpit, it's a technique consisting of inserting delays into SSH connection attempts.
Old HN discussion on Endlessh:
Isn’t this easily detectable for any malicious script? Just a small timeout and try the next port.
Does that mean that you automatically block IPs that try to login on port 22 (since they are obviously malicious/port scannners)? If yes, is there any specialized daemon for that or are you using firewall rules? Or are you running a SSH honeypot on port 22?
The post is referring to this: https://github.com/skeeto/endlessh
I'm guessing that the person you replied to is referring to the same.
A tarpit is a technique to waste your opponents time and network sockets. Usually by accepting the connection but only sending or receiving data at like 1byte/s.
Personally, I just whitelist SSH access on the firewall (UFW and/or AWS SecGrps). Everyone on the admin team has a static residential address. On the road, we use ZeroTier.
Adding: [-] whitelist allowed ip's for port 22. Requires static ip's client-side obviously [-] fail2ban if static ip's are not available.
> - Move sshd to a non standard port to avoid the nmap/bot noise.
There are scanners like shodan that will scan every single port you have now so moving it to a non-standard port doesn't stop all the attackers.
Shodan scans a handful of common ports [1] but you are right, a targeted attack will not be stopped by moving ports. Along the lines of what jms703 mentioned, moving ports just removes the noise from the bots so you can have better alerting on real attacks and not suffer from alert fatigue.
The parent post didn’t say to do this to stop attackers. They said to do this to reduce alerts in the logs.
Or Fail2Ban. I have mine setup so that it does an API call to my OPNSense firewall. A few wrong logins to my self hosted things (SSH included) and that IP is permanently blocked from everything.
If you expose sshd to the internet in the first place you are doing something wrong.
Update: they apparently also have a paid product called "Cloudflare for Teams", which also uses the same infrastructure and are targeted for companies. Probably the reason they don't block out SSH is that companies do want SSH access - but I'm pretty sure that RDP access would be higher on the list.
My friend working in an ISP has cleared the mystery for me - it's Cloudflare's Warp product (a semi-VPN (in the "teleport to other countries" sense, not the original definition)), which can be used for free, and I'm not shocked that it is being abused. Cloudflare has also categorically denied that their Worker product allows SSH connections - it is simply not designed for that, apparently.
So I don't know how to respond to this specifically except that I didn't know Cloudflare offers a VPN (apparently a limited knocked-down one though) and I'm pretty sure that other people in other ASes scan the default SSH port - in fact, I see more and more scans from ASes of cloud providers than residential ASes, whereas five years ago I mainly saw Chinese and Brazilian residential ISPs. This is why I often move the SSH port - it's not that it improves my security posture, just so that the noise in the logs are minimised.
> Cloudflare has also categorically denied that their Worker product allows SSH connections - it is simply not designed for that, apparently.
Hi, I'm the tech lead of Workers, so I'll clarify.
At present, Workers can only generate HTTP traffic, because the only API we've given Workers for network communications is `fetch()`, which is HTTP-specific. This similar to how in-browser JavaScript today cannot open arbitrary TCP connections, because there's no API for it. It's likely we'll add an API for arbitrary TCP eventually, but at the moment it's not possible that this traffic comes from Workers.
(As far as where it did come from, I don't have any internal knowledge, but WARP traffic (that's our VPN-like service) seems like an obvious explanation to me.)
His dump of packets shows an MSS of 1380 versus the more common default 1460, which sounds like it's coming from something with tunneled traffic. MSS 1380 would correspond to a source MTU of 1420, which is a typical default for a GRE tunnel in IPv4sec Tunnel mode. Pretty sure that's what WARP is.
WARP is Wireguard but yeah, otherwise this is pretty much on point. Traffic is tunneled, likely just a random WARP user abusing Cloudflares free VPN service to do some scanning.
Ah, okay. Google searches suggests the default MTU on Wireguard is also 1420, which would produce an MSS of 1380. So that makes sense.
a semi-VPN (in the "teleport to other countries" sense, not the original definition)
It's weird how the "VPN providers" have taken that acronym and turned it into what is basically another synonym for "proxy".
FYI the WARP VPN is _not anonymizing_ and doesn’t claim to be. If you connect through WARP to a website behind Cloudflare, they will include your real IP in a header.
The main point of WARP is to circumvent ISP traffic shaping (which works quite well btw).
I mean, I just learned about Warp because of this, so if there are misunderstandings that my friend didn't think to clarify (probably because she thought that I already knew Warp), then apologies. The HN edit clock has since been long-expired, so I cannot add corrections about this.
Cloudflare has a VPN product, WARP. It's possible that they are clients of theirs doing the scanning.
In any case I wouldn't think much of it. If you obsess over any garbage traffic you get you will go insane.
Maybe that's a case, but their abuse team hasn't replied anything in a 2 weeks about that after I gave them all timestamps and both source/destination IP addresses
What kind of answer would you expect, in all seriousness? The thing you are reporting is not illegal or even dodgy.
VPS and VPN providers should be very mindful of their reputation in this regard. If they get a reputation as a "black hole" where complaints vanish and nothing ever happens, the effect may be that other customers start to find themselves blocked or throttled on a subnet level.
I don’t know of any provider that would take action on reports of ssh scanning.
I report them from time to time. I'm not sure why people don't take them seriously. If 500 people a day came up to your front door and tried the knob, hell maybe even tried a couple of keys in the lock... I'm pretty sure you'd be calling the cops.
An ssh server isn’t really like a front door though. I don’t necessarily think attempting a random ssh server should be a crime.
I used to spend time on custom iptables scripts but came to the conclusion it’s much better to just architect things in a way where the bots and scanners can’t plausibly create a problem and then ignoring them.
A little bird told me a story that AWS will forward abuse reports to customers performing outbound nmap scans.
I don’t think anyone is about to block or throttle traffic from Cloudflare IP ranges.
Incoming connections? Why not. I have at times, it's pretty great.
I expect to hear who (and why) generates that kind of traffic from cloudflare owned subnets.
I can understand your frustration with background internet noise, but please note Cloudflare is not known for broadcasting their customers' names to the first abuse report with a pcap of a TCP handshake.
There may be more realistic ways to go about protecting people's SSH servers that trying to dox Cloudflare VPN users.
I can‘t understand it. There doesn‘t appear to be any downside or even abuse happening. The fact OP expects a company to explain who and why a customer of theirs did a legal non abusive act is just an outstanding level of entitlement.
Who's trying to dox those users?
I assumed OP wants to know the identity of the Cloudflare users scanning their SSH ports.
I think OP guessed it was probably not Cloudflare themselves scanning their ports, so I think that's what they meant by "hear who and why".
Maybe dox is too strong a word. My point is, from what I've heard, the general sentiment is that you're unlikely to get any information about customers just by sending abuse reports to Cloudflare.
OP obviously simply expected at least an explanation on the cause of these ssh connection probes. He got the explanation here (VPN).
If your ISP and the server support IPv6, just disable SSH on IPv4.
Some of my servers don't even have any IPv4 connectivity and there haven't been any failed SSH logins over IPv6.
My OpenSSH is located on a non standard port, 22/tcp is going to the endlessh honeypot.
> endlessh honeypot.
*tarpit
A honeypot lets people "in" to see/research malware that's in the wild:
* https://en.wikipedia.org/wiki/Honeypot_(computing)
A tarpit just takes up the attacker's resources:
Not what most people run, but SSH honeypots are also useful:
Not to disappoint you, but except for logging SSH honeypots are becoming useless (most bots automatically disconnect when they detect a long login banner).
So should I add a long banner to my server to disguise it as a honey pot, just in case?
I mean, I'm not sure that you can do that with OpenSSH though (short of recompiling it, which I do not recommend unless you're a company).
I like this solution a lot!
They won't disconnect clients over SSH scanning. That's a ridiculous expectation. It's 2021. Nobody cares.
It could also be used for IP reputation, running ssh can be a sign of a proxy or other server/service that website owners might want to block.
I think single SYN packet would be enough to know whether this port open or not, but they trying to gather full banner.
This is probably Cloudflare Workers?
I don't think that demanding that every ISP inspect and censor outgoing packets is ever going to work. Every network operator has their own definition of good and bad, and it's probably up to that operator to enforce it themselves. Regarding SSH, you can basically assume that 4.2 billion IPs will concurrently try every password database that exists against your server 24/7. Passwords are a failed experiment. People can't remember them, and attackers can easily guess them. The solution is: don't accept password logins, and ban particular IPs after a certain number of failed attempts. (You should do the same for any unauthenticated resource -- if someone gets 1000 pages from your website in a second, you should probably cut off access to them for a while. Some script has gone awry, and it's costing you money, not the author of the script.)
The topics that HN discusses can show you the breadth of opinions on this issue. One day it's "I hate cloud provider X for blocking my abusive service" with 3000 comments agreeing. The next day it's "I hate cloud provider Y for not blocking someone's abusive service" with a different 3000 commeters agreeing. No consensus will ever be reached, and there is no Internet police force you can bully into agreeing with you. I would just take precautions and move on, rather than appealing to a mob or administrative body for some meaningless justice.
Probably yes, but hey! Why do Cloudflare Worker would need SSH connection establishment? I do not asking Cloudflare to block 22/tcp entirely, but all this situation is very odd - I am seeing anomaly, reported abuse to them, but no explanation why this is happening.
About password authentication I totally agreeing with you, but this is a bit out of scope of this thread.
The only purpose of this article is to know the truth what is really happening. I have never seen such many connections to the SSH even from researchers.
I think you need to spend more time looking at logs and network traffic in general, this is standard. I bet your public home IP will be scanned at least a couple times just today.
Judging my my home network, they can probably expect an SSH connection attempt on a home IP about every 5 seconds or so, or 18,000 times a day.
For some reason, this is 6× more than a server I have on an university network.
Please name the cloud providers that have a policy of allowing malicious actors to do SSH scans? Do you mean Cloudflare?
Let me know if you get banned for doing nmap -p 22 on AWS or GCP.
They used to block port scanning, back in 2015:
> When unauthorized port scanning is detected by AWS, it is stopped and blocked.
Page 13
https://web.archive.org/web/20160331233541/https://d0.awssta...
It's not clear, now.
We get obnoxious security cgi scanner from palo alto networks tripping alerts all the time coming from gcp ranges
Or it’s not detected. They’re trying to avoid being blocked anyway.
As others have pointed out, disabling password-based logins for sshd is a must.
You can also go further if you like: Put the sshd service behind WireGuard (or spiped). Then malicious actors will never find your SSH ports.
To the best of my knowledge, the IP blocks in the 8.0.0.0/8 range announced by cloudflare are their VPN exit blocks.
I had similar issues with them in the past spewing weird traffic. Didn't really ever reach a conclusion on it, other than just reducing logging for that error condition a scanner was triggering.
How did you discover this? Anyway thanks for additional information
Enabling Warp via the "1.1.1.1" Android app gets me an 8.x.x.x VPN address, at least. This /24 appears to be routed to my city's Cloudflare node, so presumably there's a /24 per city they run this service in.
Running a quick port scan from my phone against one of my machines works, so it doesn't look like they are restricting this too heavily.
And I'm not logged into this app and haven't granted it additional permissions, so I'm not sure they have any idea who I am here.
via support ticket with cloudflare. They didn't explicitly confirm it, but my experience is that they would have corrected me if that was wrong :)
jimsi, employees are also extremely active on Twitter. It’s not the most scalable but that’s usually my goto for eyebrow raising Cf related questions.
Just use something like fail2ban to drop these at the packet filter level, either courtesy of your cloud provider or on the host itself. Make sure your sshd config is watertight.
You know, or just don’t use fail2ban: https://research.securitum.com/fail2ban-remote-code-executio... . It’s adding extra attack surface for a cosmetic benefit.
Good catch, thanks!
Remedy: Don't let fail2ban send mail, or at least remove the whois part.
Are there reputable places where people can share the IPs tripping fail2ban? Like spamhaus.org, but for scanners?
Honest question: Why bother?
nftables allows to automatize blocking of IP addresses (even with a automatic timeout) without any userspace involvement.
Pretty nifty feature.
What is the issue exactly? Is there any kind of exploitation attempts or is someone fishing for banners to do statistics upon? The second case is research, and legitimate...
> What is the issue exactly?
The OP has too much time on their hands. :)
You can play this whack-a-mole game indefinitely. If you have SSH open to the Internet, expect that others will try to connect 24/7.
I have asked Cloudflare about this traffic, but support says "Cloudflare helps protect sites, and accelerate them. We do not attack sites, and our network can't be used to generate attack traffic." They deny any malicious outgoing traffic, but after I have provided pcap dump they just ignoring me and do not reply anything. Very odd behaviour for researchers.
Cloudflare likes to use the "We're just a proxy" defense when it suits them, but with the rapid release of other products they are very capable of generating arbitrary traffic and hosting content without an origin.
Just curious/not furious: is there a convincing argument for why one should enable this particular kind of research at the cost of potentially diminished security? I recognize that this question relies on an assumption that is itself arguable, and that it might vary with context.
(Aside: I really wish English had a better way to distinguish between an earnest question and a rhetorical challenge.)
Is SSH scanning considered "abuse" now?
Port scanning was always considered abuse until researchers decided they needed additional blog post material. I worked for a “business ISP” in the early 90s and we had no qualms kicking someone off the network for it, although usually the originator of the scan had themselves been compromised… because… as I say it was considered at least a breach of terms of service.
I ran engineering for what was at the time the most popular ISP in Chicago from 1995-1997 (the company later sold to RCN) and port scanning has never been considered abuse, though there have always been people who wished it was and loudly proclaimed it so.
I'd say that port scanning is attack prep. When you port scan me I take an adversarial stance. Certainly volume plays a role in determining whether scanning is abusive or not.
There are differences between walking by and noticing a door, knocking on a door, trying a key in a door, trying to defeat the lock. Before you can break the lock you have to notice the door. I haven't met any friendly door knockers on the internet.
Yes you have. Censys is a good example (you have assuredly met Censys, whether you know it or not.)
Some ssh scanning is obviously abuse. Is it hard to make the difference ?
Why do you think this matters in any way whatsoever? Ports are there to be connected to. I don't see the problem.
So...drop 8.0.0.0/8 traffic and move on? Port scans happen all the time, some are malicious and others not.
I don't think dropping traffic from roughly 1/200th of public IP space is a real solution.
Isn't it? If you don't need to accept SSH traffic from that block you can just drop it.
Just install sshguard or some such and move on with your life. Or - block the entire IP range, and see who complains.
Yeah lots of baddies using workers. I blocked cloudflare a long time ago because of workers.
I run OSSEC that comes with automatic responses for common SSH authentication events
Cloudflare warp is essentially a free vpn, this is going to happen. Iirc they use completely separate subnets for this so you can filter it/block it. Ip starts with 8. Something
Yep, if the site you're accessing is not on the cloudflare network, they'll see an ip starting with 8.
In case Cloudflare monitors the keyword "Cloudflare" in HN comments, I'll mention it here – Cloudflare.
Cloudflare is all over HN and their CTO, jgrahamc, will frequently appear to answer questions. California is just waking up so I expect a few ho it a before we get some solid answers to this issue.
Previously this post was named in such way, but post get flagged... So I removed its mentioning and replaced with ASN