You should always verify that SSH password auth is actually off; run
ssh -v myserver : 2>&1 | grep continue
and ensure that it only gives "publickey"!
(A surprising number of VPSs will re-enable passwords in a .d config file. And really, even if you've checked for that, the extra 10 seconds to make sure is worth it.)
And from the server itself. Some versions won't allow this command without also creating a prevsep directory e.g. mkdir /run/sshd if sshd is not set to automatically start.
sshd -T | grep -i pas
passwordauthentication yes
permitemptypasswords yes
I permit this intentionally for my own reasons on my publicly accessible servers.
as an emergency recovery mechanism before you first run
sudo ufw enable
That way, if you've screwed up and locked yourself out with your new firewall rules, you can just wait 5 minutes and log back in (instead of paying for remote hands at your datacenter, or blowing away your vps and rebuilding from scratch).
Remember to re enable the firewall or stop the at job if everything works for you.
Alternately ensure you have console access from the VPS providers web console / terminal. Then you can safely stop/start sshd, vpn daemons, bork up firewall rules, etc...
On some VPS providers you can also instantly reboot into a rescue full blown OS that runs in memory and then mount and chroot into whatever disks that need to be fixed.
That's true, but I prefer to ingrain habits that'll work everywhere, instead of relying on stuff like vps console access which will work fine if you're working on your ec2 instance or your DO droplet, but will not work out so well when it's your home server or colo-ed box that you're trying to remotely secure from a hotel room while traveling.
But as @wink points out, these days you also need to ensure you've actually got at available - which is not guaranteed especially with cut down distros like Alpine.
And check if `at` works before that. Maybe I'm still scarred from that one default install years ago where it was broken, or even left out. Unlike cron I don't see it as a given these days.
For SSH, all you need is to allow only pubkey authentication. The other stuff, like Fail2ban aren’t necessary. If you want to clean up logs, you can change port number too.
Better yet, use outbound only SSH (with a tool provided by the VPS provider like AWS session manager or third party).
Note that ports which are not bound to the host (i.e., -p 5432:5432 instead of -p 127.0.0.1:5432:5432) will be accessible from the outside. This also applies if you configured UFW to block this specific port, as Docker manages its own iptables rules. https://docs.docker.com/engine/network/packet-filtering-fire...
As far as Fail2ban goes, using it to lock the door is good, but removing the door entirely is better.
Fail2ban is useful for limiting failed access attempts, but closing the SSH port altogether limits attack pathways to only trusted parties in the first place — assuming SSH isn’t meant to be publicly accessible.
There are many modern technology options for enabling private access without needing to open firewall ports, many are listed at https://zerotrustnetworkaccess.info
Of these, mesh overlay networks appear to be gaining the most traction lately, especially among the HN crowd.
I would have appreciated the rationale behind setting 'UsePAM' to 'no'. I assume it's because, with password auth disabled, it's not necessary, and better to disable something that you don't need that would otherwise add to the attack surface?
fail2ban is fine for a small VPS, but it is very slow, less necessary for key based authentication (passwords shouldn't be used, unless absolutely required) and shouldn't be used on high load systems, with that said, I've seen guides on its rewrites, which were able to handle gigabits per second. ufw can be replaced with a very simple iptables/ip6tables script, if you don't have complex routing. Not a single daemon required. Beware of things like Docker overriding them though. unattended-upgrade is often leaking memory all over the place, I use regular manual updates instead.
I recommend installing Tailscale, and using a firewall outside of the VPS like how Linode or EC2 does it. Don't even allow port 22 at all... Tailscale let's you access all your stuff without needing to expose ssh ports.
Most cloud providers will accept a cloud init cloud config file. Sometimes they call it a “script” but they still accept the Yaml file if it has the `#cloud-config`.
There are modules for setting up ssh, adding users, installing packages, etc.
Honestly I have never used it. That and OSSEC can lock people out when they are using the wrong keys or wrong passwords if passwords are enabled there are legit use cases for passwords.
Begin Disclaimer: The following comment is not for security, but only to reduce log noise :: I use a high port for non public SFTP servers to avoid noise. :: End Dislclaimer. It's not like the bots are getting in but I don't want the log noise. Not security through obscurity, just noise reduction. This is just for my bastion nodes. All other nodes require specific IP addresses or Tinc VPN to reach that high port.
To avoid 99% of the bots I use IPTables MSS rules to drop anything outside of the MSS any of my clients will show up with. I also personally drop anything with a TCP SYN packet TTL greater than 64 since all my clients are Linux. Windows is 128 and cell phones / LTE devices most of the bots are greater than 128. Don't do this in a corporate environment and instead require people use a corporate VPN. The following IPTables rules can be modified to include or exclude any ports or ranges of ports.
# (raw table for public bastion, using default port 22 as an example)
-A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d ${WAN_IP} --syn -m tcpmss ! --mss 1460 -j DROP
-A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d ${WAN_IP} --syn -m ttl --ttl-gt 64 -j DROP
People can alter their TCP TTL but in reality they do not. FWIW a bot has never reached any of my sshd daemons in 25+ years on nodes that I do not expect random IP's to connect.
sshd can also be told to only listen on IPv4 or IPv6 whichever you prefer. e.g. addressfamily inet for IPv4 only.
Use fail2ban if you use popular online web apps like Wordpress, cpanel, Drupal, etc.
It won't protect you from a direct attack, and it's no substitute for good system security, but it can quickly block attackers who "scan" your system to see what's running, to see what they can attack. It won't stop a determined attacker who will be able to attack from many locations, but that doesn't make it useless.
You can also use it actively to block people who do things like hammering a password form.
fail2ban is a critical piece of security software, as is some firewall. for those new to linux, you might as well use the one that is super-easy to install.
The opposite is true about fail2ban: it's cargo-cult security, and people shouldn't be running it. It never made any sense, but it especially makes no sense if you're going to (sensibly) disable password authentication.
I agree for ssh - but I use it on a couple servers that have WordPress for the sole purpose of blocking IPs that engage in brute force attempts.
It has a real and dramatic impact on a few things - I got CPU warnings from one server a couple weeks back because I'd inadvertently broken logging and fail2ban stopped working and someone was doing a persistent brute force at high volume for 8 hours.
After I fixed fail2ban it dropped off immediately. I know some WordPress plugins will do this but I've not had much success with them compared to fail2ban so it's still my default for this purpose.
Are there better system-level approaches than fail2ban in this case? Or is your comment mostly directed towards those using it for ssh blocking?
You're talking about it as if it only handles SSH authentication.
It's no substitute for proper security, but it's a great mechanism for quickly dropping naughty traffic. Even people who practice good security have online systems that are occasionally vulnerable to zero days, or have password forms on webapps. Dropping traffic while it's still in discovery mode is a great way to frustrate automated attacks.
Honestly, for somebody running a personal server, ufw/iptables/etc tend to not be relevant for any direct security. Basically nobody is doing anything for outbound rules other than ACCEPT, and for inbound... people can only connect to services you're running on your public interface. So for most people, they'd just be setting up ufw/iptables/etc to allow traffic to the set of services they're running (ssh, a web server, etc) anyways.
That said, I do think there's some value in understanding what your server is running and what it's exposing to the network, and setting up firewall rules is one of many ways to build familiarity with that.
fail2ban is just junk, and shouldn't be running on any modern system.
Always remember though (and, before anyone says this is obvious, I once shocked an HN reader who hadn’t thought this through):
All of your VPS security is meaningless compared to your VPS provider’s security, the security imposed on the employees of the VPS provider, the security of your VPS account password, and their susceptibility to faked subpoenas and faked emergency data requests.
Far more VPSs are compromised through local VM attacks than provider attacks.
This is a bit like saying "When you're temping your chicken, always remember that your grocery store could be putting arsenic in it at the store". Yea, that's true. But the average consumer gets sick far more often from mistakes in the kitchen than mistakes in the grocery store.
The person in question on HN was planning to self-host email. Sure, if you don’t want Google or Microsoft in particular reading it, I get it. But if you think nobody can read it now, that’s a major (potentially life threatening) mistake. Gmail might actually be safer in many circumstances.
I wouldn't say meaningless, no. If your provider is not too diligent about checking subpoenas, they will let a sophisticated targeted attacker in eventually. If your SSH password is weak, you will let any and all script kiddie in within the hour.
You should always verify that SSH password auth is actually off; run
and ensure that it only gives "publickey"!(A surprising number of VPSs will re-enable passwords in a .d config file. And really, even if you've checked for that, the extra 10 seconds to make sure is worth it.)
Good advice. Another option:
And from the server itself. Some versions won't allow this command without also creating a prevsep directory e.g. mkdir /run/sshd if sshd is not set to automatically start.
I permit this intentionally for my own reasons on my publicly accessible servers."Trust but verify" is good advice.
I'd strongly suggest adding:
as an emergency recovery mechanism before you first run That way, if you've screwed up and locked yourself out with your new firewall rules, you can just wait 5 minutes and log back in (instead of paying for remote hands at your datacenter, or blowing away your vps and rebuilding from scratch).Remember to re enable the firewall or stop the at job if everything works for you.
Alternately ensure you have console access from the VPS providers web console / terminal. Then you can safely stop/start sshd, vpn daemons, bork up firewall rules, etc...
On some VPS providers you can also instantly reboot into a rescue full blown OS that runs in memory and then mount and chroot into whatever disks that need to be fixed.
That's true, but I prefer to ingrain habits that'll work everywhere, instead of relying on stuff like vps console access which will work fine if you're working on your ec2 instance or your DO droplet, but will not work out so well when it's your home server or colo-ed box that you're trying to remotely secure from a hotel room while traveling.
But as @wink points out, these days you also need to ensure you've actually got at available - which is not guaranteed especially with cut down distros like Alpine.
And check if `at` works before that. Maybe I'm still scarred from that one default install years ago where it was broken, or even left out. Unlike cron I don't see it as a given these days.
you could always
(sleep 5m && sudo ufw disable) &
Totally, I'd rather run it directly via root not sudo to skip one more potential problem.
For SSH, all you need is to allow only pubkey authentication. The other stuff, like Fail2ban aren’t necessary. If you want to clean up logs, you can change port number too.
Better yet, use outbound only SSH (with a tool provided by the VPS provider like AWS session manager or third party).
Be careful with Docker ( crypto miner )!
https://www.reddit.com/r/docker/comments/18m1k0b/holy_sht_a_...
Note that ports which are not bound to the host (i.e., -p 5432:5432 instead of -p 127.0.0.1:5432:5432) will be accessible from the outside. This also applies if you configured UFW to block this specific port, as Docker manages its own iptables rules. https://docs.docker.com/engine/network/packet-filtering-fire...
This guide walks you through disabling password authentication for SSH, then walks you through setting up fail2ban.
Although mostly pointless it would reduce log spam.
If that's the issue why not just change port?
As far as Fail2ban goes, using it to lock the door is good, but removing the door entirely is better.
Fail2ban is useful for limiting failed access attempts, but closing the SSH port altogether limits attack pathways to only trusted parties in the first place — assuming SSH isn’t meant to be publicly accessible.
There are many modern technology options for enabling private access without needing to open firewall ports, many are listed at https://zerotrustnetworkaccess.info
Of these, mesh overlay networks appear to be gaining the most traction lately, especially among the HN crowd.
I would have appreciated the rationale behind setting 'UsePAM' to 'no'. I assume it's because, with password auth disabled, it's not necessary, and better to disable something that you don't need that would otherwise add to the attack surface?
They are probably just trying to avoid PAM mistakes which are common when people hand edit pam modules.
For completeness sake if someone is using SELinux Enforcing mode then UsePam will likely need to be Yes to avoid breaking sshd mandatory access rules.
fail2ban is fine for a small VPS, but it is very slow, less necessary for key based authentication (passwords shouldn't be used, unless absolutely required) and shouldn't be used on high load systems, with that said, I've seen guides on its rewrites, which were able to handle gigabits per second. ufw can be replaced with a very simple iptables/ip6tables script, if you don't have complex routing. Not a single daemon required. Beware of things like Docker overriding them though. unattended-upgrade is often leaking memory all over the place, I use regular manual updates instead.
Has anybody achieved a workflow that needs only browser based shells, like EC2 instance connect?
I recommend installing Tailscale, and using a firewall outside of the VPS like how Linode or EC2 does it. Don't even allow port 22 at all... Tailscale let's you access all your stuff without needing to expose ssh ports.
It’s insane to me that in 2024 it still takes this much back and forth to set up a server.
I even recently tried automating this but quickly realized I was falling into an XKCD trap because the process is so damn painful.
Most cloud providers will accept a cloud init cloud config file. Sometimes they call it a “script” but they still accept the Yaml file if it has the `#cloud-config`.
There are modules for setting up ssh, adding users, installing packages, etc.
https://cloudinit.readthedocs.io/en/latest/index.html
opinions on the suggested fail2ban and ufw?
Honestly I have never used it. That and OSSEC can lock people out when they are using the wrong keys or wrong passwords if passwords are enabled there are legit use cases for passwords.
Begin Disclaimer: The following comment is not for security, but only to reduce log noise :: I use a high port for non public SFTP servers to avoid noise. :: End Dislclaimer. It's not like the bots are getting in but I don't want the log noise. Not security through obscurity, just noise reduction. This is just for my bastion nodes. All other nodes require specific IP addresses or Tinc VPN to reach that high port.
To avoid 99% of the bots I use IPTables MSS rules to drop anything outside of the MSS any of my clients will show up with. I also personally drop anything with a TCP SYN packet TTL greater than 64 since all my clients are Linux. Windows is 128 and cell phones / LTE devices most of the bots are greater than 128. Don't do this in a corporate environment and instead require people use a corporate VPN. The following IPTables rules can be modified to include or exclude any ports or ranges of ports.
People can alter their TCP TTL but in reality they do not. FWIW a bot has never reached any of my sshd daemons in 25+ years on nodes that I do not expect random IP's to connect.sshd can also be told to only listen on IPv4 or IPv6 whichever you prefer. e.g. addressfamily inet for IPv4 only.
Use fail2ban if you use popular online web apps like Wordpress, cpanel, Drupal, etc.
It won't protect you from a direct attack, and it's no substitute for good system security, but it can quickly block attackers who "scan" your system to see what's running, to see what they can attack. It won't stop a determined attacker who will be able to attack from many locations, but that doesn't make it useless.
You can also use it actively to block people who do things like hammering a password form.
fail2ban is a critical piece of security software, as is some firewall. for those new to linux, you might as well use the one that is super-easy to install.
The opposite is true about fail2ban: it's cargo-cult security, and people shouldn't be running it. It never made any sense, but it especially makes no sense if you're going to (sensibly) disable password authentication.
I agree for ssh - but I use it on a couple servers that have WordPress for the sole purpose of blocking IPs that engage in brute force attempts.
It has a real and dramatic impact on a few things - I got CPU warnings from one server a couple weeks back because I'd inadvertently broken logging and fail2ban stopped working and someone was doing a persistent brute force at high volume for 8 hours.
After I fixed fail2ban it dropped off immediately. I know some WordPress plugins will do this but I've not had much success with them compared to fail2ban so it's still my default for this purpose.
Are there better system-level approaches than fail2ban in this case? Or is your comment mostly directed towards those using it for ssh blocking?
Just SSH.
You're talking about it as if it only handles SSH authentication.
It's no substitute for proper security, but it's a great mechanism for quickly dropping naughty traffic. Even people who practice good security have online systems that are occasionally vulnerable to zero days, or have password forms on webapps. Dropping traffic while it's still in discovery mode is a great way to frustrate automated attacks.
ufw yes, fail2ban doesn't necessarily do much for you if you have disabled password authentication.
Honestly, for somebody running a personal server, ufw/iptables/etc tend to not be relevant for any direct security. Basically nobody is doing anything for outbound rules other than ACCEPT, and for inbound... people can only connect to services you're running on your public interface. So for most people, they'd just be setting up ufw/iptables/etc to allow traffic to the set of services they're running (ssh, a web server, etc) anyways.
That said, I do think there's some value in understanding what your server is running and what it's exposing to the network, and setting up firewall rules is one of many ways to build familiarity with that.
fail2ban is just junk, and shouldn't be running on any modern system.
Always remember though (and, before anyone says this is obvious, I once shocked an HN reader who hadn’t thought this through):
All of your VPS security is meaningless compared to your VPS provider’s security, the security imposed on the employees of the VPS provider, the security of your VPS account password, and their susceptibility to faked subpoenas and faked emergency data requests.
Far more VPSs are compromised through local VM attacks than provider attacks.
This is a bit like saying "When you're temping your chicken, always remember that your grocery store could be putting arsenic in it at the store". Yea, that's true. But the average consumer gets sick far more often from mistakes in the kitchen than mistakes in the grocery store.
The person in question on HN was planning to self-host email. Sure, if you don’t want Google or Microsoft in particular reading it, I get it. But if you think nobody can read it now, that’s a major (potentially life threatening) mistake. Gmail might actually be safer in many circumstances.
If you don’t want Google or Microsoft reading your email, Proton would be better still than Gmail, wouldn’t it?
If any of your recipients are using Google or Microsoft then they already have your email whether you use Proton or not.
I wouldn't say meaningless, no. If your provider is not too diligent about checking subpoenas, they will let a sophisticated targeted attacker in eventually. If your SSH password is weak, you will let any and all script kiddie in within the hour.