If you don't allow password logins so ssh only accepts public/private keypair authentication I think you have a secure setup. Limiting the IPs allowed to connect in the firewall will block the bots that probe port 22 and brute-force attacks, but those aren't going to succeed anyway. As far as I know it's not possible to brute force or otherwise hack ssh with ssl keypair authentication directly. An attacker would need access to your ssl private key. The ssh protocol resists MITM attacks as well.

If anyone knows about actual working attacks on ssh servers that don't allow password authentication I'd like to know about it. I don't mean state-level attackers who would probably hack into your laptop or strong-arm the hosting provider.

Theoretically, a vulnerability in SSH could expose you, but I think those are quite rare. The bigger risk of keeping the server exposed is probably that some application opens another port. But that won't happen on Port 22, so that one could stay open to all.

I think ssh has been scrutinized by hackers already. Vulnerabilities I know about have to do with servers using deprecated encryption algorithms.

Firewall rules in front of ssh are a good idea but I don’t think that adds security to ssh itself. It does block apps opening ports, as you point out.

OpenSSH is next to Wireguard in security, and, in my view, not too far behind (roughly similar to OpenVPN).

I don’t think state level actors can compromise it, if properly configured.

It’s far easier to hack the end point and steal the keys.

You could try using Tailscale. It's a wireguard based mesh VPN. Just set it up on your device, your server, and you're good to go. You can just only allow SSH over Tailscale, without worrying about anybody else trying to get in.

Other alternatives include ZeroTier and Nebula afaik, but I've never tried either.

(Still use SSH keys regardless though.)

Personally I don't think there is any merit of using tailscale over just public key auth. The way that can go wrong is if there is a vulnerability in sshd, which would be well disastrous.

Zerotier is the easy way around. Also remember to block all ssh access at standard ports.

That implies running SSH on another, non-standard port makes it safer -- when a simple port scan would reveal it. It's a classic security through obscurity fallacy, IMHO,

I did not say/imply that the user should implement a ONLY non-standard port as security.

Sure " simple port scan would reveal it" - but for those other script kiddes that do not do have it prevents it. And BTW, every bit helps of security helps. zerotier at an alternative port is slightly better than port 22.

Oracle has a generous free tier for OCI instances/VMs, and includes a public IP. You can use that as an ssh bastion/jumphost.

Or if don’t want to do business with Oracle, you can find VPS/VM providers who offer tiny instances for $10-12/yr or less. You don’t need much ram for a bastion. 128M ram would do the trick, and even 64M (you’d have to use a stripped down image/distro though).

There are even providers who offer instances for even less $ and only give you an ipv6 range + NAT IPv4 address with a small port range. This would also work perfectly fine as a bastion.

Just be careful with a bastion host that you have a fallback. If anything happens to the bastion host and you don't have access to that IP, your VPS is basically lost. If you use a bastion host, I'd at least have two.

This shouldn't be a big problem as you can use configuration management such Ansible to get around. In case of bastion loss you can simply create a new instance and run the script to recover your instances.

The same question applies to oracle VPS: how to allow only few IPs?

A compromise of oracle VPS and attacking AWS VPS is same as attacking AWS VPS from internet. If oracle VPS is not compromised, neither will be AWS.

Don’t see the point in this case.

The advantage is that you only need to harden one instance. The others are safe as they're basically in an "internal" network (firewall blocks all other IPs). With that bastion host, you'd do anything to make it as secure as possible (fail2ban or the like, authkey, block countries you won't access from, etc). For small projects, it's also reasonable to check logs from one host, but not to do so for 10 hosts.

Sorry I am missing your point. That makes sense if first VPS is serving multiple ssh servers; so you harden one instance. In this case, the person has only one instance. Instead of hardening oracle VPs, they could harden aws instance.

Bastion makes sense if it’s locked down more than destination. This doesn’t apply if there is only one destination and one public service (SSH).

I suggest using AWS AMS or putting it behind vpn.

If his AWS instance is running internet facing applications and services, then it makes sense to have the AWS firewall lock down port 22 to a single IP or two (eg your bastions), and also have that AWS firewall only allow all access to those specific internet facing ports for any relevant applications. Yes you can and should also harden at the OS level. But it’s smart to utilize AWS’ security as well as much as possible on any instance that you need to be in production and available. You want to limit it’s exposure in any way you can.

So back to the bastion. You have the bastion open to all IPs for port 22 because you want to be able to connect to it from anywhere. Yes you also of course lock it down and use best practice sshd config measures. But you only have ssh running and you use keys with passphrase for your outbound ssh connections for increased security. You keep it updated.

Your bastion will not be as locked down as your AWS instance though because it won’t have that AWS security in front of it, but you’re not concerned that much about your bastion, because it only has ssh listening, and you’ve disabled root ssh and password login, you keep it regularly updated, and the only thing you ever do with it is ssh to it and then ssh again to your important endpoints using a key WITH a passphrase.

Your AWS instance is your primary concern and is the important thing that you care about the most here. So you put the most protection in front of that in front of key services. Like OP said, ssh to the AWS instance sort of becomes “internal” so to speak, as you can only come in from the bastion with a key and passphrase.

Yes. Thank you for expanding my answer :)

Even better, no open port anywhere is actually needed.

I’m definitely not following you..

I find this very useful. I will setup a bastion. Thank you

I have a BuyVM.net 3.50/mo KVM slice that has been idling for maybe a year (yeah I know I need to get on it..), so I can transfer that to you if you want. BuyVM has been around forever and they are awesome, but are almost always sold out (just checked and they are).

If you want to reply with your contact details I can see if I can get that going and you can take over the vps if you want..

Other options for providers:

- netcup.eu (I use them.. they don’t have small instances but they are super great prices for the sizes they offer)

- Hetzner Cloud (never used but heard great things and their prices are very low)

- OVHCloud (I have dedi servers from Kimsufi and SoYouStart which have been great— OVH is the parent company)

- prgmr.com - these guys have been around since the very beginning of vps hosting. They wrote the book (quite literally) on Xen ( https://www.amazon.com/Book-Xen-Practical-System-Administrat... )

- low $ NAT VPS options:

https://clients.inceptionhosting.com/cart.php?gid=13

https://hosting.gullo.me/pricing

- free ipv6-only vps (or $1/mo to add an ipv4):

https://www.euserv.com/en/virtual-private-server/root-vserve...

- An informative resource: https://lowendbox.com/blog/free-vps-providers/

On your ssh bastion make sure to at the very least:

- have minimal services running, preferably only ssh

- if you have other services running then use iptables or a firewall frontend to block all incoming ports except for the ones you specifically need/want open

- disable root ssh login

- disable password login (eg use ssh keys and preferably with a passphrase too)

- you can also use something like fail2ban or denyhosts but it’s not really necessary if you’ve also done all the above

- yum or apt update it frequently

- For your ssh connection from the bastion to your AWS instance or any other important ssh destinations from the bastion, use a key with a passphrase for increased security (on the off chance your bastion gets compromised)

- Don’t do anything else too important on your ssh bastion.. eg don’t have any important stuff or work laying around on it or other services/applications running. Just use it as a jumphost only.

is there any way to just tunnel the ssh traffic trough the bastion but let the ssh authentication be done from my computer instead?

that way i would not need to keep the keys in the bastion server at all.

Yes, it’s fairly trivial with ProxyCommand or ProxyJump switch:

https://www.redhat.com/sysadmin/ssh-proxy-bastion-proxyjump

An AWS specific solution would be to use Systems Manager Sessions Manager to provide access into the instances and disallow all public access to the instances. No SSH. Identity is provided by IAM. As long as you use AWS cli and have configured it to login with your IAM creds, you can simply get a session via SSM. Added bonus is the ability to easily get an audit log in cloudwatch/S3. And no SSH keys/Linux user accounts to manage.

We use this for all our bastion hosts.

You need the agent running too but I believe this is the most secure way.

Just update the security group to your new IP address when it changes. You can also run a VPN, but that's more expensive

Definitely use SSM, it takes care of everything for you and you don't need a bastion.

Set your current IP in the security group and update it whenever it changes. You could make the update through the aws cli, this way you don’t have to login into the console every time

ip=$(curl https://api.ipify.org)

tag_name="tmpip"

tag_time=$(date -u '+%Y-%m-%dT%H:%M:%SZ')

aws ec2 authorize-security-group-ingress --group-id sg-86169665d2453e4 --protocol tcp --port 22 --cidr "$ip/32" --tag-specifications "ResourceType=security-group-rule,Tags=[{Key=Name,Value=${tag_name}},{Key=added,Value=${tag_time}}]"

The tag enables replacing the existing rule via tag Name, the further you go the easier it is to use the API SDKs

This is really a creative solution with no extra costs!

Use ssh pubkey auth and disable password. Set up a totp challenge after pubkey auth. Switch the default ssh port. Restrict ssh login to 1 user who's name isn't 'root' or 'ubuntu' or something everyone would look for. Wrap that in wireguard, and have ssh only listen on the wireguard interface if your worried, but that's an unnecessary extra step.

I'm lazy and just manually update my IP in the security group. It's not like the server shuts down when my home IP changes.

Disallow password logins, change ssh port, change/hide sshd identity information or configure port knocking.

Is changing SSH port really necessary? If your SSH is vulnerable, attackers will find the Port for it.

Surprisingly, just changing the port is highly effective. Scanning every ipv4 address still chews bandwidth even for just a handful of ports.

Add ipv6 into the mix and it's straight up infeasible to scan for even ONE port on every host!

Port knocking + key auth + non-default port is pretty damn good security, even against zero days in SSH.

So the solution is just to use IPV6 only SSH? I'm serious, you could use an ip just for SSH, making it very hard for anyone even to get the server address. And it's not that hard to be able to access ipv6 addresses.

Even if you don’t have a static IP, you can probably restrict to a /24 subnet or maybe /16.

Additionally, you can ensure password access is disabled and use ssh keys along with 2FA.

My ISP has at least 10 ranges (a result of the shortage and mergers) and there's little info about which ranges I could get IPs from.

Just curious but what would adding a /24 or /16 do if we're still allowing 0.0.0.0?

You would set it based on the range your ISP tends to assign you, and remove 0.0.0.0 for the ssh port.

Unfortunately. I have seen some ISPs DHCP servers assign IPs with no particular subnet(s). Could be a case here as well.

That’d be frustrating, not just for this. I’d probably be looking into a solution like Tailscale to tunnel out. Or just develop a gnarled security rule list full of rich history. :)

SSM

+1 - AWS Systems Manager Session Manager [1] is the future! (although it does have a ridiculous name...) It's free, and configuring its agent can enable you to do other useful maintenance tasks through it.

Assuming you're using Amazon Linux 2 as your AMI, it's just as simple as assigning a instance profile with the right permissions (you can use the managed AmazonSSMManagedInstanceCore policy [2]). And you'll then be able to connect through the AWS Console or CLI.

[1]: https://docs.aws.amazon.com/systems-manager/latest/userguide...

[2]: https://console.aws.amazon.com/iam/home#/policies/arn:aws:ia...

I will be going with this solution as it is free and simple. Thank you for briefing on this!

Tailscale, Openvpn, Your isp probably only rotates your IP every 2 weeks